Apps that use emails as the main mechanism to report errors and exceptions always seem like a good idea at first. As Java开发人员 we think, “hey, I’ll immediately know when something goes wrong!”

However, at some point you are bound to realize that the error emails are getting away from you and you need to clean up your email inbox. 你可以忽略404(大多数时候), 收到新注册的通知也不错, but at the end of the day it may become a bit too hectic or time consuming.

Sooner or later, distinguishing “actionable” emails from pointless notifications becomes difficult. You can set up rules and filters on your mail, but even these can start to become unmanageable.

This Java教程 会考虑使用的可能性吗 Logstash to regain control and clean up your email inbox and make your error emails manageable again, 所有这些都不需要改变应用程序中的任何东西.

Logstash设置和输入

The first step in our Logstash tutorial is to ensure that all the email you receive from your system goes to one folder. Since we’re moving all of the sorting and managing out of your inbox, 它不再是一个大文件夹了. Go ahead and remove all the folders and filters you already have set up, 然后减少到一个文件夹和一个滤镜.

将所有电子邮件从“system@my-awesomeapp”中移出.到文件夹“myawesomeappemail”. If you have a separate mailbox for these emails, then it will be even easier.

Now we can set up Logstash to poll that folder and parse the mails using the IMAP plugin. Version 1.4.2只支持从 Inbox，但有一个相对简单的 fix 这已经应用于 version 1.5 允许轮询特定文件夹. 如果你没有单独的邮箱, please apply the patch to the IMAP plugin in your Logstash instance before continuing.

# /etc/logstash/conf.d / 01-imap-input.conf
input {
  imap {
	host => "imap.yourmailserver.com"
	user => "username"
	password => "password"
	secure => true
	fetch_count => 15
	folder => "MyAwesomeAppEmails" # This line will only work if you apply the above mentioned patch
  }
}

This will start checking new emails and parsing them into Logstash events.

Logstash过滤器

A lot of useful data is parsed from emails into different event properties - notice that the email timestamp is used as the “@timestamp” for the event. 进一步, from the headers alone we can do things like identify the host that the error originated from:

filter {
  mutate {
	replace => [ "received", "%{[received][-1]}" ]
  }
  grok {
	match => [ "received", "from %{HOSTNAME:hostname} \(%{HOSTNAME:full_hostname} \[%{IP:ip}\]\)" ]
  }
  mutate {
	remove_field => [ "received" ]
  }
}

然而，这还不足以驯服你的错误邮件. We need a bit more, specifically the following three steps describing:

• 错误的类型
• 错误的严重程度
• 邮件中包含的任何错误细节

假设您输入了错误的名称e.g. “小部件失败”, as well as the severity of the error “ERROR” in the subject of the email like this: “ERROR: Widget Failed in /var/www/myapp/foobar.php 20”.

我们将使用它来设置事件的几个属性:

filter {
  grok {
	match => [ "subject", "%{WORD:severity}: %{DATA:type} in %{PATH:path} %{POSINT:line}" ]
  }
}

Logstash附带了许多 预定义的模式 你可以在日志和其他地方看到. Use these to build up your Grok patterns and make them easier to read. 我们用了“WORD”(一个单词), “DATA”(非贪心的统称), “PATH” (a Unix or Windows file path) and “POSINT” (a positive integer). 你也可以用 神交调试器 调试您的Grok模式. Messages that don’t match the pattern will be tagged with the “_grokparsefailure” tag.

这就解决了类型问题, severity, source file, and line of the error - basically all relevant meta data of the event. 现在进入细节部分.

微调日志库

You may have previously added a nice header or footer signature to your email, 做一个有礼貌的人, 但现在它挡道了. 让我们把它从消息中删除, as well as any trailing whitespace so that we can parse the rest of the email for the error stacktrace:

filter {
  mutate {
	# gsub takes 3 elements per substitution: field, regular expression and substitution
	gsub => [
  	“信息”，“你的开发团队”，“”，
  	"message"， "更多错误细节:"，""
	]
  }
  mutate {
	strip => "message"
  }
  # Split the message into an array of lines, each containing a line of the stacktrace
  mutate {
	split => [ "message", "\n" ]
  }
}

“gsub”突变使用标准 Ruby Regexp object, so all the options and features are available in logstash as well.

通过Elasticsearch和Amazon SNS输出

让我们将其应用到Elasticsearch实例中，使用 Logstash Elasticsearch output so that we can easily search and quantify the data we’re collecting:

output {
  如果"_grokparsefailure"不在[tags] {
	Elasticsearch {}
  }
}

We’re excluding all messages that didn’t Grok properly by checking for the “_grokparsefailure” tag. The assumption is that only emails whose subject matches the specified subject should be interpreted as error emails. 除此之外，它是简单直接的.

Logstash comes with a plethora of outputs, so let’s enhance this even more using SNS output 通知我们使用亚马逊网站的重大错误 简易通知服务 (SNS). We’ll assume that all errors of type “notifiable” need to generate a notification. If you’re not on an EC2 instance, you’ll need to specify an AWS key and secret.

output {
  如果[tags]{中的"notifiable"
	sns {
  	region => "us-east-1"
  	arn => "arn:aws:sns:us-east-1:1234567890123456:mytopic"
  	access_key_id => "AWS ACCESS_KEY" # Only specify these if you're not on an EC2 instance
  	secret_access_key => "AWS ACCESS SECRET" # Only specify these if you're not on an EC2 instance
	}
  }
}

将错误标记为“应通知的”取决于您. You can do that by looking at either the severity of the error or looking at the type of the error.

现在你又可以阅读重要的邮件了, 你可以确保你不会错过重要的错误邮件. 您还可以对要修复的错误做出明智的决定, as you can see how often an error occurs and when it occurred last. 搜索特定的错误也更加容易和快速, thanks to the awesome searching powers of Elasticsearch outlined in this Logstash tutorial.