我们从服务器发送日志,并在每台服务器上使用logstash进行装运。
所以我们从glob "/root/Desktop/Logstash-Input/**/*_log"读取日志。
input { file{ path => "/root/Desktop/Logstash-Input/**/*_log" start_position => "beginning" } }
从这个glob我们从我们想要添加到事件的path提取字段 。 例如:从目录path提取server , logtype等。我们这样做:
filter { grok { match => ["path", "/root/Desktop/Logstash-Input/(?<server>[^/]+)/(?<logtype>[^/]+)/(?<logdate>[\d]+.[\d]+.[\d]+)/(?<logfilename>.*)_log"] } }
然后我们使用lumberjack输出插件将这些日志输出到中央logstash服务器。
output { lumberjack { hosts => ["xx.xx.xx.xx"] port => 4545 ssl_certificate => "./logstash.pub" } stdout { codec => rubydebug } }
问题在于,运送到中央服务器的日志会丢失使用grok添加的字段。 例如中央服务器上不存在server , logtype等。 但是,客户机控制台显示添加的字段,但在中央logstash服务器上仅message , timestamp , version 。
客户端(从哪里发送日志)控制台:
output received {:event=>{"message"=>"2014-05-26T00:00:01+05:30 host crond[268]: (root) CMD (2014/05/31/server2/cron/log)", "@version"=>"1", "@timestamp"=>"2014-07-16T06:07:21.927Z", "host"=>"host", "path"=>"/root/Desktop/Logstash-Input/Server2/CronLog/2014.05.31/cron_log", "server"=>"Server2", "logtype"=>"CronLog", "logdate"=>"2014.05.31", "logfilename"=>"cron"}, :level=>:debug, :file=>"(eval)", :line=>"37"} { "message" => "2014-05-26T00:00:01+05:30 bx920as1 crond[268]: (root) CMD (2014/05/31/server2/cron/log)", "@version" => "1", "@timestamp" => "2014-07-16T06:07:21.927Z", "host" => "host", "path" => "/root/Desktop/Logstash-Input/Server2/CronLog/2014.05.31/cron_log", "server" => "Server2", "logtype" => "CronLog", "logdate" => "2014.05.31", "logfilename" => "cron" }
中央服务器(日志传送到)控制台 :
{ "message" => "2014-07-16T05:33:17.073+0000 host 2014-05-26T00:00:01+05:30 bx920as1 crond[288]: (root) CMD (2014/05/31/server2/cron/log)", "@version" => "1", "@timestamp" => "2014-07-16T05:34:02.370Z" }
因此,运送时,grokked域将被丢弃。 为什么这样?
我怎样才能保留字段?
解决了:
我通过添加codec => "json"到我的伐木工人输出和input来解决它。
输出:
output { lumberjack { hosts => ["xx.xx.xx.xx"] port => 4545 ssl_certificate => "./logstash.pub" codec => "json" }
input:
input { lumberjack { port => 4545 ssl_certificate => "/etc/ssl/logstash.pub" ssl_key => "/etc/ssl/logstash.key" codec => "json" } }