Logstash版本1.5.0.1
我正在尝试使用logstash s3input插件来下载云端日志和cloudfront编解码器插件来过滤stream。
我安装了bin/plugin install logstash-codec-cloudfront 。
我得到以下内容： 错误：对象：#Version：1.0不是这个包装的合法参数，因为它不响应“读取”。
以下是/var/logs/logstash/logstash.log中的完整错误消息
{:timestamp=>"2015-08-05T13:35:20.809000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::S3 bucket=>\"[BUCKETNAME]\", prefix=>\"cloudfront/\", region=>\"us-east-1\", type=>\"cloudfront\", secret_access_key=>\"[SECRETKEY]/1\", access_key_id=>\"[KEYID]\", sincedb_path=>\"/opt/logstash_input/s3/cloudfront/sincedb\", backup_to_dir=>\"/opt/logstash_input/s3/cloudfront/backup\", temporary_directory=>\"/var/lib/logstash/logstash\">\n Error: Object: #Version: 1.0\n is not a legal argument to this wrapper, cause it doesn't respond to \"read\".", :level=>:error}
我的logstashconfiguration文件：/etc/logstash/conf.d/cloudfront.conf
input { s3 { bucket => "[BUCKETNAME]" delete => false interval => 60 # seconds prefix => "cloudfront/" region => "us-east-1" type => "cloudfront" codec => "cloudfront" secret_access_key => "[SECRETKEY]" access_key_id => "[KEYID]" sincedb_path => "/opt/logstash_input/s3/cloudfront/sincedb" backup_to_dir => "/opt/logstash_input/s3/cloudfront/backup" use_ssl => true } }
我正在使用一个类似的s3inputstream成功，让我的云轨日志到logstash基于来自一个stackoverflowpost的答案 。
来自s3的CloudFront日志文件（我只包含文件中的头文件）：
#Version: 1.0 #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type
标题看起来基本上是基于cloudfront插件github repo cloudfront_spec.rb和官方AWS CloudFront 访问日志文档中第26-29行的正确格式。
有任何想法吗？ 谢谢！
[更新9/9/2015]
基于这篇文章，我尝试使用gzip_lines编解码器插件，安装bin/plugin install logstash-codec-gzip_lines并用filterparsing文件，不幸的是我得到了完全相同的错误。 看起来这是日志文件的第一个字符有# 。
为了logging，这里是新的尝试，包括由于四个新字段而parsing云端日志文件的更新模式：
/etc/logstash/conf.d/cloudfront.conf
input { s3 { bucket => "[BUCKETNAME]" delete => false interval => 60 # seconds prefix => "cloudfront/" region => "us-east-1" type => "cloudfront" codec => "gzip_lines" secret_access_key => "[SECRETKEY]" access_key_id => "[KEYID]" sincedb_path => "/opt/logstash_input/s3/cloudfront/sincedb" backup_to_dir => "/opt/logstash_input/s3/cloudfront/backup" use_ssl => true } } filter { grok { type => "cloudfront" pattern => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}" } mutate { type => "cloudfront" add_field => [ "listener_timestamp", "%{date} %{time}" ] } date { type => "cloudfront" match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ] } }
我有同样的问题，从改变
codec > "gzip_lines"
至
codec => "plain"
在input中为我固定。 看起来像S3input自动解压缩gzip文件。 https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L13
这里的FTR是为我工作的完整configuration：
input { s3 { bucket => "[BUCKET NAME]" delete => false interval => 60 # seconds prefix => "CloudFront/" region => "us-east-1" type => "cloudfront" codec => "plain" secret_access_key => "[SECRETKEY]" access_key_id => "[KEYID]" sincedb_path => "/opt/logstash_input/s3/cloudfront/sincedb" backup_to_dir => "/opt/logstash_input/s3/cloudfront/backup" use_ssl => true } } filter { if [type] == "cloudfront" { if ( ("#Version: 1.0" in [message]) or ("#Fields: date" in [message])) { drop {} } grok { match => { "message" => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}" } } mutate { add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "listener_timestamp", "%{date} %{time}" ] } date { match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ] } date { locale => "en" timezone => "UCT" match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ] target => "@timestamp" add_field => { "debug" => "timestampMatched"} } } }