日志存储 :从时间戳部分创建指纹

Question

我在基于客户端IP和包含日期+小时的时间戳创建指纹时遇到问题。 我正在使用日志 7.3.1。这里是我的配置文件的相关部分

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date{ 
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
...
ruby{
code => "
keydate = Date.parse(event.get('timestamp'))
event.set('keydate', keydate.strftime('%Y%m%d-%H'))
"
}
fingerprint {
key => "my_custom_secret"
method => "SHA256"
concatenate_sources => "true"
source => [
"clientip",
"keydate"
]
}
}

问题出在"红宝石"块中。我尝试了多种方法来计算密钥日期，但没有一种方法可以在不给我错误的情况下工作。 最后一个(使用此配置文件(是

[ERROR][logstash.filters.ruby    ] Ruby exception occurred: Missing Converter handling for full class name=org.jruby.ext.date.RubyDateTime, simple name=RubyDateTime

输入文档

{
"timestamp" => "19/Sep/2019:00:07:56 +0200",
"referrer" => "-",
"@version" => "1",
"@timestamp" => 2019-09-18T22:07:56.000Z,
...
"request" => "index.php",
"type" => "apache_access",
"clientip" => "54.157.XXX.XXX",
"verb" => "GET",
...
"tags" => [
[0] "_rubyexception"  # generated by the ruby exception above
],
"response" => "200"
}

预期产出

{
"timestamp" => "19/Sep/2019:00:07:56 +0200",
"referrer" => "-",
"@version" => "1",
"@timestamp" => 2019-09-18T22:07:56.000Z,
...
"request" => "index.php",
"type" => "apache_access",
"clientip" => "54.157.XXX.XXX",
"verb" => "GET",
...
"keydate" => "20190919-00", #format : YYYYMMDD-HH
"fingerprint" => "ab347766ef....1190af",
"response" => "200"
}

与往常一样，非常感谢您的帮助！

Answer 1

我建议删除 ruby 代码片段并使用内置日期过滤器：https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

您在 ruby 代码段中所做的正是日期过滤器所做的 - 从字段中提取时间戳并将其重建为您想要的格式。

另一种选择(不太推荐，但也可以(是使用 grok 来提取时间戳的相关部分并以不同的方式组合它们。