如何用日志时间戳替换logstash@timestamp

Question

我在日志中的时间戳的格式如下

2016-04-07 18:11:38.169  which is  yyyy-MM-dd HH:mm:ss.SSS

这个日志文件不是活的(存储的/旧的)，我正试图用logstash@timestamp值替换这个timpestamp，以改进Kibana可视化。

我在logstash中的过滤器如下

grok {
match => {
"message" => [ "(?<timestamp>(d){4}-(d){2}-(d){2} (d){2}:(d){2}:(d){2}.(d){3}) %{SYSLOG5424SD} ERROR u%{BASE16FLOAT}.%{JAVACLASS} - TransId:2b948ed5-12c0-4ae0-9b99-f1ee01191001 - TransactionId ::"2b948ed5-12c0-4ae0-9b99-f1ee01191001"- Actual Time taken to process :: %{NUMBER:responseTime:int}" ]
} 
}
date {
match => [ "timestamp:date" , "yyyy-MM-dd HH:mm:ss.SSS Z"  ]
timezone => "UTC"
target => "@timestamp" 
} 

但是，它并没有取代@timestamp值，Json值

{
"_index": "logstash-2017.02.09",
"_type": "logs",
"_id": "AVoiZq2ITxwgj2avgkZa",
"_score": null,
"_source": {
"path": "D:\SoftsandTools\Kibana\Logs_ActualTimetakentoprocess.log",
"@timestamp": "2017-02-09T10:23:58.778Z", **logstash @timestamp**
"responseTime": 43,
"@version": "1",
"host": "4637",
"message": "2016-04-07 18:07:01.809 [SimpleAsyncTaskExecutor-3] ERROR s.v.wsclient.RestClient - TransId:2b948ed5-12c0-4ae0-9b99-f1ee01191001 - TransactionId ::"2b948ed5-12c0-4ae0-9b99-f1ee01191001"- Actual Time taken to process :: 43",
"timestamp": "2016-04-07 18:07:01.809"   **Mine time stamp**
}

样本日志行-

2016-04-07 18:11:38.171 [SimpleAsyncTaskExecutor-1] ERROR s.v.wsclient.RestClient - TransId:2b948ed5-12c0-4ae0-9b99-f1ee01191001 - TransactionId ::"2b948ed5-12c0-4ae0-9b99-f1ee01191001"- Actual Time taken to process :: 521

你能帮忙告诉我吗？我要去哪里。。

Answer 1

为了使用日志行的timestamp，您基本上应该有一个grok匹配：

grok {
patterns_dir => ["give your path/patterns"]
match => { "message" => "^%{LOGTIMESTAMP:logtimestamp}%{GREEDYDATA}" }          
}

在你的模式文件中，确保在日志中有与你的timestamp匹配的模式，它可能看起来像这样：

LOGTIMESTAMP %{YEAR}%{MONTHNUM}%{MONTHDAY} %{TIME}

然后，一旦你完成了grok过滤，你就可以使用过滤后的值，比如：

mutate {
add_field => { "newtimestamp" => "%{logtimestamp}" }
remove_field => ["logtimestamp"]
}
date {
match => [ "newtimestamp" , "ISO8601" , "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"  <-- the timestamp which you wanted to apply on
locale => "en"
timezone => "UTC"
}

希望这能有所帮助！

Answer 2

您可以使用logstash 的日期过滤器插件

date {
match => ["timestamp", "UNIX"]
}