关于ELK中时间戳的问题?

Question

我在ELK中面临时间戳问题，

现在我面临的问题是ELK的@timestamp字段，它在导入旧日志文件时显示当前日期时间

我需要从自定义datetime字段从日志更新@timestamp。

下面是日志示例。

{​​​​​​​​ "datetime":"2021-08-24 04:13:39,167", "servername":"vm-ws", "serverip":"(null)", "process":"4656", "thread":"4", "level":"DEBUG", "appname":"AcManager", "page":"Program.cs ","method":"ExecuteAsync","line":"63","message":"Starting AcMa Module","otherinfo":{​​​​​​​​"token":"null","clientip":"null","clientbrowserversion":"null","clienttype":"null"}​​​​​​​​,"moreinfo":"null"}​​​​​​​​

我在logstash

中使用了以下配置的grok过滤器

input {
stdin {
type => "stdin-type"
}
file {
type => "json"
path => [ "/home/testuser/mylogs/*.log", "/home/testuser/mylogs/*/*.log" ]
start_position => "beginning"
}
}
filter {
date {
match => ["datetime", "yyyy-MM-dd HH:mm:ss"]
target => ["@timestamp"]
}
# Step 1. Extract the JSON String, put it in a temporary field called "payload_raw"
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
grok {
match => {
"message" => [ "%{JSON:payload_raw}" ]
}
pattern_definitions => {
"JSON" => "{.*$"
}
}
# Step 2. Parse the temporary "payload_raw" field, put the parsed data in a field called "payload"
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html
json {
source => "payload_raw"
target => "payload"
}

# Step 3. Remove the temporary "payload_raw" field (and other fields)
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html
mutate {
remove_field => [ "payload_raw","message" ]
}
#   Tried this but not working
#   date {
#                match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
#                target => "@timestamp"
#        }
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts  => "localhost:9200"
}
}

Answer 1

Since your "date "字段存在于"有效负载"中。字段，您需要以这种方式提及该字段:

date {
match => [ "[payload][datetime]", "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
}

应该是正确的脚本:

input {
stdin {
type => "stdin-type"
}
file {
type => "json"
path => [ "/home/testuser/mylogs/*.log", "/home/testuser/mylogs/*/*.log" ]
start_position => "beginning"
}
}
filter {
# Step 1. Extract the JSON String, put it in a temporary field called "payload_raw"
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
grok {
match => {
"message" => [ "%{JSON:payload_raw}" ]
}
pattern_definitions => {
"JSON" => "{.*$"
}
}
# Step 2. Parse the temporary "payload_raw" field, put the parsed data in a field called "payload"
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html
json {
source => "payload_raw"
target => "payload"
}

# Step 3. Remove the temporary "payload_raw" field (and other fields)
# Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html
mutate {
remove_field => [ "payload_raw","message" ]
}
#   Try this
date {
match => [ "[payload][datetime]", "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
}

}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts  => "localhost:9200"
}
}