我是ELK的新手。我正在尝试使用Logstash和grok来解析日志。
Logstash配置：
input { stdin { } }
filter {
grok {
match => { "message" => "%{SPACE}%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{SYSLOG5424SD}%{SPACE}%{LOGLEVEL:Log_level}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}-%{SPACE}[operation=%{WORD:operation},%{SPACE}duration=%{NUMBER:duration}%{SPACE}sec]%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}(?:[^:]+)caller:%{SPACE}%{IPV4:caller},%{SPACE}username:%{SPACE}%{WORD:username}(?:[^:]+)%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}Headers:%{GREEDYDATA:Headers}%{SPACE}Payload:%{SPACE}{%{SPACE}%{GREEDYDATA:Payload}%{SPACE}}nn(?m)%{GREEDYDATA:java_stack_trace}nn" }
}
}
output {
file {
path => ["/tmp/test_log/output/output.log"]
codec => rubydebug
}
}
日志：
2022-03-28 01:19:58,178 [default task-5117] INFO LoggingFeature_LONGOPS - [operation=getClientStatus, duration=14.90 sec] REQ_IN
ID: 304711
Address: http://192.168.0.1:8080/test/test/services/getClientStatus [caller: 192.168.0.1, username: TEST_TEST]
HttpMethod: POST
Content-Type: application/json
Headers: {Authorization=********, Accept=application/json, text/plain, */*, User-Agent=axios/0.21.4, connection=close, content-type=application/json, Host=192.168.0.1:8080, Conten$
Payload:
{"data1":"7777777777","data2":1111}
Cpu usage (90 samples by 100 ms, total 9 sec):
java.lang.Thread.run (90) - 100%
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run (90) - 100%
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask (90) - 100%
在grokdebug.herokuapp.com上，它会给出一个包含所有所需数据的正常响应，而直接在Logstash中，第一行之后会出现错误：
{
"operation" => "getClientStatus",
"host" => "test",
"Log_level" => "INFO",
"@timestamp" => 2022-03-30T08:49:35.228Z,
"Parameters" => "LoggingFeature_LONGOPS ",
"duration" => "14.90",
"timestamp" => "2022-03-28 01:19:58,178",
"message" => "2022-03-28 01:19:58,178 [default task-5117] INFO LoggingFeature_LONGOPS - [operation=getClientStatus, duration=14.90 sec] REQ_IN",
"@version" => "1"
}
{
"@timestamp" => 2022-03-30T08:49:35.231Z,
"tags" => [
[0] "_grokparsefailure"
],
"host" => "test",
"message" => " ID: 304711",
"@version" => "1"
}
请告诉我，我做错了什么？
我试过使用
(?m)%{GREEDYDATA:Parameters:}
指定了输入中的多行编解码器
input {
file {
path => "/tmp/test_log/input/*.log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
还启用了选项
config.support_escapes: true
在logstash.yml 中
input {
file {
path => "/tmp/test_log/input/*.log"
codec => multiline {
pattern => "^%{DATE_EU}"
negate => true
what => previous
}
}