如何在logstash过滤器中从多个日志行生成单个输出json对象?

Question

我是Logstash和Grok过滤器的新手。我想解析这样的日志 -

2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | CommittedVirtualMemorySize :: 401186816 
2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreePhysicalMemorySize :: 1751130112 
2018-01-11 17:17:16,072 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreeSwapSpaceSize :: 4294967295 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuLoad :: -1.0 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuTime :: 47471104300 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | SystemCpuLoad :: 1.0 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalPhysicalMemorySize :: 4285849600 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalSwapSpaceSize :: 4294967295 

到这样的 JSON 对象 -

{
"timestamp": "2018-01-11 17:17:16,071",
"log_level": "DEBUG",
"thread_name": "Thread-2",
"class": "com.example.monitor.MonitorHelper",
"method": "cpuMonitoring",
"line_number": "307",
"CommittedVirtualMemorySize": "401186816",
"FreePhysicalMemorySize": "1751130112",
"FreeSwapSpaceSize": "4294967295",
"ProcessCpuLoad": "-1.0",
"ProcessCpuTime": "47471104300",
"SystemCpuLoad": "1.0",
"TotalPhysicalMemorySize": "4285849600",
"TotalSwapSpaceSize": "4294967295"
}

截至目前，我的 grok 模式是 -

%{TIMESTAMP_ISO8601：timestamp} \| %{LOGLEVEL：log_level} \|[(？\b[\w-]+\b)] \|%{JAVAFILE：class}：%{JAVAMETHOD：method}(%{NUMBER：line_number}) \|%{贪婪数据：log_message}

为每个输入日志行提供多个输出行。JSON 对象看起来像这样 -

{
"timestamp": "2018-01-11 17:17:16,071",  
"log_level": "DEBUG",
"thread_name": "Thread-2",
"class": "com.example.monitor.MonitorHelper",
"method": "cpuMonitoring",
"line_number": "307",
"log_message": "CommittedVirtualMemorySize :: 401186816 "
}

你能帮我解决这个问题需要寻找什么吗？

Answer 1

第一个建议是将原始日志输出更改为单行。

如果您不能，并且您正在使用 filebeat 来发送文件，请使用 FB 的多行配置在将其发送到 logstash 之前合并这些行。

如果您不使用 filebeat，则可以尝试在 logstash 中使用多行编解码器。