使用grok过滤Logstash中的Apache错误日志

Question

我需要使用grok过滤apache错误日志。请帮我处理我无法创建的模式！。

我的示例日志：

2020-10-07T01:21:26.403-0400    ERROR   [reload]        cfgfile/list.go:96  Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:36.404-0400    ERROR   [reload]        cfgfile/list.go:96  Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:38.925-0400    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://IP:5044)): dial tcp IP:5044: i/o timeout
2020-10-07T01:21:38.925-0400    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://IP:5044)) with 26743 reconnect attempt(s)
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:191     done
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:168     done

我通过grok模式了解，我们可以在下面使用这些，但我不知道如何在grok模式中使用：

# Error logs
HTTPD20_ERRORLOG [%{HTTPDERROR_DATE:timestamp}] [%{LOGLEVEL:loglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG [%{HTTPDERROR_DATE:timestamp}] [%{WORD:module}:%{LOGLEVEL:loglevel}] [pid %{POSINT:pid}(:tid %{NUMBER:tid})?]( (%{POSINT:proxy_errorcode})%{DATA:proxy_message}:)?( [client %{IPORHOST:clientip}:%{POSINT:clientport}])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

有人能帮忙吗！提前感谢！

Answer 1

处理完样本数据后，这个grok模式必须工作：

filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601}%{SPACE}%{LOGLEVEL}(%{SPACE}[%{WORD:action}])?%{SPACE}%{WORD:package}/%{WORD:class}.go:%{INT:line:number}%{SPACE}%{GREEDYDATA:message}$" }
}
}

你的数据不完全是http，所以需要一个自定义模式，我想我的grok在没有空间的情况下一定更容易阅读，我建议你使用mutate gsub来统一空间(请记住我的解决方案中最后一句名为"message"的话(。

您可以在此处了解有关此模式和其他模式的更多详细信息。