Elasticsearch:Grok管道不工作(不适用于日志)

Question

我创建了一个管道："ExtractOwasErrorFields"；其中grok处理器从消息字段中提取字段。当在Kibana测试它时，它工作得很好->堆栈管理->Ingest Pipelines->测试管道。我通过提供文档的_id和_index来使用真实的日志文档。处理器按照预期从消息字段中提取字段。当我将处理器添加到我的filebeat.yml时，"发现"中根本没有显示任何文档：

output.elasticsearch:
hosts: ["elasticsearch:9200"]
username: xxx
password: xxx
pipeline: ExtractOwaspErrorFields

我的管道配置如下：

{
"ExtractOwaspErrorFields" : {
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"%{OWASP_ERRORLOG}"
],
"pattern_definitions" : {
"OWASP_ERRORLOG" : "\[%{HTTPDERROR_DATE:timestamp}\] \[:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]?( \[client %{IPORHOST:client}:%{POSINT:clientport}\]) \[client %{IPORHOST:client_ip}\] ModSecurity: ?(%{APACHE_ERROR_MESSAGE:error}) \[file "%{PATH:matching_rule_file}"\] \[line "%{POSINT:matching_rule_line}"\] \[id "%{NUMBER:matching_rule_id}"\] \[msg "%{DATA:owasp_message_string}"\] \[data "%{DATA:owasp_message_data}"\] \[severity "%{WORD:owasp_severity}"\] \[ver "%{DATA:owasp_version}"\] %{GREEDYDATA:tags} \[hostname "%{HOSTNAME:hostname}"\] \[uri "%{URIPATHPARAM:uri}"\] \[unique_id "%{DATA:unique_id}"\]",
"APACHE_ERROR_MESSAGE" : "( .+?(?= \[%{WORD} "))"
},
"if" : "ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'",
"ignore_failure" : true
}
}
]
}
}

我测试管道的样本文件是：

{
"docs":
[
{
"_id": "OzAKyoIBILrgz4V8VcpG",
"_index": "filebeat-7.17.5-2022.08.02-000001",
"_source": {
"docker": {
"container": {
"labels": {
"com_docker_stack_namespace": "modsecurity"
}
}
},
"ecs": {
"version": "1.12.0"
},
"stream": "stderr",
"message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match "(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)" at ARGS:test. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "71"] [id "930110"] [msg "Path Traversal Attack (/../)"] [data "Matched Data: ../ found within ARGS:test: ../"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/126"] [hostname "modsecurity"] [uri "/"] [unique_id "YwSeR-gNfcPQLAl5gStNfAAAAQE"]"
}
}
]
}

管道测试结果：

{
"docs": [
{
"doc": {
"_index": "filebeat-7.17.5-2022.08.02-000001",
"_type": "_doc",
"_id": "OzAKyoIBILrgz4V8VcpG",
"_source": {
"owasp_severity": "CRITICAL",
"owasp_message_string": "Path Traversal Attack (/../)",
"pid": "226",
"error": " Warning. Pattern match "(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)" at ARGS:test.",
"tid": "139758264993536",
"clientport": "41062",
"docker": {
"container": {
"labels": {
"com_docker_stack_namespace": "modsecurity"
}
}
},
"owasp_version": "OWASP_CRS/3.3.2",
"hostname": "modsecurity",
"ecs": {
"version": "1.12.0"
},
"stream": "stderr",
"client": "10.0.1.48",
"client_ip": "10.0.1.48",
"timestamp": "Tue Aug 23 11:30:47.675452 2022",
"unique_id": "YwSeR-gNfcPQLAl5gStNfAAAAQE",
"matching_rule_id": "930110",
"message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match "(?:^|[\\\\/])\\\\.\\\\.(?:[\\\\/]|$)" at ARGS:test. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "71"] [id "930110"] [msg "Path Traversal Attack (/../)"] [data "Matched Data: ../ found within ARGS:test: ../"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/126"] [hostname "waf"] [uri "/"] [unique_id "YwSeR-gNfcPQLAl5gStNfAAAAQE"]",
"uri": "/",
"owasp_message_data": "Matched Data: ../ found within ARGS:test: ../",
"tags": "[tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/126"]",
"loglevel": "error",
"matching_rule_file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
"matching_rule_line": "71"
},
"_ingest": {
"timestamp": "2022-08-23T12:08:05.5017157Z"
}
}
}
]
}

有人能告诉我我的管道出了什么问题，或者我错过了什么；"测试管道"；但当我在filebeat.yml中将它们用于真实日志时，就不会了非常感谢

Answer 1

通过将管道重命名为仅包含小写字符来解决此问题。显然，filebeat.yml的解析将管道值转换为小写，因此在弹性端找不到。