将输入标识符添加到docker中logstash的日志记录中

Question

我有一个http_poller插件，在我的logstash管道中有多个URL和一个用于弹性搜索的输出。我还有一个自定义的log4j2.properties文件(其中还没有任何自定义文件(。我遇到的问题是，当output.aelasticsearch失败时，日志记录没有给我足够的上下文来说明哪个输入url是原因。所以有几个问题：

• 如何将对象中的字段添加到日志中？我见过%notEmpty{[%X{pipeline.id}]}的使用，但不知道如何使用管道中文档中的字段
• 如何在http_poller插件中为每个url添加一个字段，该字段可以从输出插件添加到日志记录中

input {
http_poller {
id => "medium-pull"
urls => {
url1 => {               
method => post
url => "${ROOT}/endpoint1"
body => '{"ids": [],  "limit": 10000,  "page": 1}'
}
url2 => {               
method => post
url => "${ROOT}/endpoint2"
body => '{"ids": [],  "limit": 10000,  "page": 1}'
}

错误日志示例：

logstash-pull1   | [2021-07-22T14:11:35,112][WARN ][logstash.outputs.elasticsearch][main][elasticsearch]  
Could not index event to Elasticsearch. 
{:status=>404, :action=>["index", {:_id=>"%{id}", :_index=>"prefix-%{objectType}", :routing=>nil, :_type=>"_doc"}, 
#<LogStash::Event:0x77f611be>], :response=>{"index"=>{"_index"=>"prefix-%{objectType}", "_type"=>"_doc", "_id"=>"%{id}", "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [prefix-%{objectType}] and [action.auto_create_index] ([".security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"]) doesn't match", "index_uuid"=>"_na_", "index"=>"prefix-%{objectType}"}}}}

Answer 1

通常应该打印"name"=>quot；url"；在具有现有配置的输出事件中。这样，您应该能够识别哪些url存在问题等。

另一种方法是具有两个具有不同id的http_pollers，例如

input {
http_poller {
id => "medium-pull"
urls => {
url1 => {               
method => post
url => "${ROOT}/endpoint1"
body => '{"ids": [],  "limit": 10000,  "page": 1}'
}
}
}
http_poller {
id => "large_pull"
urls => {
url2 => {               
method => post
url => "${ROOT}/endpoint2"
body => '{"ids": [],  "limit": 10000,  "page": 1}'
}
}
}
}

我相信你可能已经看过这份文件了，但以防万一https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html