日志存储管道不存在

Question

我正从另一个问题进入这个问题。

我创建了如下管道。

{
"bool-pipeline": {
"description": "converts FALSE/TRUE to boolean",
"processors": [
{
"convert": {
"field": "secure_flag",
"type": "boolean",
"ignore_missing": true
}
}
]
}
}

我的有效值为FALSE、TRUE、[Blanks]

我能够取回管道。

我在Logstash配置中添加了如下内容。

input {
file {
path => "/Users/gibbs/Documents/search/mini_system.csv"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => [
"secure_flag",
"mini_system_key",
"hw_contract_end_date"
]
separator => ","
}
mutate {
remove_field => ["path", "host"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
hosts => ["127.0.0.1:9200"]
index => "mini_system"
document_id => "%{mini_system_key}"
pipeline => "%{bool-pipeline}"
}
}

但当我加载数据时，它会抛出以下错误。

[2020-05-10T16:13:05,691][DEBUG][o.e.a.b.T.BulkRequestModifier] [gopir-mac-1] failed to execute pipeline [_none] for document [mini_system/_doc/50395971|1038832]
java.lang.IllegalArgumentException: pipeline with id [%{bool-pipeline}] does not exist
at org.elasticsearch.ingest.IngestService.executePipelines(IngestService.java:407) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.ingest.IngestService.access$000(IngestService.java:75) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:384) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]

我一无所知。我已经试了好几个小时了。

有什么建议吗？

当我检查日志时，我在弹性搜索日志中看到了这个错误。Logstash只有一行。

Answer 1

您的摄取管道名称是bool-pipeline，您应该在elasticsearch中使用它，而不是在%{bool-pipeline}中使用它。

%{FIELD}用于与事件相关的值，但您的情况并非如此，使用%{bool-pipeline}将使logstash尝试从事件中名为bool-pipeline的字段中提取具有摄取管道名称的值，由于该字段不存在，您将收到此错误。