在插入到Elasticsearch之前删除NULL值的嵌套字段

Question

我使用Metricbeat查询数据库并将结果发送到Logstash。其中一个查询返回一个可能包含NULL值的日期列，让我们称之为"；records_last_failure"；。

我在logstash中得到以下错误：

[2020-12-29T15:45:59,077][WARN ][logstash.outputs.elasticsearch][my_pipeline] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-index", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x4f7d2f63>], :response=>{"index"=>{"_index"=>"my-index", "_type"=>"_doc", "_id"=>"wRItr3YBUsMlz2Mnbnlz", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [sql.metrics.string.records_last_failure] of type [date] in document with id 'wRItr3YBUsMlz2Mnbnlz'. Preview of field's value: 'NULL'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [NULL] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

我的目标是在尝试将文档保存到Elasticsearch之前删除此字段。我试着在Logstash中使用以下两个过滤器，但它们都没有帮助，我仍然会收到上面提到的错误。过滤器：

filter {
......
......
if "[sql.metrics.string.records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql.metrics.string.records_last_failure]" ]
}
}
if "[sql][metrics][string][records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql][metrics][string][records_last_failure]" ]
}
}
}

Answer 1

不确定原因或方式，但在几次重新启动后，logstash开始正常工作。

我的json看起来像下面的例子：

"a" : 1,
"b" : 2,
"sql" : {
"metrics" : {
"string" : {
"record_last_failure" : "NULL",
"other_field" : 5
}
} 
}

如果字段sql.metrics.string.record_last_failure设置为null，我想删除它。我在过滤器中使用的语法：

if "[sql][metrics][string][records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql][metrics][string][records_last_failure]" ]
}