我如何完全解析json到ElasticSearch?

Question

我正在解析mongodb输入到logstash，配置文件如下:

input {
mongodb {
uri => "<mongouri>"
placeholder_db_dir => "<path>"
collection => "modules"
batch_size => 5000
}
}
filter {
mutate  {
rename => { "_id" => "mongo_id" }
remove_field => ["host", "@version"]
}
json   {
source => "message"
target => "log"
} 
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "mongo_log_modules"
}
}

从集合中输出2/3个文档到elasticsearch。

{
"mongo_title" => "user",
"log_entry" => "{"_id"=>BSON::ObjectId('60db49309fbbf53f5dd96619'), "title"=>"user", "modules"=>[{"module"=>"user-dashboard", "description"=>"User Dashborad"}, {"module"=>"user-assessment", "description"=>"User assessment"}, {"module"=>"user-projects", "description"=>"User projects"}]}",
"mongo_id" => "60db49309fbbf53f5dd96619",
"logdate" => "2021-06-29T16:24:16+00:00",
"application" => "mongo-modules",
"@timestamp" => 2021-10-02T05:08:38.091Z
}
{
"mongo_title" => "candidate",
"log_entry" => "{"_id"=>BSON::ObjectId('60db49519fbbf53f5dd96644'), "title"=>"candidate", "modules"=>[{"module"=>"candidate-dashboard", "description"=>"User Dashborad"}, {"module"=>"candidate-assessment", "description"=>"User assessment"}]}",
"mongo_id" => "60db49519fbbf53f5dd96644",
"logdate" => "2021-06-29T16:24:49+00:00",
"application" => "mongo-modules",
"@timestamp" => 2021-10-02T05:08:38.155Z
}

似乎stdout的输出将不可解析的代码抛出到

"log_entry">

添加"rename"字段"modules"不添加字段

我已经尝试了grok突变过滤器，但在_id %{DATA}， %{QUOTEDSTRING}和%{WORD}不适合我。

我也试过更新一个嵌套映射到索引，似乎没有工作

还有什么我可以尝试得到完全嵌套的代码到elasticsearch?

Answer 1

解决方案是使用mutate

进行过滤

mutate { gsub => [ "log_entry", "=>", ": " ] }
mutate { gsub => [ "log_entry", "BSON::ObjectId('([0-9a-z]+)')", '"1"' ]}
json { source => "log_entry" remove_field => [ "log_entry" ] }

输出到stdout

"_id" => "60db49309fbbf53f5dd96619",
"title" => "user",
"modules" => [
[0] {
"module" => "user-dashboard",
"description" => "User Dashborad"
},
[1] {
"module" => "user-assessment",
"description" => "User assessment"
},
[2] {
"module" => "user-projects",
"description" => "User projects"
}
],