Elasticsearch搜索查询指南

Question

目前我正在制作ELK 8(Filebeat, Logstash, Elasticsearch &;卡在中间

示例日志在这里，

2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session opened OId 31324
2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Transaction started TId T312487
........
2021-07-30 16:34:22 INFO  com.example.OrderRest [tegq842sh980] Session opened OId 31325
2021-07-30 16:34:22 INFO  com.example.OrderRest [tegq842sh980] Transaction started TId T312488
........
2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session closed
2021-07-30 16:34:22 INFO  com.example.OrderRest [tegq842sh980] Session closed

问题:如果我用OId查询31324年，只获取以下匹配的文档

GET _search
{
"query":{
"match": {"OId": "31324"}
}
}

结果:

2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session opened OId 31324

相反，结果还应该包括事务生成的所有下一个匹配记录，请求idfhxh750df392像

2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session opened OId 31324
2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Transaction started TId T312487
........
2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session closed

例如，如果我用TId查询T312487，结果应为

2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Transaction started TId T312487
........
2021-07-30 16:34:22 INFO  com.example.OrderRest [fhxh750df392] Session closed

我是新来的ELK谁能指导我怎么做

Answer 1

您可以使用下面的grok模式将日志转换为结构:

%{DATESTAMP:date}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{GREEDYDATA:class}%{SPACE}[%{GREEDYDATA:pid}]%{SPACE}%{GREEDYDATA:message}

上面的grok模式将以以下格式转换每个日志行:

{
"date": "21-07-30 16:34:22",
"level": "INFO",
"pid": "fhxh750df392",
"message": "Session opened OId 31324",
"class": "com.example.OrderRest "
}

我将括号中的值视为进程id，例如[fhxh750df392]。

一旦你在索引中有了以上的日志，你可以使用两个单独的查询来获取数据:

首先查询获取匹配查询的pid:

{
"size": 0,
"query": {
"match": {
"message": "31324"
}
}, 
"aggs": {
"pid": {
"terms": {
"field": "pid.keyword",
"size": 10
}
}
}
}

第一次查询的结果:

"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"pid" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "fhxh750df392",
"doc_count" : 1
}
]
}
}

第二个查询这里你需要使用pid从上面的响应，并传递到下面的查询。

{
"size": 20,
"_source": false, 
"query": {
"terms": {
"pid": [
"fhxh750df392"
]
}
},
"collapse": {
"field": "pid.keyword",
"inner_hits": {
"name": "logs"
}
}
}

第二个查询的响应，与您正在查找的类似:

"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "logtest",
"_type" : "_doc",
"_id" : "BMXTFIEB3bStuRaWDAtm",
"_score" : 1.0,
"fields" : {
"pid.keyword" : [
"fhxh750df392"
]
},
"inner_hits" : {
"logs" : {
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logtest",
"_type" : "_doc",
"_id" : "BMXTFIEB3bStuRaWDAtm",
"_score" : 1.0,
"_source" : {
"date" : "21-07-30 16:34:22",
"level" : "INFO",
"pid" : "fhxh750df392",
"message" : "Session opened OId 31324",
"class" : "com.example.OrderRest "
}
},
{
"_index" : "logtest",
"_type" : "_doc",
"_id" : "BcXTFIEB3bStuRaWkAuX",
"_score" : 1.0,
"_source" : {
"date" : "21-07-30 16:34:22",
"level" : "INFO",
"pid" : "fhxh750df392",
"message" : "Transaction started TId T312487",
"class" : "com.example.OrderRest "
}
},
{
"_index" : "logtest",
"_type" : "_doc",
"_id" : "BsXTFIEB3bStuRaWwAua",
"_score" : 1.0,
"_source" : {
"date" : "21-07-30 16:34:22",
"level" : "INFO",
"pid" : "fhxh750df392",
"message" : "Session closed",
"class" : "com.example.OrderRest "
}
}
]
}
}
}
}
]
}