计算Elasticsearch中滑动时间窗口的变化百分比

Question

Elasticsearch新手。我有一系列类似的日志消息

{
"@timestamp": "whatever",
"type": "toBeMonitored",
"success": true
}

与昨天相同的时间间隔相比，我的任务是对成功消息总量的-30%的变化做出反应。因此，如果我在今天上午8点进行检查，我应该将今天从午夜到上午8点的总计数与昨天的相同间隔进行比较。

我尝试创建一个日期直方图聚合，但我希望将diff百分比作为查询结果，而不是在开发方面进行计算。

{
"size": 0, 
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"histo": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "1h"
}
}
}
}

你知道如何做到这一点吗？

Answer 1

您可以利用derivative管道聚合来实现您所期望的：

POST /sales/_search
{
"size": 0, 
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"histo": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "1h"
},
"aggs": {
"successDiff": {
"derivative": {
"buckets_path": "_count"
}
}
}
}
}
}

在每个bucket中，您将获得上一个bucket中的文档计数与当前bucket中文档计数之间的差异。

Answer 2

最终放弃了date_histogram聚合并使用了date_range聚合。使用它要容易得多，即使它与昨天同期相比没有返回差异。我用代码做到了。

{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"ranged_documents": {
"date_range": {
"field": "@timestamp",
"ranges": [
{
"key": "yesterday",
"from": "now-1d/d",
"to": "now-24h/h"
},
{
"key": "today",
"from": "now/d",
"to": "now/h"
}
],
"keyed": true
}
}
}
}

此查询将产生与下面的结果类似的结果

{
"_shards": {
"total": 42,
"failed": 0,
"successful": 42,
"skipped": 0
},
"hits": {
"hits": [],
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": null
},
"took": 134,
"timed_out": false,
"aggregations": {
"ranged_documents": {
"buckets": {
"yesterday": {
"from_as_string": "2020-10-12T00:00:00.000Z",
"doc_count": 268300,
"to_as_string": "2020-10-12T12:00:00.000Z",
"from": 1602460800000,
"to": 1602504000000
},
"today": {
"from_as_string": "2020-10-13T00:00:00.000Z",
"doc_count": 251768,
"to_as_string": "2020-10-13T12:00:00.000Z",
"from": 1602547200000,
"to": 1602590400000
}
}
}
}
}