使用Promtail的多个条件将日志行拖放到Loki

Question

我想使用来自两个不同 JSON 字段的 AND 条件在 Promtail 中删除行。

我有这样的 JSON 日志行。

{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}  
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET /path HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 2"}

我的本地 Promtail 配置如下所示。

clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: testing-my-job-drop
pipeline_stages:
- match:
selector: '{job="my-job"}'
stages:
- json:
expressions:
http_user_agent:
request:
- drop:
source: "http_user_agent"
expression: "user agent 1"
# I want this to be AND
- drop:
source: "request"
expression: "GET / HTTP/1.1"
drop_counter_reason: my_job_healthchecks
static_configs:
- labels:
job: my-job

像这样使用 Promtail 配置会从我的两个 JSON 字段中使用 OR 删除行。

如何调整我的配置，以便我只删除http_user_agent=user agent 1和request=GET / HTTP/1.1的行？

Answer 1

如果提供多个选项，它们将被视为 AND 子句，其中每个选项必须为 true 才能删除日志。 如果您希望使用 OR 子句删除，请指定多个删除阶段。

https://grafana.com/docs/loki/latest/clients/promtail/stages/drop/#drop-stage

按时间或长度删除日志

将删除所有早于 24 小时或超过 8kb 字节的日志

- json:
expressions:
time:
msg:
- timestamp:
source: time
format: RFC3339
- drop:
older_than: 24h
- drop:
longer_than: 8kb

按正则表达式和长度删除日志

将删除包含单词debug并且长度超过1kb字节的所有日志

- drop:
expression: ".*debug.*"
longer_than: 1kb

Answer 2

clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: testing-my-job-drop
pipeline_stages:
- match:
selector: '{job="my-job"}'
stages:
- json:
expressions:
http_user_agent:
request:
- labels:
http_user_agent:
request:
#### method 1
- match:
selector: '{http_user_agent="user agent 1"}'
stages:
- drop:
source: "request"
expression: "GET / HTTP/1.1"
drop_counter_reason: my_job_healthchecks
## they are both conditions match will drop
#### method 2
- match:
selector: '{http_user_agent="user agent 1",request="GET / HTTP/1.1"}'
action: drop
#### method 3, incase regex pattern.
- match: 
selector: '{http_user_agent="user agent 1"} |~ "(?i).*GET / HTTP/1.1.*"'
action: drop
static_configs:
- labels:
job: my-job

match stage包括match stage。