FluentBit:添加动态es索引

Question

我有一个正在工作的fluent-bit:1.7我需要OUTPUT到Elasticsearch，并基于k8s标签=name创建一个动态索引。我想要索引的以下约定：

infra-${app_name}-yyyy.mm.dd

示例：infra-mongodb-2021.01.01、infra-postgresql-2021.01.01、infra-kafka-2021.01.01等…

这是我的过滤器和输出配置：

[FILTER]
Name             kubernetes
Match            kube.*
Merge_Log        Off
Keep_Log         Off
[OUTPUT]
Name  es
Match kube.*
Host ${ES_HOST}
Logstash_Format On
Logstash_Prefix_Key kubernetes['labels']['name']

但它生成以下索引：mongodb-2021.01.01

差不多完成了，我只需要始终添加infra-前缀。

需要明确的是，我需要这样的东西：

Logstash_Prefix_Key infra-${kubernetes['labels']['name']}

Answer 1

我使用Lua插件在记录中创建一个具有索引名称的字段，然后将该字段用作Logstash_Prefix_Key

Lua脚本(基于https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua)：

function append_es_index(tag, timestamp, record)
new_record = record
if (record["cluster_name"] ~= nil) then
es_index = record["cluster_name"]
else
es_index = "k8s"
end
if (record["kubernetes"] ~= nil) then
kube = record["kubernetes"]
if (kube["namespace_name"] ~= nil and string.len(kube["namespace_name"]) > 0) then
es_index = es_index .. "." .. kube["namespace_name"]
end

if (kube["labels"] ~= nil) then
labels = kube["labels"]
if (labels["app"] ~= nil and string.len(labels["app"]) > 0) then
es_index = es_index .. "." .. labels["app"]
elseif (labels["k8s-app"] ~= nil and string.len(labels["k8s-app"]) > 0) then 
es_index = es_index .. "." .. labels["k8s-app"]
elseif (labels["name"] ~= nil and string.len(labels["name"]) > 0) then 
es_index = es_index .. "." .. labels["name"]
end
end
end
new_record["es_index"] =  es_index
return 1, timestamp, new_record
end 

Fluentbit过滤器配置：

[FILTER]
Name                kubernetes
...
[FILTER]
Name record_modifier
Match *
Record cluster_name my-test-cluster
[FILTER]
Name    lua
Match   *
script  /fluent-bit/scripts/append_es_index.lua
call    append_es_index

Answer 2

实现类似结果的另一种方法是为您的Pods 创建标签或注释

例如

[OUTPUT]
Name                es
Logstash_Prefix_Key kubernetes['labels']['log-key']
...

基于文档的yml部署

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
log-key: infra-nginx # log-key label will be used in Logstash_Prefix_Key
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80

通过这种方式，您将能够组合多个"；动态的"；数据，例如名称空间