无法参数化输出Logstash索引- AWS s3输入

Question

早上好

我们正在尝试使用Logstash从AWS s3收集日志。下面是Logstash.conf文件:

input {
tcp {
port => 5000
codec => json
}
s3 {
# Keys for user in prod env
id => "prod" 
access_key_id => "key1" 
secret_access_key => "secretkey1"
region => "eu-central-1"
bucket => "buket1"
prefix => "prefix1/"
include_object_properties => true
}
s3 {
# Keys for user in preprod env
id => "preprod"
access_key_id => "key2" 
secret_access_key => "secretkey2"
region => "eu-central-1"
bucket => "bucket2"
prefix => "prefix2/"
include_object_properties => true
}

}
filter {
# Matching prod logs
if [id] == "prod" {
mutate {
add_field => { "env" => "prod" }
}
}
else if [id] == "preprod" {
mutate {
add_field => { "env" => "preprod" }
}
}
}
output {
opensearch {
hosts       => "https://osb-platform"
user => "username"
password => "password"
index       => "4s-%{[env]}-logs-%{+YYYY.MM.dd}"
ecs_compatibility => disabled
ssl_certificate_verification => false
}
}

如您所见，我有来自两个不同环境的两个输入，我想将来自不同环境的日志放在不同的索引上。这是通过对输入的id进行过滤，根据环境添加一个新字段来完成的。

日志被正确收集，但是索引没有被区分:从OpenSearch中我可以看到一个单独的索引，叫做4s-%{[env]}- Logs -2022.05.30，就像它不能解析env变量。

你能帮帮我吗?

谢谢,

卢卡

Answer 1

任何输入中的id选项用于在监视日志存储时帮助识别输入，它不用于过滤，因为它不存在于文档中。

你需要使用标签或类型选项来完成。

例如，使用type:

s3 {
# Keys for user in prod env
type => "prod" 
access_key_id => "key1" 
secret_access_key => "secretkey1"
region => "eu-central-1"
bucket => "buket1"
prefix => "prefix1/"
include_object_properties => true
}

那么你的过滤器将是:

filter {
if [type] == "prod" {
mutate {
add_field => { "env" => "prod" }
}
} else if [type] == "preprod" {
mutate {
add_field => { "env" => "preprod" }
}
}
}

如果你想使用tags，你需要这样做:

s3 {
# Keys for user in prod env
tags => ["prod"] 
access_key_id => "key1" 
secret_access_key => "secretkey1"
region => "eu-central-1"
bucket => "buket1"
prefix => "prefix1/"
include_object_properties => true
}

filter {
if "prod" in [tags] {
mutate {
add_field => { "env" => "prod" }
}
} else if "preprod" in [tags] {
mutate {
add_field => { "env" => "preprod" }
}
}
}