logstash输出序列号为的弹性搜索索引

Question

我使用的是带有Logstash 7.10的AWS Elastic Search(7.10版(。其目的是将logstash中的内容发送到弹性搜索，并使用策略在特定大小或时间后滚动索引。

policy: {
"policy_id": "Rollover_Policy",
"description": "roller index",
"last_updated_time": 1634910129219,
"schema_version": 1,
"error_notification": null,
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_size": "1mb"
}
}
],
"transitions": [
{
"state_name": "warm"
}
]
},
{
"name": "warm",
"actions": [
{
"replica_count": {
"number_of_replicas": 1
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "1h"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"products*"
],
"priority": 100,
"last_updated_time": 1634910129219
}
]
}

当我试图在logstash输出插件中将ilm_enabled设置为true时，它无法连接到弹性搜索xpack API。

注意：在AWS弹性搜索中不支持xpack和ILM。

elasticsearch {  
hosts => "${elasticsearch_endpoint}"
user => "${elasticsearch_user}"
password => "${elasticsearch_password}"
ilm_enabled => true
ilm_rollover_alias => "products"
ilm_pattern => "{now/d}-000001"
ilm_policy => "Rollover_Policy"
}

因此，我已将ilm_enabled标志更改为false，并尝试使用以下选项

elasticsearch {
hosts => "${elasticsearch_endpoint}"
user => "${elasticsearch_user}"
password => "${elasticsearch_password}"
ilm_enabled => false
index => "products-%{+YYYY.MM.dd}-000001"
}

现在的问题是，即使在滚动之后，logstash仍然将文档发送到001索引，而不是新索引。如果我不在索引名称中给出-000001，则滚动将失败。

Answer 1

使用弹性中的以下REST请求创建索引。由于索引名称具有日期模式，因此滚动将创建具有当前日期的新索引。

PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E
{
"settings":{
"number_of_shards":1,
"number_of_replicas":1
},
"aliases": {
"products":  {
"is_write_index": true
}
}

创建索引模式模板以及滚动别名

PUT _index_template/products_logs
{
"index_patterns": [
"products*"
],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"opendistro": {
"index_state_management": {
"rollover_alias": "products"
}
}
}
}
}

在logstash输出插件中，给出以下详细信息以将数据发送到弹性搜索

elasticsearch {  
hosts => "${elasticsearch_endpoint}"
user => "${elasticsearch_user}"
password => "${elasticsearch_password}"
ilm_enabled => false 
index => "products"
}

注意：索引名称表示索引的别名。