如何在K8s上为Elasticsearch部署应用配置



我在更改部署在K8s中的Elasticsearch配置时遇到了问题。我想将此配置应用于我的Elasticsearch节点

# Force all memory to be locked, forcing the JVM to never swap
bootstrap.mlockall: true
## Threadpool Settings ##
# Search pool
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100
# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300
# Index pool
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 100
# Indices settings
indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb
# Cache Sizes
indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire: 6h
indices.cache.filter.size: 15%
indices.cache.filter.expire: 6h
# Indexing Settings for Writes
index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

要在K8s中部署Elasticsearch,我需要执行两个步骤:步骤1:创建.yml文件,就像这个

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: esc-lamtest
namespace: lamtest
spec:
version: 7.9.0
nodeSets:
- name: basic-1
count: 3
config:
node.master: true
node.data: true
node.ingest: true
# Search pool
thread_pool.search.queue_size: 50
thread_pool.search.size: 20
thread_pool.search.min_queue_size: 10
thread_pool.search.max_queue_size: 100  
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 150Gi
storageClassName: vnptit-nfs
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms4g -Xmx4g
resources:
requests:
memory: 0Gi
cpu: 0Gi
limits:
memory: 0Gi
cpu: 0Gi                

http:tls:自签名证书:禁用:真正的

步骤2:我运行命令行:

kubectl apply -f lamtest.yaml

但我只能为";搜索池";。当我为# Bulk pool # Index pool # Indices settings # Cache Sizes and # Indexing Settings for Writes应用配置时,我的Elasticsearch失败了,这是我的日志

"Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.queue_size] did   you mean any of [thread_pool.get.queue_size, thread_pool.write.queue_size, thread_pool.analyze.queue_size, thread_pool.search.queue_size, thread_pool.listener.queue_size]?", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.size] did you mean any of [thread_pool.get.size, thread_pool.write.size, thread_pool.analyze.size, thread_pool.search.size]?", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]"] } 
uncaught exception in thread [main] 
java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.type] please check that any  required plugins are installed, or check the breaking changes documentation for removed settings 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) 
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) 
at org.elasticsearch.node.Node.<init>(Node.java:385) 
at org.elasticsearch.node.Node.<init>(Node.java:277) 
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) 
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) 
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) 
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) 
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) 
at org.elasticsearch.cli.Command.main(Command.java:90) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) 

有关完整的错误详细信息,请参阅/usr/share/elastsearch/logs/esc-lamtest.log 上的日志

错误消息非常清楚,您没有为其他线程池定义正确的配置,请仔细注意错误消息的第一行。

"已抑制:java.lang.IllegalArgumentException:未知设置[thread_pool.bulk.queue_size]你的意思是[thread_pool.get_queue_size、thread_pool.write_queue_size],thread_pool.analyze.queue_size、thread_pool.search_queue_size,thread_pool.elistener.queue_size]&";,

您已将大容量队列大小定义为thread_pool.bulk.queue_size,这是不正确的,并且由于您可以在ES中读取线程池,因此bulk没有线程池,相反,它使用write线程池处理来自同一文档的bulk请求,因此将其更改为thread_pool.write.queue_size将适用于此配置

write用于单个文档索引/删除/更新和批量请求。线程池类型是固定的,分配的处理器的大小为#,queue_size为10000。此池的最大大小为1+#分配的处理器。

现在,为了修复bulkIndex和其他设置的此问题,请验证您是否使用了正确的配置名称threadpools可以从任何运行的ES实例中获得,并且您可以轻松地构造相应的配置。

http://localhost:9200/_nodes/stats?pretty给出了所有的CCD_ 10,下面列出了一些。

"thread_pool": {
"analyze": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 1
},
"ccr": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
}

最新更新