小贝子编程

elasticsearch pod由于节点组down和up而处于pending状态



描述bug:我已经在我的EKS集群中部署了PVC的elasticsearch，并且工作得很好，没有问题，我们尝试缩小节点和扩大规模，开始面临问题，pod处于待处理状态。

复制步骤:

1.helm repo add elastic https://helm.elastic.co
2.with Helm 3: helm install elasticsearch elastic/elasticsearch
3. git clone git@github.com:elastic/helm-charts.git
4. went to elastcsearch working directory and run below
5. kubectl create ns elastic
6. helm install elasticsearch . -n elastic

预期行为:

集群将按预期运行，没有问题，并尝试扩展

kubectl scale --replicas=0 sts/elasticsearch-master -n elastic
kubectl scale --replicas=3 sts/elasticsearch-master -n elastic

现在我的pod是挂起状态和卡住

kubectl logs pod/elasticsearch-master-0 -n elastic
{"type": "server", "timestamp": "2022-05-09T01:03:18,156Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.17.3], pid[7], build[default/docker/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Linux/5.4.181-99.354.amzn2.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,159Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,160Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8450783023809497710, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [repository-s3]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,547Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ml", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,629Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.data", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,644Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdck)]], net usable_space [9.6gb], net total_space [9.7gb], types [ext4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,645Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,725Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.master", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,729Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.remote_cluster_client", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,730Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ingest", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,846Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [VeyMV5ZNRMW3UyW8taNGkQ], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,848Z", "level": "CRITICAL", "component": "o.e.d.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]", "key": "legacy role settings", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:33,947Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/180] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-05-09T01:03:35,946Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,230Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,232Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-8450783023809497710/geoip-databases/VeyMV5ZNRMW3UyW8taNGkQ]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,421Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,455Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,556Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-05-09T01:03:39,663Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,582Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
(base) devops@devops-XPS-15-9570:~/Desktop/Repo$ kubectl logs pod/elasticsearch-master-0 -n elastic
{"type": "server", "timestamp": "2022-05-09T01:03:18,156Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.17.3], pid[7], build[default/docker/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Linux/5.4.181-99.354.amzn2.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,159Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,160Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8450783023809497710, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [repository-s3]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,547Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ml", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,629Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.data", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,644Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdck)]], net usable_space [9.6gb], net total_space [9.7gb], types [ext4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,645Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,725Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.master", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,729Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.remote_cluster_client", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,730Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ingest", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,846Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [VeyMV5ZNRMW3UyW8taNGkQ], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,848Z", "level": "CRITICAL", "component": "o.e.d.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]", "key": "legacy role settings", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:33,947Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/180] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-05-09T01:03:35,946Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,230Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,232Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-8450783023809497710/geoip-databases/VeyMV5ZNRMW3UyW8taNGkQ]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,421Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,455Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,556Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-05-09T01:03:39,663Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "starting ..." }
{"type": "server", "timestamp": "2022-05-09T01:03:41,260Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,261Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "deprecation component started" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,458Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {192.168.131.86:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,582Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,636Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "cluster UUID [miG-kHiRS02W-Tf8F-Nqjg]" }
{"type": "server", "timestamp": "2022-05-09T01:03:52,646Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:02,648Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,649Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,650Z", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "timed out while waiting for initial discovery state - timeout: 30s" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,661Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {192.168.131.86:9200}, bound_addresses {0.0.0.0:9200}" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,661Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "started" }
{"type": "server", "timestamp": "2022-05-09T01:04:14,174Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }
{"type": "server", "timestamp": "2022-05-09T01:04:22,650Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:24,129Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }
{"type": "server", "timestamp": "2022-05-09T01:04:32,651Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:34,136Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }

[Github link issue] (https://github.com/elastic/helm-charts/issues/1654)

不能直接缩小主节点。要缩小Elasticsearch主节点，您需要首先将节点排除在投票之外。否则，pod将查找旧的主节点(即。

假设你有一个SatefulSet主节点，有3个副本(节点名称:elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2)，你想把它缩小到1个副本。

  • 步骤1:排除elasticsearch-master-1和elasticsearch-master-2投票:

    $ curl -X POST "localhost:9200/_cluster/voting_config_exclusions?node_names=elasticsearch-master-1&pretty"
$ curl -X POST "localhost:9200/_cluster/voting_config_exclusions?node_names=elasticsearch-master-2&pretty"

  • 步骤2:现在缩小副本:

    $ kubectl scale --replicas=1 sts/elasticsearch-master -n elastic

  • 步骤3:删除排斥性，这样一旦您再次扩展，它们就可以自动连接:

    $ curl -X DELETE "localhost:9200/_cluster/voting_config_exclusions?wait_for_removal=false&pretty"

裁判:删除主节点

注意:对于密码保护的Elasticsearch，您需要通过curl命令提供用户名和密码。如$ curl -u 'username:password' ....

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium