我从fluentd pod中得到了这个错误,它们一直在重新启动。我在kuberentes v1.17-eks-4c6976上运行这个。
不知道是什么原因。如果有任何帮助,我们将不胜感激。
/usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin_helper/http_server/compat/webrick_handler.rb:26: warning: The called method `build' is defined here
2020-11-23 18:02:08 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-11-23 18:02:09.126315296 +0000 chunk="5b4c9fd811e8162eb94f03d8cec677e5" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch-master", :port=>9200, :scheme=>"http", :path=>""}): read timeout reached"
2020-11-23 18:02:08.126340601 +0000 fluent.warn: {"retry_time":0,"next_retry_seconds":"2020-11-23 18:02:09.126315296 +0000","chunk":"5b4c9fd811e8162eb94f03d8cec677e5","error":"#<Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure: could not push logs to Elasticsearch cluster ({:host=>"elasticsearch-master", :port=>9200, :scheme=>"http", :path=>""}): read timeout reached>","message":"[elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-11-23 18:02:09.126315296 +0000 chunk="5b4c9fd811e8162eb94f03d8cec677e5" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached""}
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:1055:in `rescue in send_bulk'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:1017:in `send_bulk'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:842:in `block in write'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:841:in `each'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:841:in `write'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1136:in `try_flush'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1442:in `flush_thread_run'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-11-23 18:02:08 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000 chunk="5b4c9fd80c4e40f1d7a4a799916ae12b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch-master", :port=>9200, :scheme=>"http", :path=>""}): read timeout reached"
2020-11-23 18:02:08.127449054 +0000 fluent.warn: {"retry_time":1,"next_retry_seconds":"2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000","chunk":"5b4c9fd80c4e40f1d7a4a799916ae12b","error":"#<Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure: could not push logs to Elasticsearch cluster ({:host=>"elasticsearch-master", :port=>9200, :scheme=>"http", :path=>""}): read timeout reached>","message":"[elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000 chunk="5b4c9fd80c4e40f1d7a4a799916ae12b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached""}
fluent插件elasticsearch的默认request_timeout
值为5s,当fluentd有大量积压工作要在大量消息中回放回elasticsearch时,该值通常会太短。
因此,您可能希望将fluentd配置中弹性搜索输出的request_timeout
值增加到15s
,甚至更高,比如60s
。指定时间单位(如s
(也很重要,而不仅仅是指定60
的值。
该设置的文档可以在此处查看:https://github.com/uken/fluent-plugin-elasticsearch#request_timeout
这也可能表明您的弹性搜索节点/集群无法足够快地获取数据。