Filebeat初始化失败，出现10.96.0.1:443 i/o超时错误

Question

在我的k8s集群中，节点重新启动后，filebeat连接失败。其他k8s节点工作正常。

来自filebeat pod的日志：

2020-08-30T03:18:58.770Z    ERROR   kubernetes/util.go:90   kubernetes: Querying for pod failed with error: performing request: Get https://10.96.0.1:443/api/v1/namespaces/monitoring/pods/filebeat-gfg5l: dial tcp 10.96.0.1:443: i/o timeout
2020-08-30T03:18:58.770Z    INFO    kubernetes/watcher.go:180   kubernetes: Performing a resource sync for *v1.PodList
2020-08-30T03:19:28.771Z    ERROR   kubernetes/watcher.go:183   kubernetes: Performing a resource sync err performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout for *v1.PodList
2020-08-30T03:19:28.771Z    INFO    instance/beat.go:357    filebeat stopped.
2020-08-30T03:19:28.771Z    ERROR   instance/beat.go:800    Exiting: error initializing publisher: error initializing processors: performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
Exiting: error initializing publisher: error initializing processors: performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

出现错误并重复启动pod。我也重新启动了这个节点，但它没有工作。

filebeat版本为6.5.2，使用守护程序集进行部署。是否存在类似的已知问题？

除filebeat外，所有pod在该节点上的工作都没有问题。

更新：

apiVersion: v1
data:
filebeat.yml: |-
filebeat.inputs:
- type: docker
multiline.pattern: '^[[:space:]]+'
multiline.negate: false
multiline.match: after
symlinks: true
cri.parse_flags: true
containers:
ids: [""]
path: "/var/log/containers"
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 1
target: message_json
overwrite_keys: false
when:
contains:
source: "/var/log/containers/app"
- add_kubernetes_metadata:
in_cluster: true
default_matchers.enabled: false
matchers:
- logs_path:
logs_path: /var/log/containers/
output:
logstash:
hosts:
- logstash:5044
kind: ConfigMap
metadata:
creationTimestamp: "2020-01-06T09:31:31Z"
labels:
k8s-app: filebeat
name: filebeat-config
namespace: monitoring
resourceVersion: "6797684985"
selfLink: /api/v1/namespaces/monitoring/configmaps/filebeat-config
uid: 52d86bbb-3067-11ea-89c6-246e96da5c9c

Answer 1

add_kubernetes_metadata查询https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0失败。正如上面的讨论所证明的那样，这可以通过重新启动Beat来解决临时网络接口问题。