NotReady节点带有ContainerGCFailed警告

Question

参见以下事件:

Events:
Type     Reason             Age                   From     Message
----     ------             ----                  ----     -------
Warning  ContainerGCFailed  58s (x1775 over 30h)  kubelet  rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16797216 vs. 16777216)

和in条件:

Conditions:
Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
----             ------  -----------------                 ------------------                ------                       -------
MemoryPressure   False   Sat, 19 Nov 2022 17:17:30 -0600   Wed, 16 Nov 2022 22:28:31 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
DiskPressure     False   Sat, 19 Nov 2022 17:17:30 -0600   Wed, 16 Nov 2022 22:28:31 -0600   KubeletHasNoDiskPressure     kubelet has no disk pressure
PIDPressure      False   Sat, 19 Nov 2022 17:17:30 -0600   Wed, 16 Nov 2022 22:28:31 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
Ready            False   Sat, 19 Nov 2022 17:17:30 -0600   Fri, 18 Nov 2022 11:03:06 -0600   KubeletNotReady              PLEG is not healthy: pleg was last seen active 30h17m27.791101751s ago; threshold is 3m0s

如何解释这些信息?原因是什么呢?

Answer 1

信息比较明显

如何解读这些信息?

Kubernetes使用Garbage collection清理集群资源。kubelet每五分钟对未使用的映像和每分钟对未使用的容器执行垃圾收集。原因"ContainerGCFailed";表示它在GC进程中失败。

原因是什么?

Kubelet为gRPC消息设置的限制是16MB。当您有很多容器(可能是死的)时，gRPC消息的大小超过了它，并且kubelet接收rpc error。

可能的解决方案:

删除那些旧的死容器并将--maximum-dead-containers=1000添加到Kubelet中以解决问题。