kubelet.service:单元在未就绪状态节点错误中从 kubernetes 集群进入失败状态

Question

我正在尝试在具有 1 个主节点和 2 个工作节点的 kubernetes 集群中部署一个 springboot 微服务。当我尝试使用命令获取节点状态时sudo kubectl get nodes，我得到的工作节点之一尚未准备就绪。它显示"未就绪"状态。

当我申请对以下命令进行故障排除时，

sudo journalctl -u kubelet

我得到的响应就像kubelet.service: Unit entered failed state和 kubelet 服务停止一样。以下是我在应用命令时得到的响应sudo journalctl -u kubelet.

-- Logs begin at Fri 2020-01-03 04:56:18 EST, end at Fri 2020-01-03 05:32:47 EST. --
Jan 03 04:56:25 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi
Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.053962     970 server.go:416] Version: v1.17.0
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.084061     970 plugins.go:100] No cloud provider specified.
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.235928     970 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.280173     970 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curre
Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: I0103 04:56:38.107966     970 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: F0103 04:56:38.109401     970 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable swa
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.901632    1433 server.go:416] Version: v1.17.0
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907654    1433 plugins.go:100] No cloud provider specified.
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907806    1433 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.947107    1433 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr
Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: I0103 04:56:49.263777    1433 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to
Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: F0103 04:56:49.264219    1433 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.712729    1500 server.go:416] Version: v1.17.0
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.714927    1500 plugins.go:100] No cloud provider specified.
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.715248    1500 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.763508    1500 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.956706    1500 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: F0103 04:56:59.957078    1500 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.

日志文件：服务：设备进入故障状态

我尝试重新启动库贝莱特。但节点状态仍然没有变化。仅未就绪状态。

更新

当我尝试命令systemctl list-units --type=swap --state=active时，我得到以下响应，

docker@MILDEVKUB040:~$ systemctl list-units --type=swap --state=active
UNIT                                            LOAD   ACTIVE SUB    DESCRIPTION
dev-mapper-MILDEVDCR01x2dx2dvgx2dswap_1.swap loaded active active /dev/mapper/MILDEVDCR01--vg-swap_1

重要

当我遇到节点未准备好的此类问题时，每次我都需要禁用交换并需要重新加载守护程序和 kubelet。在该节点变为就绪状态后。我需要再次重复同样的事情。

我怎样才能找到一个永久的解决方案？

Answer 1

failed to run Kubelet: running with swap on is not supported, please disable swap

您需要在系统上禁用交换才能使 kubelet 工作。您可以禁用与sudo swapoff -a交换

对于基于 systemd 的系统，还有另一种使用交换单元启用交换分区的方法，即使您已使用swapoff -a关闭了交换，只要systemd重新加载，也会启用

https://www.freedesktop.org/software/systemd/man/systemd.swap.html

使用systemctl list-units --type=swap --state=active检查您是否有任何交换单位

您可以使用systemctl mask <unit name>永久禁用任何活动的交换单元。

注意：请勿使用systemctl disable <unit name>禁用交换单元，因为当 systemd 重新加载时，交换单元将再次激活。仅使用systemctl mask <unit name>。

要确保在系统由于电源循环或任何其他原因重新启动时不会重新启用交换，请删除或注释掉/etc/fstab中的交换条目

总结：

1. 运行sudo swapoff -a

2. 检查您是否有带有命令systemctl list-units --type=swap --state=active的交换单元。如果有任何活动的交换单位，请使用systemctl mask <unit name>屏蔽它们

3. 删除/etc/fstab中的交换条目

Answer 2

根本原因是交换空间。要完全禁用，请执行以下步骤：

• 运行swapoff -a：这将立即禁用交换，但会在重新启动时激活
• 从/etc/fstab中删除任何交换条目 重新引导系统。

如果掉期消失了，很好。如果由于某种原因它仍然在这里，你 必须删除交换分区。重复步骤 1 和 2，然后 使用fdisk或parted删除(现在未使用的(交换分区。 在这里要格外小心：删除错误的分区将带来灾难性的后果 影响！

• reboot

这应该可以解决您的问题。

Answer 3

删除/etc/fstab会产生 vm 错误，我认为我们应该找到另一种解决此问题的方法。我试图删除fstab，所有命令(安装，ping和其他命令(错误。