k8s node notReady之kubelet cgroup driver: "cgroupfs" is different from docker

曾经终败给现在 2023-07-16 14:58 10阅读 0赞

# 问题 #

worker节点运行正常，因为安装其他软件我就重启了一下，结果发现kubectl get nodes显示node1 notReady，这时执行kubectl describe node node1发现node1上的kubelet没有正常运行，  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70]

我手动执行systemctl start kubelet.service没有报错，但是当我执行systemctl statuskubelet.service发现服务失败了。

执行journalctl -xeu kubelet  
发现具体的错误信息：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 1]  
关键信息就是：

Mar 19 03:11:12 node1 kubelet[125980]: F0319 03:11:12.805339  125980 server.go:273] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

# 解决办法 #

临时将/etc/docker/daemon.json中的对应参数修改为cgroupfs

[root@node1 ~]# cat /etc/docker/daemon.json
    {
      "exec-opts": ["native.cgroupdriver=cgroupfs"],
      "registry-mirrors": ["http://hub-mirror.c.163.com"]
    }
    [root@node1 ~]#

重启docker与kubelet  
systemctl restart docker  
systemctl status kubelet.service  
现在kubelet正常运行了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 2]

# 分析 #

我在另外一台worker节点执行ps -ef|grep kubelet发现两个节点启动参数竟然不同， 我当时安装时没有做什么特殊配置。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 3]

[root@node1 ~]#  systemctl status  kubelet.service
    ● kubelet.service - kubelet: The Kubernetes Node Agent
       Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
      Drop-In: /usr/lib/systemd/system/kubelet.service.d
               └─10-kubeadm.conf
       Active: activating (auto-restart) (Result: exit-code) since Thu 2020-03-19 03:12:47 CST; 7s ago
         Docs: https://kubernetes.io/docs/
      Process: 126926 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
     Main PID: 126926 (code=exited, status=255)
    
    Mar 19 03:12:47 node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
    Mar 19 03:12:47 node1 systemd[1]: Unit kubelet.service entered failed state.
    Mar 19 03:12:47 node1 systemd[1]: kubelet.service failed.

根据启动参数$KUBELET\_KUBECONFIG\_ARGS $KUBELET\_CONFIG\_ARGS $KUBELET\_KUBEADM\_ARGS $KUBELET\_EXTRA\_ARGS 逐个分析其中不同

[root@node1 ~]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
    # Note: This dropin only works with kubeadm and kubelet v1.11+
    [Service]
    Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/k                  ubernetes/kubelet.conf"
    Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
    # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS vari                  able dynamically
    EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
    # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should                   use
    # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourc                  ed from this file.
    EnvironmentFile=-/etc/sysconfig/kubelet
    ExecStart=
    ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
    [root@node1 ~]# cat /etc/sysconfig/kubelet
    KUBELET_EXTRA_ARGS=
    [root@node1 ~]# cat /var/lib/kubelet/kubeadm-flags.env
    KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/                  google_containers/pause:3.1"
    [root@node1 ~]#

看到这里我们就直达还另外一种办法，也就是修改/var/lib/kubelet/kubeadm-flags.env文件，将其中的–cgroup-driver=cgroupfs 修改为  
–cgroup-driver=systemd

# 办法二（也是推荐） #

将/var/lib/kubelet/kubeadm-flags.env文件中的–cgroup-driver=cgroupfs 修改为  
–cgroup-driver=systemd

也就是docker配置项和kubelet的一致都是systemd。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70]: /images/20230528/ab686b547eb84e8ca84ec34314294be1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 1]: /images/20230528/f02a8d7293a24f0787a7b39ef98fc825.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 2]: /images/20230528/bf847cafd2614b9d91daf5c5d8c3cbac.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_16_color_FFFFFF_t_70 3]: /images/20230528/c70ae0e416454db4b5896fc81ea45273.png