failed to create kubelet: misconfiguration: kubelet cgroup driver: “systemd“ is different from docke

不念不忘少年蓝@ 2022-12-28 01:38 69阅读 0赞

在k8s集群中调度pod时，发现调度到其中一个节点的pod都是pending状态，怀疑该节点有问题。通过kubectl get node -o wide发现该节点为notReady状态。  
在该问题节点上执行命令`journalctl -f -u kubelet`查看日志状态

[root@k8s-node02 docker.service.d]# journalctl -f -u kubelet
    -- Logs begin at 二 2020-09-01 00:02:17 CST. --
    12月 13 12:31:40 k8s-node02 kubelet[22475]: I1213 12:31:40.943056   22475 client.go:75] Connecting to docker on unix:///var/run/docker.sock
    12月 13 12:31:40 k8s-node02 kubelet[22475]: I1213 12:31:40.943098   22475 client.go:104] Start docker client with request timeout=2m0s
    12月 13 12:31:40 k8s-node02 kubelet[22475]: W1213 12:31:40.958727   22475 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
    12月 13 12:31:40 k8s-node02 kubelet[22475]: I1213 12:31:40.958757   22475 docker_service.go:238] Hairpin mode set to "hairpin-veth"
    12月 13 12:31:40 k8s-node02 kubelet[22475]: I1213 12:31:40.960800   22475 docker_service.go:253] Docker cri networking managed by cni
    12月 13 12:31:40 k8s-node02 kubelet[22475]: I1213 12:31:40.970314   22475 docker_service.go:258] Docker Info: &{ ID:LQ2Y:BCW5:SPKQ:67DS:VLFK:6KRX:ZYIN:2EB6:MWXS:3VZM:YONB:R3JK Containers:31 ContainersRunning:15 ContainersPaused:0 ContainersStopped:16 Images:23 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{ Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:36 SystemTime:2020-12-13T12:31:40.96153653+08:00 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.4.224-1.el7.elrepo.x86_64 OperatingSystem:CentOS Linux 7 (Core) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0007040e0 NCPU:2 MemTotal:2079207424 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:k8s-node02 Labels:[] ExperimentalBuild:false ServerVersion:19.03.9 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{ Path:runc Args:[]}] DefaultRuntime:runc Swarm:{ NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ ID:7ad184331fa3e55e52b890ea95e65ba581ae3429 Expected:7ad184331fa3e55e52b890ea95e65ba581ae3429} RuncCommit:{ ID:dc9208a3303feef5b3839f4323d9beb36df0a9dd Expected:dc9208a3303feef5b3839f4323d9beb36df0a9dd} InitCommit:{ ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:[]}
    12月 13 12:31:40 k8s-node02 kubelet[22475]: F1213 12:31:40.970424   22475 server.go:273] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
    12月 13 12:31:40 k8s-node02 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
    12月 13 12:31:40 k8s-node02 systemd[1]: Unit kubelet.service entered failed state.
    12月 13 12:31:40 k8s-node02 systemd[1]: kubelet.service failed.

其中错误原因描述：`failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"`，发现设置的docker设置cgroup名字与kubelete的不一致，修改`/etc/docker/daemon.json`配置如下

{ 
            "exec-opts": ["native.cgroupdriver=systemd"],
            "log-driver": "json-file",
            "log-opts": { 
                    "max-size": "100m"
            }
    }

再用`journalctl -f -u kubelet`命令查看节点状态，发现节点正常，Ready状态。