kubelet nodelost

Bertha 。 2022-05-31 02:18 149阅读 0赞

#### 问题现状 ####

docker 的日志

-- Logs begin at Sat 2018-02-03 12:35:25 CST, end at Sun 2018-02-04 11:46:01 CST. --
    Feb 03 12:35:33 slave-20-62 systemd[1]: Starting Docker Application Container Engine...
    Feb 03 12:35:33 slave-20-62 dockerd[837]: time="2018-02-03T12:35:33.667743244+08:00" level=info msg="libcontainerd: new containerd process, pid: 1982"
    Feb 03 12:35:35 slave-20-62 dockerd[837]: time="2018-02-03T12:35:35.808441707+08:00" level=warning msg="devmapper: Base device already exists and has filesyst
    Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.579529820+08:00" level=info msg="Graph migration to content-addressability took 0.00 secon
    Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.580349391+08:00" level=info msg="Loading containers: start." Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.613936523+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.614235546+08:00" level=warning msg="libcontainerd: failed to retrieve container 5f5f047768 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.615290237+08:00" level=error msg="devmapper: Error unmounting device 3c157393255d6657ac8d0 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.615311282+08:00" level=error msg="Error unmounting container 5f5f047768e711581d903c25b1cd4 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.624309819+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.624501126+08:00" level=warning msg="libcontainerd: failed to retrieve container eb2785f933 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.743901719+08:00" level=error msg="devmapper: Error unmounting device 45e036367371436343ca3 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.743935926+08:00" level=error msg="Error unmounting container eb2785f93392b61542bdf5bb25fd2 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.751767858+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.752010072+08:00" level=warning msg="libcontainerd: failed to retrieve container 1a7acbc6a9 Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.962211682+08:00" level=error msg="devmapper: Error unmounting device a92c743704e337abc0efc Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.962546456+08:00" level=error msg="Error unmounting container 1a7acbc6a994d726221a44378b34c Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.966324973+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:36 slave-20-62 dockerd[837]: time="2018-02-03T12:35:36.966754177+08:00" level=warning msg="libcontainerd: failed to retrieve container 6e10c9371a Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.060795285+08:00" level=error msg="devmapper: Error unmounting device 6c04339aea2b17e18596e Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.060823849+08:00" level=error msg="Error unmounting container 6e10c9371a4dd74be1144e7a6e8fb Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.064038456+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.066017376+08:00" level=warning msg="libcontainerd: failed to retrieve container 33f909bd83 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.211899306+08:00" level=error msg="devmapper: Error unmounting device df5f132df51d113b61865 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.211939733+08:00" level=error msg="Error unmounting container 33f909bd83099261ac7a994a7eb20 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.222190609+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.222520756+08:00" level=warning msg="libcontainerd: failed to retrieve container b1cd71763f Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.295978930+08:00" level=error msg="devmapper: Error unmounting device 4b760fd85bd0f49b69ea3 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.296205078+08:00" level=error msg="Error unmounting container b1cd71763f45c953e9240430f037a Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.304591170+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.304908446+08:00" level=warning msg="libcontainerd: failed to retrieve container b727e29d0b Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.478698939+08:00" level=error msg="devmapper: Error unmounting device 44ac531faa2be14da9fde Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.478734939+08:00" level=error msg="Error unmounting container b727e29d0be17f1e8e21902858057 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.482136060+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.482559633+08:00" level=warning msg="libcontainerd: failed to retrieve container a2d8e654ad Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.664304273+08:00" level=error msg="devmapper: Error unmounting device 095c7f481ba0dadb72ed3 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.664569691+08:00" level=error msg="Error unmounting container a2d8e654ad40a56cfcbda10011d7d Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.673821329+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.674159966+08:00" level=warning msg="libcontainerd: failed to retrieve container 60b9b41b6c Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.674242394+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.835724701+08:00" level=error msg="devmapper: Error unmounting device 0d18053ea087719999511 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.835960408+08:00" level=error msg="Error unmounting container 60b9b41b6c5f23745ad26a27fa8a0 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.840234257+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.840583993+08:00" level=warning msg="libcontainerd: failed to retrieve container 7ad45ab716 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.840671041+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.987328086+08:00" level=error msg="devmapper: Error unmounting device bfc796c01e58a0c7076af Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.987429236+08:00" level=error msg="Error unmounting container 7ad45ab716a109225c5c966e8a234 Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.995543646+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.995992438+08:00" level=warning msg="libcontainerd: failed to retrieve container c406fbb51e Feb 03 12:35:37 slave-20-62 dockerd[837]: time="2018-02-03T12:35:37.996246603+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.117153705+08:00" level=error msg="devmapper: Error unmounting device 6101df46b7e1b3af0dc88 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.117436009+08:00" level=error msg="Error unmounting container c406fbb51e78e75110a6701ec2dc4 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.125618314+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.126037475+08:00" level=warning msg="libcontainerd: failed to retrieve container f4b5c22c89 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.126157946+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.269680508+08:00" level=error msg="devmapper: Error unmounting device e2e28ece0d0e36ad8123d Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.269890540+08:00" level=error msg="Error unmounting container f4b5c22c89784674470004b6db262 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.273251029+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.273825823+08:00" level=warning msg="libcontainerd: failed to retrieve container 608772a9ba Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.273889493+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.366186815+08:00" level=error msg="devmapper: Error unmounting device ccb8523df7df392151ed4 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.366374274+08:00" level=error msg="Error unmounting container 608772a9ba9cd77ac730b5d21eb48 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.374630764+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.374898960+08:00" level=warning msg="libcontainerd: failed to retrieve container e96c5a189b Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.375038211+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.483771523+08:00" level=error msg="devmapper: Error unmounting device 157410846a637aa07c1d9 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.483965541+08:00" level=error msg="Error unmounting container e96c5a189bafac43f2317aacd2658 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.487746253+08:00" level=warning msg="libcontainerd: client is out of sync, restore was call Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.488003795+08:00" level=warning msg="libcontainerd: failed to retrieve container 659080b24d Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.488085477+08:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/l Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.593339467+08:00" level=error msg="devmapper: Error unmounting device 4ac407c550a09770a775a Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.593700188+08:00" level=error msg="Error unmounting container 659080b24d5613023c1fa1e323897 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.781501897+08:00" level=info msg="Firewalld running: false" Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.911184552+08:00" level=info msg="Removing stale sandbox 57b078805b449187911b7d45e9b822eb73 Feb 03 12:35:38 slave-20-62 dockerd[837]: time="2018-02-03T12:35:38.952440423+08:00" level=info msg="Removing stale sandbox 7e1394f258ac5c989b4a08b0e7d86745bb Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.004278832+08:00" level=info msg="Removing stale sandbox 9cf0744b408160aa5fc22357dce58946f3 Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.060982934+08:00" level=info msg="Removing stale sandbox bd25d83c7d67b3d76358a4cc4d0698874c Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.117818833+08:00" level=info msg="Removing stale sandbox c769a6835a26b44ec111a5132e0f27e7ce Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.187430818+08:00" level=info msg="Removing stale sandbox df567145a1d1e68328039f2dcdb4b92447 Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.246458145+08:00" level=info msg="Removing stale sandbox 38ac4ac57c5a5acb0fdb29007d4937c9bd Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.270332596+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 1 Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.335535793+08:00" level=info msg="Loading containers: done." Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.516202247+08:00" level=info msg="Daemon has completed initialization" Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.516249120+08:00" level=info msg="Docker daemon" commit=092cba3 graphdriver=devicemapper ve Feb 03 12:35:39 slave-20-62 dockerd[837]: time="2018-02-03T12:35:39.526370442+08:00" level=info msg="API listen on /var/run/docker.sock" Feb 03 12:35:39 slave-20-62 systemd[1]: Started Docker Application Container Engine.

kubernetes的日志

[root@master ~]# kubectl log  calico-node-gcfcf calico-node -n kube-system
    W0204 11:44:59.640560   12494 cmd.go:392] log is DEPRECATED and will be removed in a future version. Use logs instead.
    Error from server: Get https://slave-20-62:10250/containerLogs/kube-system/calico-node-gcfcf/calico-node: dial tcp 10.39.20.62:10250: getsockopt: connection refused

systemctl 的日志

[root@slave-20-62 ~]# systemctl status kubelet
    ● kubelet.service - kubernetes Kubelet
       Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: disabled)
       Active: inactive (dead)
         Docs: https://github.com/GoogleCloudPlatform/kubernetes

#### 解决方案 ####

目前是重启kubelet

systemctl restart kubelet