kubelet not ready 问题排查

- 日理万妓 2022-12-16 14:24 145阅读 0赞

## 背景 ##

使用kubectl 发现Node不正常。然后查看Node的日志出现如下

Warning  ImageGCFailed            53s    kubelet, 192.168.60.31  failed to get imageFs info: unable to find data in memory cache
      Normal   NodeHasSufficientMemory  53s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientMemory
      Normal   NodeHasNoDiskPressure    53s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasNoDiskPressure
      Normal   NodeHasSufficientPID     53s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientPID
      Normal   Starting                 53s    kubelet, 192.168.60.31  Starting kubelet.
      Normal   NodeHasSufficientMemory  40s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientMemory
      Normal   Starting                 40s    kubelet, 192.168.60.31  Starting kubelet.
      Warning  ImageGCFailed            40s    kubelet, 192.168.60.31  failed to get imageFs info: unable to find data in memory cache
      Normal   Starting                 28s    kubelet, 192.168.60.31  Starting kubelet.
      Warning  ImageGCFailed            28s    kubelet, 192.168.60.31  failed to get imageFs info: unable to find data in memory cache
      Normal   NodeHasSufficientMemory  28s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientMemory
      Normal   NodeHasNoDiskPressure    28s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasNoDiskPressure
      Normal   NodeHasSufficientPID     28s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientPID
      Warning  ImageGCFailed            15s    kubelet, 192.168.60.31  failed to get imageFs info: unable to find data in memory cache
      Normal   NodeHasSufficientPID     15s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientPID
      Normal   NodeHasSufficientMemory  15s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientMemory
      Normal   NodeHasNoDiskPressure    15s    kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasNoDiskPressure
      Normal   Starting                 15s    kubelet, 192.168.60.31  Starting kubelet.
      Normal   Starting                 3s     kubelet, 192.168.60.31  Starting kubelet.
      Warning  ImageGCFailed            3s     kubelet, 192.168.60.31  failed to get imageFs info: unable to find data in memory cache
      Normal   NodeHasSufficientMemory  2s     kubelet, 192.168.60.31  Node 192.168.60.31 status is now: NodeHasSufficientMemory

## 处理方法 ##

这个问题明显是Docker ImageGC 的问题。应该是内存不够了。导致的kubelet挂掉了。但是查看df -h 。磁盘空间还是够的。

journalct 查看kubelet日志  
journalctl -u kubelet | grep -i garbage

解决方案。先让docker自己清理一下镜像文件。然后重启kubelet和docker

docker system prune
    systemctl stop kubelet
    systemctl stop docker
    systemctl start docker
    systemctl start kubelet

## 参考链接 ##

https://stackoverflow.com/questions/62020493/kubernetes-1-18-warning-imagegcfailed-error-failed-to-get-imagefs-info-unable-t