k8s修改默认存储路径及容器存储空间资源限制ephemeral-storage

迷南。 2022-12-07 12:15 653阅读 0赞

k8s1.8引入的特性，限制容器存储空间的使用；对于容器资源隔离来说，非常有用，万一应用程序失控，写大量日志把node空间写满，影响就大了。

使用很简单，和cpu\\memcache一样，如：

1.  `resources:`
2.  `requests:`
3.  `cpu: 1`
4.  `memory: 2048Mi`
5.  `ephemeral-storage: 2Gi`
6.  `limits:`
7.  `cpu: 2`
8.  `memory: 2048Mi`
9.  `ephemeral-storage: 5Gi`

但，这玩意生效有条件：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NkbWVp_size_16_color_FFFFFF_t_70][]

猛一看，ephemeral-storage只能对镜像存放在“根分区”下的容器有效，也就是默认的"Docker Root Dir: /var/lib/docker"必须在根分区下；对于一个正常点的运维来说，程序路径与根分区分离是基本的做法，对于一个有节操的k8s运维来说，将/var/lib/docker用独立分区，再正常不过了。

测试结果如下：

docker Version: 18.09.8  
k8s version:1.13.8  
Docker Root Dir: /var/lib/docker  
kubelet的--root-dir: 默认（/var/lib/kubelet）

/var/lib/docker在根分区下，ephemeral-storage有效果  
/var/lib/docker不在根分区下（作为单独分区），ephemeral-storage没有效果

这有点沮丧，这么有用的功能难道不能派上用场，不太相信，求助github，有线索：[https://github.com/kubernetes/enhancements/issues/361][https_github.com_kubernetes_enhancements_issues_361]

其中有这样的回复：

The behavior you describe should work regardless of this feature. Make sure you have [\--root-dir][--root-dir] set correctly. Docker reports its root directory to the kubelet, so as long as your images are stored on the same partition that contains `/var/lib/docker` (or whatever your docker root dir is), this should work correctly.

这句话貌似有误，/var/lib/docker应该写错了，`换成``/var/lib/kubelet才好理解，因为/var/lib/kubelet是--root-dir的默认配置，总的来说，意思是只要“Docker Root Dir: /var/lib/docker”和“kubelet --root-dir”在一个分区，就能起作用。`

`测试结果就是如此。`

`/var/lib/docker是独立分区的情况下，怎样实现kubelet的root-dir与/var/lib/docker一个分区呢？两个选择：`

**`方案1. 修改root-dir`**

kubectl drain nodename
    systemctl stop docker
    systemctl stop kubelet
     
    修改/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf：
    增加--root-dir=/var/lib/docker/kubelet/
    将/var/lib/kubelet/修改为/var/lib/docker/kubelet/
     
    修改/etc/kubernetes/kubelet.conf
    将/var/lib/kubelet/修改为/var/lib/docker/kubelet/
     
    mv /var/lib/kubelet /var/lib/docker
     
    systemctl daemon-reload
    systemctl start docker
    systemctl start kubelet

有个遗留问题，重启kubelet后，又自动生产了以下目录，但kubelet运行正常

1.  `# tree /var/lib/kubelet -L 3`
2.  `/var/lib/kubelet`
3.  `└── device-plugins`
4.  `├── DEPRECATION`
5.  `├── kubelet_internal_checkpoint`
6.  `└── kubelet.sock`

**方案2.root-dir软链到/var/lib/docker下**

kubectl drain nodename
    systemctl stop docker
    systemctl stop kubelet
     
    mv /var/lib/kubelet /var/lib/docker
    ln -s /var/lib/kubelet /var/lib/docker/kubelet
     
    systemctl start docker
    systemctl start kubelet
    systemctl uncordon nodename

PS：上述mv操作前，先df确认下是否有/var/lib/kubelet下的文件被mount，有则先umount再mv，否则报错“Device or resource busy”

# mv kubelet/ /var/lib/docker
    mv: cannot remove ‘kubelet/pods/73a3d42a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/kube-proxy-token-jccg4’: Device or resource busy
    mv: cannot remove ‘kubelet/pods/73a36f7a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/etcd-certs’: Device or resource busy
    mv: cannot remove ‘kubelet/pods/73a36f7a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/calico-node-token-tzfv8’: Device or resource busy
    mv: cannot remove ‘kubelet/pods/e2542d86-ceef-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/node-exporter-token-5926x’: Device or resource busy
     
    # df -h
    tmpfs                        20517564       0  20517564   0% /var/lib/kubelet/pods/73a3d42a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/kube-proxy-token-jccg4
    tmpfs                        20517564       0  20517564   0% /var/lib/kubelet/pods/73a36f7a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/etcd-certs
    tmpfs                        20517564       0  20517564   0% /var/lib/kubelet/pods/73a36f7a-b2a5-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/calico-node-token-tzfv8
    tmpfs                        20517564       0  20517564   0% /var/lib/kubelet/pods/e2542d86-ceef-11e9-8e8d-005056b4f9d3/volumes/kubernetes.io~secret/node-exporter-token-5926

测试结果：

在容器中dd生成一个5G的文件，终于可以evicted了。

# kubectl get pods -o wide                          
    NAME                           READY   STATUS    RESTARTS   AGE     IP              NODE                   NOMINATED NODE   READINESS GATES
    ptest-trade-747b894f54-mhrv4   0/1     Evicted   0          3m37s   <none>          lin-40-16-206.lb.com   <none>           <none>
    ptest-trade-747b894f54-tx847   0/1     Running   0          26s     10.46.206.96    lin-40-16-206.lb.com   <none>           <none>
     
     
    # kubectl describe pod p7881-trade-747b894f54-mhrv4 
    Events:
      Warning  Evicted              12s  kubelet, lin-40-16-206.lb.com  Pod ephemeral local storage usage exceeds the total limit of containers 5Gi.
      Warning  ExceededGracePeriod  2s   kubelet, lin-40-16-206.lb.com  Container runtime did not kill the pod within specified grace period.
      Normal   Killing              1s   kubelet, lin-40-16-206.lb.com  Killing container with id docker://ptest-trade:Need to kill Pod

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NkbWVp_size_16_color_FFFFFF_t_70]: /images/20221123/803885c12ccd4df69c5af55a9bd35cb9.png
[https_github.com_kubernetes_enhancements_issues_361]: https://github.com/kubernetes/enhancements/issues/361
[--root-dir]: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go#L367