【kubernetes/k8s概念】k8s 坑问题汇总

矫情吗；* 2022-04-02 09:10 2592阅读 0赞

## 1. Pod始终处于Pending状态 ##

如果Pod保持在`Pending`的状态，意味着无法被正常的调度到节点上。由于某种系统资源无法满足Pod运行的需求

*  **系统没有足够的资源**：已经用尽了集群中所有的CPU或内存资源。需要清理一些不在需要的Pod，调整它们所需的资源量，或者向集群中增加新的节点。
 *  **用户指定了`hostPort`**：通过`hostPort`用户能够将服务暴露到指定的主机端口上，会限制Pod能够被调度运行的节点。

## 2. Pod始终处于Waiting状态 ##

Pod处在`Waiting`的状态，说明已经被调度到了一个工作节点，却无法在那个节点上运行。可以使用kubectl describe 含有更详细的错误信息。最经常导致Pod始终`Waiting`的原因是无法下载镜像

## 3. Pod 处于 CrashLoopBackOff 状态 ##

CrashLoopBackOff 状态说明容器曾经启动了，但又异常退出了。此时 Pod 的 RestartCounts 通常是大于 0

*  容器进程退出
 *  健康检查失败退出
 *  OOMKilled

## 5. Pod一直崩溃或运行不正常 ##

可以使用kubectl describe以及kubectl logs排查问题，但是这个一般也不确定

情况有：健康检测失败，OOM情况，或者容器运行生命周期结束

### 6. 集群雪崩需给Kubelet预留资源 ###

[从一次集群雪崩看Kubelet资源预留的正确姿势 - WaltonWang's Blog - OSCHINA - 中文开源技术交流社区][Kubelet_ - WaltonWang_s Blog - OSCHINA -]

*  `Node Allocatable Resource = Node Capacity - Kube-reserved - system-reserved - eviction-threshold`

> \--eviction-hard=memory.available<1024Mi,nodefs.available<10%,nodefs.inodesFree<5% \\
> 
> \--system-reserved=cpu=0.5,memory=1G \\ --kube-reserved=cpu=0.5,memory=1G \\
> 
> \--kube-reserved-cgroup=/system.slice/kubelet.service \\
> 
> \--system-reserved-cgroup=/system.slice \\
> 
> \--enforce-node-allocatable=pods,kube-reserved,system-reserved \\

## 7. nfs挂载错误wrong fs type, bad option, bad superblock ##

根据错误提示，查看/sbin/mount.<type>文件，果然发现没有/sbin/mount.nfs的文件，安装nfs-utils即可

## 8. kube-apiserver accept4: too many open files ##

http: Accept error: accept tcp 0.0.0.0:6443: accept4: too many open files; retrying in 1s

查看apiserver进程，lsof -p $pid，发现占用65540个，查看cat /proc/$pid/limits发现限制在65536个，查看占用的一大堆10250的某个kubelet，发现如下错误

>      perationExecutor.UnmountVolume started for volume "makepool1-web3" (UniqueName: "kubernetes.io/nfs/7be05590-3a46-11e9-906c-20040fedf0bc-makepool1-web3") pod "7be05590-3a46-11e9-906c-20040fedf0bc" (UID: "7be05590-3a46-11e9-906c-20040fedf0bc")
> 
>     nestedpendingoperations.go:263\] Operation for "\\"kubernetes.io/nfs/7be05590-3a46-11e9-906c-20040fedf0bc-makepool1-web3\\" (\\"7be05590-3a46-11e9-906c-20040fedf0bc\\")" failed. No retries permitted until 2019-03-07 12:31:28.78976568 +0800 CST m=+7328011.532812666 (durationBeforeRetry 2m2s). Error: "UnmountVolume.TearDown failed for volume \\"makepool1-web3\\" (UniqueName: \\"kubernetes.io/nfs/7be05590-3a46-11e9-906c-20040fedf0bc-makepool1-web3\\") pod \\"7be05590-3a46-11e9-906c-20040fedf0bc\\" (UID: \\"7be05590-3a46-11e9-906c-20040fedf0bc\\") : Unmount failed: exit status 16\\nUnmounting arguments: /var/lib/kubelet/pods/7be05590-3a46-11e9-906c-20040fedf0bc/volumes/kubernetes.io~nfs/makepool1-web3\\nOutput: umount.nfs: /var/lib/kubelet/pods/7be05590-3a46-11e9-906c-20040fedf0bc/volumes/kubernetes.io~nfs/makepool1-web3: device is busy\\n\\n"

目前解决方案：  
    kubectl delete --grace-period=0 --force  
    [https://github.com/kubernetes/kubernetes/issues/51835][https_github.com_kubernetes_kubernetes_issues_51835]

### 9. Kubernetes Pod无法删除,Docker: Device is busy问题的解决 ###

参考： [https://fav.snadn.cn/article/snapshot?id=131\#问题发现][https_fav.snadn.cn_article_snapshot_id_131]

查看证书过期时间

openssl x509 -in xxx.crt -text -noout

> Certificate:  
>     Data:  
>         Version: 3 (0x2)  
>         Serial Number: 2 (0x2)  
>     Signature Algorithm: sha256WithRSAEncryption  
>         Issuer: CN=master-node-ca@1574650458  
>         Validity  
>             Not Before: Nov 25 01:54:18 2019 GMT  
>             Not After : Nov 24 01:54:18 2020 GMT  
>

# 10. k8s 证书过期，一年时间， #

openssl x509 -in /etc/kubernetes/ssl/kubernetes.csr -noout -text |grep ' Not '

> ## 2. 自动轮换 kubelet 证书 ##
> 
> 注：kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启。方法是：
> 
> ### 2.1 增加 kubelet 参数 ###
> 
> \--feature-gates=RotateKubeletServerCertificate=true
> 
> ### 2.2 增加 controller-manager 参数 ###
> 
> \--experimental-cluster-signing-duration=87600h0m0s  
> \--feature-gates=RotateKubeletServerCertificate=true
> 
> ### 2.3 创建 rbac 对象 ###
> 
> 创建rbac对象，允许节点轮换kubelet server证书：
> 
>     apiVersion: rbac.authorization.k8s.io/v1
>     kind: ClusterRole
>     metadata:
>       annotations:
>         rbac.authorization.kubernetes.io/autoupdate: "true"
>       labels:
>         kubernetes.io/bootstrapping: rbac-defaults
>       name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
>     rules:
>     - apiGroups:
>       - certificates.k8s.io
>       resources:
>       - certificatesigningrequests/selfnodeserver
>       verbs:
>       - create
>     ---
>     apiVersion: rbac.authorization.k8s.io/v1
>     kind: ClusterRoleBinding
>     metadata:
>       name: kubeadm:node-autoapprove-certificate-server
>     roleRef:
>       apiGroup: rbac.authorization.k8s.io
>       kind: ClusterRole
>       name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
>     subjects:
>     - apiGroup: rbac.authorization.k8s.io
>       kind: Group
>       name: system:nodes

第二种方法：修改源码kubeadm编译

< kubernetes - 1.14 版本，修改文件 staging/src/k8s.io/client-go/util/cert/cert.go

>= kubernetes - 1.14 版本，修改文件 cmd/kubeadm/app/util/pkiutill/pki-helpers.go

> // NewSignedCert creates a signed certificate using the given CA certificate and key  
> func NewSignedCert(cfg \*certutil.Config, key crypto.Signer, caCert \*x509.Certificate, caKey crypto.Signer) (\*x509.Certificate, error) \{  
>         
> 
> **        const durationTen = time.Hour \* 24 \* 355 \* 10**
> 
>         certTmpl := x509.Certificate\{  
>                 Subject: pkix.Name\{  
>                         CommonName:   cfg.CommonName,  
>                         Organization: cfg.Organization,  
>                 \},  
>                 DNSNames:     cfg.AltNames.DNSNames,  
>                 IPAddresses:  cfg.AltNames.IPs,  
>                 SerialNumber: serial,  
>                 NotBefore:    caCert.NotBefore,  
>                 // NotAfter:     time.Now().Add(kubeadmconstants.CertificateValidity).UTC(),  
>                 NotAfter:     time.Now().Add(durationTen).UTC(),  
>                 KeyUsage:     x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,  
>                 ExtKeyUsage:  cfg.Usages,  
>         \}

编译 make WHAT=cmd/kubeadm GOFLAGES=-v

kubeadm alpha certs renew all --config=/usr/local/install-k8s/core/kubeadm-config.yaml

### 11. k8s无法删除namespace 提示 Terminating ###

解决方法：

> kubectl get ns ns-xxx-zhangzhonglin-444c6833 -o json > ns-delete.json
> 
> 删除文件中spec.finalizers字段
> 
> "spec": \{  
>     \},

注：在执行命令前，要先克隆一个新会话，执行 kubectl proxy --port=8081

> curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns-delete.json http://127.0.0.1:8081/api/v1/namespaces/ns-xxx-zhangzhonglin-444c6833/finalize

## 12. [Kubernetes: No Route to Host][Kubernetes_ No Route to Host] ##

Error getting server version: Get https://10.200.0.1:443/version?timeout=32s: dial tcp 10.200.0.1:443: connect: no route to host

解决方法： iptables -F

苦

### 13. kubeadm kube-controller-manager does not have ceph rbd binary anymore    ###

Error: "failed to create rbd image: executable file not found in $PATH, command output: "

###    [https://github.com/kubernetes/kubernetes/issues/56990][https_github.com_kubernetes_kubernetes_issues_56990] ###

yum install -y ceph-common

### 14. monclient: \_check\_auth\_rotating possible clock skew, rotating keys expired way too early (before 2019-05-06 02:00:09.601676) ###

ceph osd问题，主要是时钟不同步问题

### 15. helm报这个错误 Helm: Error: no available release name found ###

因为 tiller没有正确的角色权限

> kubectl create serviceaccount --namespace kube-system tiller
> 
> kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
> 
> kubectl patch deploy --namespace kube-system tiller-deploy -p '\{"spec":\{"template":\{"spec":\{"serviceAccount":"tiller"\}\}\}\}'

### 16.  attachdetach-controller  Multi-Attach error for volume "pvc-d0fde86c-8661-11e9-b873-0800271c9f15" Volume is already used by pod ###

[The controller-managed attachment and detachment is not able to detach a rbd volume from a lost node \#62061][The controller-managed attachment and detachment is not able to detach a rbd volume from a lost node_62061]

[https://github.com/kubernetes/kubernetes/issues/70349][https_github.com_kubernetes_kubernetes_issues_70349]

[https://github.com/kubernetes/kubernetes/pull/45346][https_github.com_kubernetes_kubernetes_pull_45346]

[https://github.com/kubernetes/kubernetes/issues/53059][https_github.com_kubernetes_kubernetes_issues_53059]

[https://github.com/kubernetes/kubernetes/pull/40148][https_github.com_kubernetes_kubernetes_pull_40148]

Vsphere Cloud Provider: failed to detach volume from shutdown node \#75342

[Don't try to attach volumes which are already attached to other nodes \#45346][https_github.com_kubernetes_kubernetes_pull_45346]

[Pods with volumes stuck in ContainerCreating after cluster node is deleted from OpenStack \#50200][Pods with volumes stuck in ContainerCreating after cluster node is deleted from OpenStack_50200]

[Don't try to attach volumes which are already attached to other nodes\#40148][https_github.com_kubernetes_kubernetes_pull_40148]

[ Pods with volumes stuck in ContainerCreating after cluster node is powered off in vCenter \#50944][Pods with volumes stuck in ContainerCreating after cluster node is powered off in vCenter_50944]

[Pod mount Ceph RDB volumn failed due to timeout. "timeout expired waiting for volumes to attach or mount for pod" \#75492][Pod mount Ceph RDB volumn failed due to timeout. _timeout expired waiting for volumes to attach or mount for pod_75492] (没人跟帖)

17. kubelet 挂掉，csi-rbdplugin 依然建在（statefuleset）

### 18. [k8s pv无法删除问题][Kubernetes_ No Route to Host] ###

pv始终处于“Terminating”状态，而且delete不掉

` `删除k8s中的记录   `kubectl patch pv xxx -p '{"metadata":{"finalizers":null}}'`

### 19. Volumes fail to clean up when kubelet restart due to race between actual and desired state \#75345     ###

Fix race condition between actual and desired state in kublet volume manager \#75458

Pod is stuck in Terminating status forever after Kubelet restart \#72604

### 20. when using ValidatingWebhookConfiguration for deployment subresource(scale) validation. `Internal error occurred: converting (extensions.Deployment).Replicas to (v1beta1.Scale).Replicas: Selector not present in src` ###

`   该问题已经修复，v15版本`

[https://github.com/kubernetes/kubernetes/pull/76849/commits][https_github.com_kubernetes_kubernetes_pull_76849_commits]

#  #

### 21. Error from server: Get https://master-node:10250/containerLogs/default/csi-hostpathplugin-0/node-driver-registrar: dial tcp: lookup master-node on 114.114.114.114:53: no such host ###

解决方法，在 /etc/hosts 添加 192.168.X.X master-node

22. [calico/node is not ready: BIRD is not ready: BGP not established][calico_node is not ready_ BIRD is not ready_ BGP not established] with

主要原因是 calico 傻，没有识别到网卡

modified `calico.yaml` file to include:

- name: IP_AUTODETECTION_METHOD
          value: "interface=ens.*"

### 23. client-go@v11.0.0+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher ###

>   可以尝试手动替换k8s.io/apimachinery@v0.17.0为k8s.io/apimachinery@release-1.14来解决。在终端执行\# go mod download -json k8s.io/apimachinery@release-1.14

### 24. [无法删除image报rbd: error: image still has watchers解决方法][image_rbd_ error_ image still has watchers] ###

参考. [无法删除image报rbd: error: image still has watchers解决方法][image_rbd_ error_ image still has watchers]

> # 解决思路： #
> 
> 在Ceph集群日常运维中，管理员可能会遇到有的image删除不了的情况：  
> **1） 由于image下有快照信息，只需要先将快照信息清除，然后再删除该image即可**  
> **2） 该image仍旧被一个客户端在访问，具体表现为该image中有watcher。如果该客户端异常了，那么就会出现无法删除该image的情况**
> 
> 对于第一种情况，很好解决，下面要说的是第二种情况该如何解决。解决之前先科普一下watcher相关的知识：  
> Ceph中有一个watch/notify机制(粒度是object)，它用来在不同客户端之间进行消息通知，使得各客户端之间的状态保持一致，而每一个进行watch的客户端，对于Ceph集群来说都是一个watcher。
> 
> # 解决方法： #
> 
> # 1. 查看当前image上的watcher #
> 
> ## 查看方法一： ##
> 
>     [root@node3 ~]# rbd status foo
>     watcher=192.168.197.157:0/1135656048 client.4172 cookie=1
> 
> 这种查看方法简单快捷，值得推荐
> 
> ## 查看方法二： ##
> 
> **1） 首先找到image的header对象**
> 
>     [root@node3 ~]# rbd info foo
>     rbd image 'foo':
>             size 1024 MB in 256 objects
>             order 22 (4096 kB objects)
>             block_name_prefix: rbd_data.1041643c9869
>             format: 2
>             features: layering
>             flags: 
>             create_timestamp: Tue Oct 17 10:20:50 2017
> 
> 由该image的block\_name\_prefix为 rbd\_data.1041643c9869，可知该image的header对象为**rbd\_header.1041643c9869**，得到了header对象后，查看watcher信息
> 
> **2） 查看该image的header对象上的watcher信息**
> 
>     [root@node3 ~]# rados -p rbd listwatchers rbd_header.1041643c9869
>     watcher=192.168.197.157:0/1135656048 client.4172 cookie=1
> 
> # 2. 删除image上的watcher #
> 
> ## 2.1 把该watcher加入黑名单： ##
> 
>     [root@node3 ~]# ceph osd blacklist add 192.168.197.157:0/1135656048 
>     blacklisting 192.168.197.157:0/1135656048 until 2017-10-18 12:04:19.103313 (3600 sec)
> 
> ## 2.2 查看占用该image的watcher： ##
> 
>     [root@node3 ~]# rados -p rbd listwatchers  rbd_header.1041643c9869
>     [root@node3 ~]#
> 
> 异常客户端的watcher信息已经不存在了，之后我们就可以对该image进行删除操作了
> 
> ## 2.3 删除该image： ##
> 
>     [root@node3 ~]# rbd rm foo
>     Removing image: 100% complete...done.
> 
> # 3. 后续操作 #
> 
> 实际上做完上面的已经解决了问题，不过最好还是把加入黑名单的客户端移除，下面是有关黑名单的相关操作
> 
> ## 3.1 查询黑名单列表： ##
> 
>     [root@node3 ~]# ceph osd blacklist ls
>     listed 1 entries
>     192.168.197.157:0/1135656048 2017-10-18 12:04:19.103313
> 
> ## 3.2 从黑名单移出一个客户端： ##
> 
>     [root@node3 ~]# ceph osd blacklist rm 192.168.197.157:0/1135656048 
>     un-blacklisting 192.168.197.157:0/1135656048
> 
> ## 3.3 清空黑名单： ##
> 
>     [root@node3 ~]# ceph osd blacklist clear
>      removed all blacklist entries
> 
> # 参考文献 #
> 
> [删除 Ceph 的image报rbd: error: image still has watchers][Ceph _image_rbd_ error_ image still has watchers]

### 25. x509: subject with cn=metrics-client is not in the allowed list: \[aggregator\] ###

请求头部标识不正确，在kube-apiserver中增加配置

--requestheader-allowed-names=aggregator,metrics-client

### 26. metrics-server 安装问题 ###

unable to fully collect metrics: unable to fully scrape metrics from source kubelet\_summary:master-node: unable to fetch metrics from Kubelet master-node (master-node): Get https://master-node:10250/stats/summary/: dial tcp: lookup master-node on 10.200.254.254:53: no such host

>       - name: tmp-dir  
>         emptyDir: \{\}  
>       containers:  
>       - name: metrics-server  
>         image: zhangzhonglin/metrics-server-amd64:v0.3.6  
>         imagePullPolicy: IfNotPresent  
>         args:  
> **          - --kubelet-preferred-address-types=InternalIP,Hostname  
>           - --kubelet-insecure-tls**  
>           - --cert-dir=/tmp  
>           - --secure-port=4443

metrics-server这个容器不能通过CoreDNS 10.200.254.254 解析各Node的主机名，metrics-server连节点时默认是连接节点的主机名，需要加个参数，让它连接节点的IP，同时因为10250是https端口，连接它时需要提供证书，所以加上–kubelet-insecure-tls，表示不验证客户端证书

### 27. Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ###

>   \# kubectl logs aws-node-rvz95 -nkube-system  
>   Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log aws-node-rvz95)

解决方法：

> \# kubectl create clusterrolebinding kubernetes --clusterrole=cluster-admin --user=kubernetes  
> clusterrolebinding.rbac.authorization.k8s.io/kubernetes created

### 28. etcd报错：failed to send out heartbeat on time  etcdserver: server is likely overloaded ###

心跳检测报错主要与磁盘速度、cpu和网络有关

> disk运行过慢导致的，leader一般会在心跳包里附带一些metadata，leader需要先把这些数据固化到磁盘上，然后才能发送。写磁盘过程可能要与其他应用竞争，或者因为磁盘是一个虚拟的或者是SATA类型的导致运行过慢，此时只有更好更快磁盘硬件才能解决问题。
> 
> CPU计算能力不足
> 
> 网速过慢，根据机房间的RTT调整heartbeat-interval，而参数election-timeout则至少是heartbeat-interval的5倍

### 29. etcdserver: applying raft message exceeded backend quota ###

etcdserver: mvcc: database space exceeded

took (489ns) to execute, err is etcdserver: no space

默认etcd空间配额大小为 2G，超过 2G 将不再写入数据。通过给etcd配置 --quota-backend-bytes 参数增大空间配额，最大支持 8G。

>     -quota-backend-bytes 8589934592

# get current revision
    $ rev=$(ETCDCTL_API=3 etcdctl --endpoints=:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*')
    # compact away all old revisions
    $ ETCDCTL_API=3 etcdctl compact $rev
    compacted revision 1516
    # defragment away excessive space
    $ ETCDCTL_API=3 etcdctl defrag
    Finished defragmenting etcd member[127.0.0.1:2379]
    # disarm alarm
    $ ETCDCTL_API=3 etcdctl alarm disarm
    memberID:13803658152347727308 alarm:NOSPACE
    # test puts are allowed again
    $ ETCDCTL_API=3 etcdctl put newkey 123
    OK

## 30. etcdserver: too many requests ##

如果 Raft 模块已提交的索引（committed index）比已应用到状态机的索引（applied index）多于 5000，就返回 "etcdserver: too many requests" 给 client

提交到 Raft 模块的请求，都会做一些简单的限速判断

# 31. 无法访问 gcr.io #

*  [地址1][1]registry.aliyuncs.com/google\_containers
 *  [地址2][2]registry.cn-hangzhou.aliyuncs.com/google\_containers

## 32. (as uid:107, gid:107): Permission denied ##

libguestfs: error: could not create appliance through libvirt.

Try running qemu directly without libvirt using this environment variable:  
export LIBGUESTFS\_BACKEND=direct

Original error from libvirt: Cannot access storage file '/root/-image-build/-image.qcow2' (as uid:107, gid:107): Permission denied \[code=38 int1=13\]

/etc/libvirt/qemu.conf    user=root

### 33. metrics-server 报错 error: metrics not available yet ###

[Kubelet_ - WaltonWang_s Blog - OSCHINA -]: https://my.oschina.net/jxcdwangtao/blog/1629059
[https_github.com_kubernetes_kubernetes_issues_51835]: https://github.com/kubernetes/kubernetes/issues/51835
[https_fav.snadn.cn_article_snapshot_id_131]: https://fav.snadn.cn/article/snapshot?id=131#%E9%97%AE%E9%A2%98%E5%8F%91%E7%8E%B0
[Kubernetes_ No Route to Host]: https://mp.csdn.net/postedit/85257134
[https_github.com_kubernetes_kubernetes_issues_56990]: https://github.com/kubernetes/kubernetes/issues/56990
[The controller-managed attachment and detachment is not able to detach a rbd volume from a lost node_62061]: https://github.com/kubernetes/kubernetes/issues/62061
[https_github.com_kubernetes_kubernetes_issues_70349]: https://github.com/kubernetes/kubernetes/issues/70349
[https_github.com_kubernetes_kubernetes_pull_45346]: https://github.com/kubernetes/kubernetes/pull/45346
[https_github.com_kubernetes_kubernetes_issues_53059]: https://github.com/kubernetes/kubernetes/issues/53059
[https_github.com_kubernetes_kubernetes_pull_40148]: https://github.com/kubernetes/kubernetes/pull/40148
[Pods with volumes stuck in ContainerCreating after cluster node is deleted from OpenStack_50200]: https://github.com/kubernetes/kubernetes/issues/50200
[Pods with volumes stuck in ContainerCreating after cluster node is powered off in vCenter_50944]: https://github.com/kubernetes/kubernetes/issues/50944
[Pod mount Ceph RDB volumn failed due to timeout. _timeout expired waiting for volumes to attach or mount for pod_75492]: https://github.com/kubernetes/kubernetes/issues/75492
[https_github.com_kubernetes_kubernetes_pull_76849_commits]: https://github.com/kubernetes/kubernetes/pull/76849/commits
[calico_node is not ready_ BIRD is not ready_ BGP not established]: https://stackoverflow.com/questions/54465963/calico-node-is-not-ready-bird-is-not-ready-bgp-not-established
[image_rbd_ error_ image still has watchers]: https://www.cnblogs.com/sisimi/p/7776633.html
[Ceph _image_rbd_ error_ image still has watchers]: http://www.cnblogs.com/boshen-hzb/p/6756484.html
[1]: https://www.cnblogs.com/tylerzhou/p/registry.aliyuncs.com/google_containers
[2]: https://www.cnblogs.com/tylerzhou/p/registry.cn-hangzhou.aliyuncs.com/google_containers