【katacontainer基础知识】kata-container介绍与原理

谁践踏了优雅 2022-03-26 02:52 2394阅读 0赞

# WHY ? #

**       弥补了传统容器技术安全性的缺点，**Kata Containers通过使用硬件虚拟化来达到容器隔离的目的。每一个container/pod 都是基于一个独立的kernel实例来作为一个轻量级的虚拟机。自从每一个container/pod运行与独立的虚拟机上，他们不再从宿主机内核上获取相应所有的权限。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70][]

# WHAT ? #

>       kata containers是由OpenStack基金会管理的容器项目。kata containers整合了Intel的 Clear Containers 和 Hyper.sh 的 runV，能够支持不同平台的硬件，并符合Open Container Initiative规范，同时还可以兼容k8s的 CRI（Container Runtime Interface）接口规范。项目包含几个配套组件，即Runtime，Agent， Proxy，Shim等。项目已于6月份release了1.0版本。  
>       从docker架构上看，kata-container和原来的runc是平级的。大家知道docker只是管理容器生命周期的框架，真正启动容器最早用的是LXC，然后是runc，现在也可以换成kata了。所以说kata-container可以当做docker的一个插件，启动kata-container可以通过docker命令。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 1][]

# **kata container组件** #

**   1. Agent**

>        Kata-agent运行在guest负责管理容器。Kata-agent的执行单元是定义了一系列命名空间的沙盒。每个VM可以运行多个容器，支持k8s一个pod运行多个容器的需求。不过目前docker中，kata-runtime只能一个pod一个容器。Kata-agent通过gRPC和其他kata组件通信。Kata-agent使用libcontainer管理容器的生命周期

**  2. Runtime**

>      kata-runtime是一个OCI兼容的容器运行时，负责处理OCI运行时规范指定的所有命令并启动kata-shim实例

配置文件是/usr/share/defaults/kata-containers/configuration.toml

# XXX: WARNING: this file is auto-generated.
    # XXX:
    # XXX: Source file: "cli/config/configuration.toml.in"
    # XXX: Project:
    # XXX:   Name: Kata Containers
    # XXX:   Type: kata
    
    [hypervisor.qemu]
    path = "/usr/bin/qemu-lite-system-x86_64"
    kernel = "/usr/share/kata-containers/vmlinuz.container"
    image = "/usr/share/kata-containers/kata-containers.img"
    machine_type = "pc"
    
    # Optional space-separated list of options to pass to the guest kernel.
    # For example, use `kernel_params = "vsyscall=emulate"` if you are having
    # trouble running pre-2.15 glibc.
    #
    # WARNING: - any parameter specified here will take priority over the default
    # parameter value of the same name used to start the virtual machine.
    # Do not set values here unless you understand the impact of doing so as you
    # may stop the virtual machine from booting.
    # To see the list of default parameters, enable hypervisor debug, create a
    # container and look for 'default-kernel-parameters' log entries.
    kernel_params = ""
    
    # Path to the firmware.
    # If you want that qemu uses the default firmware leave this option empty
    firmware = ""

**  3. Proxy**

>         默认使用virtio-serial和VM通信。VM可以运行多个容器进程。在使用virtio-serial的情况下，与每个进程相关联的I/O流需要在主机上多路复用和解复用。  
>         Kata-proxy给多个kata-shim和kata-runtime客户端提供对kata-agent提供访问，它的主要作用是在每个kata-shim和kata-agent之间路由I/O流和信号。Kata-proxy连接到kata-agent的unix域套接字上，这个套接字是kata-proxy启动时kata-runtime提供的

**  4. Shim**

runtime运行在宿主机上，不能直接监控运行在虚拟机里的进程，最多只能看到QEMU进程。kata-shim监控容器进程，处理容器的所有I/O流，包括stdout、stdin和stderr，以及转发所有的要发送出去的信号。  
        Kata-shim还有其他功能：

*  通过一个UNIX域套接字连接到kata-proxy。这个套接字在kata-runtime启动kata-shim的时候，由kata-runtime传给kata-shim，同时带上了containerID和execID，两个ID用来识别shim管理的是哪个容器。
 *  读取VM内部容器进程的输出流和错误流
 *  使用SignalProcessRequest API转发从reaper到kata-proxy的信号
 *  监控终端修改，并使用grpc TtyWinResize API转发到kata-proxy

kata-runtime kata-env

> **\[Meta\]**  
>   Version = "1.0.18"
> 
> **\[Runtime\]**  
>   Debug = false  
>   Path = "/usr/bin/kata-runtime"  
>   \[Runtime.Version\]  
>     Semver = "1.3.1"  
>     Commit = "258eae0"  
>     OCI = "1.0.1"  
>   \[Runtime.Config\]  
>     Path = "/etc/kata-containers/configuration.toml"
> 
> **\[Hypervisor\]**  
>   MachineType = "pc"  
>   Version = "QEMU emulator version 2.11.0\\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"  
>   Path = "/usr/bin/qemu-lite-system-x86\_64"  
>   BlockDeviceDriver = "virtio-scsi"  
>   EntropySource = "/dev/urandom"  
>   Msize9p = 8192  
>   MemorySlots = 10  
>   Debug = false  
>   UseVSock = false
> 
> **\[Image\]**  
>   Path = "/usr/share/kata-containers/kata-containers-image\_clearlinux\_1.3.1\_agent\_c7fdd324cda.img"
> 
> **\[Kernel\]**  
>   Path = "/usr/share/kata-containers/vmlinuz-4.14.67.16-4.4.container"  
>   Parameters = ""
> 
> **\[Initrd\]**  
>   Path = ""
> 
> **\[Proxy\]**  
>   Type = "kataProxy"  
>   Version = "kata-proxy version 1.3.1-d364b2e"  
>   Path = "/usr/libexec/kata-containers/kata-proxy"  
>   Debug = false
> 
> **\[Shim\]**  
>   Type = "kataShim"  
>   Version = "kata-shim version 1.3.1-58f757d"  
>   Path = "/usr/libexec/kata-containers/kata-shim"  
>   Debug = false
> 
> **\[Agent\]**  
>   Type = "kata"
> 
> **\[Host\]**  
>   Kernel = "3.10.0-957.el7.x86\_64"  
>   Architecture = "amd64"  
>   VMContainerCapable = true  
>   SupportVSocks = false  
>   \[Host.Distro\]  
>     Name = "CentOS Linux"  
>     Version = "7"  
>   \[Host.CPU\]  
>     Vendor = "GenuineIntel"  
>     Model = "Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz"
> 
> **\[Netmon\]**  
>   Version = "kata-netmon version 1.3.1"  
>   Path = "/usr/libexec/kata-containers/kata-netmon"  
>   Debug = false  
>   Enable = false

# HOW ? #

###   1. centos 7 安装docker ###

>     yum -y install yum-utils
>     yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
>     yum -y install docker-ce

###   2. centos 7 安装kata-containers ###

>     source /etc/os-release
>     echo $VERSION_ID
>     VERSION_ID=$VERSION_ID yum-config-manager --add-repo \
>     "http://download.opensuse.org/repositories/home:/katacontainers:/release/CentOS_${VERSION_ID}/home:katacontainers:release.repo"
>     yum -y install kata-runtime kata-proxy kata-shim

###   3. 配置docker启动runtime参数 ###

--add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime

[Unit]
    Description=Docker Application Container Engine
    Documentation=http://docs.docker.com
    After=network.target docker-storage-setup.service
    Wants=docker-storage-setup.service
    
    [Service]
    Type=notify
    Environment=GOTRACEBACK=crash
    ExecReload=/bin/kill -s HUP $MAINPID
    Delegate=yes
    KillMode=process
    ExecStart=/usr/bin/dockerd -s overlay2 --storage-opt overlay2.override_kernel_check=1 \
              --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime \
              $DOCKER_OPTS \
              $DOCKER_STORAGE_OPTIONS \
              $DOCKER_NETWORK_OPTIONS \
              $DOCKER_DNS_OPTIONS \
              $INSECURE_REGISTRY
    LimitNOFILE=1048576
    LimitNPROC=1048576
    LimitCORE=infinity
    TimeoutStartSec=1min
    Restart=on-abnormal
    
    [Install]
    WantedBy=multi-user.target

###   4. 重启docker进程 ###

systemctl daemon-reload && systemctl restart docker

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 2][]

启动容器，查看容器状态，docker stats，同时发现启动qemu进程

![20190115113122507.png][]

# katacontainer kubernetes 集成安装 #

### a. 使用 rpm 安装 ###

http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86\_64:/master/CentOS\_7/x86\_64/  
     http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86\_64:/stable-1.3/CentOS\_7/x86\_64/4 安装

### b. runc ###

不用 docker 跟 k8s 通信而是用 cri-o，同时支持普通容器和 kata 容器，要单独安装 runc  
   https://github.com/opencontainers/runc/releases/download/v1.0.0-rc8/runc.amd64

chmod +x runc.amd64  
   mv runc.amd64 /usr/bin/runc

### c. 安装 crictl CLI ###

[https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.13.0/crictl-v1.13.0-linux-amd64.tar.gz][https_github.com_kubernetes-sigs_cri-tools_releases_download_v1.13.0_crictl-v1.13.0-linux-amd64.tar.gz]

### d. crio ###

https://kubernetes.io/docs/setup/production-environment/container-runtimes/\#cri-o

>    Prerequisites  
> modprobe overlay  
> modprobe br\_netfilter
> 
> \# Setup required sysctl params, these persist across reboots.  
> cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF  
> net.bridge.bridge-nf-call-iptables  = 1  
> net.ipv4.ip\_forward                 = 1  
> net.bridge.bridge-nf-call-ip6tables = 1  
> EOF
> 
> sysctl --system

\# Install prerequisites  
yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-311-candidate/x86\_64/os/

\# Install CRI-O  
yum install --nogpgcheck cri-o

/etc/crio/crio.conf 可以使用命令 "crio config --default >/etc/crio/crio.conf" 创建，然后修改 registries 即可：

### e. 配置 crio ###

修改 kata-runtime

> \# runtime\_untrusted\_workload is the OCI compatible runtime used for untrusted  
> \# container workloads. This is an optional setting, except if  
> \# default\_container\_trust is set to "untrusted".  
> runtime\_untrusted\_workload = "/usr/bin/kata-runtime"

修改镜像仓库

> registries = \[  
>     "docker.io"  
> \]

修改 cgroupfs 等

> \# cgroup\_manager is the cgroup management implementation to be used  
> \# for the runtime.  
> cgroup\_manager = "cgroupfs"

修改 pause 地址

> \# pause\_image is the image which we use to instantiate infra containers.  
> pause\_image = "zhangzhonglin/pause-amd64:3.0"

修改 plugin\_dir 路径

> \# plugin\_dir is is where CNI plugin binaries are stored.  
> plugin\_dir = "/opt/k8s/bin"  
>

Start CRI-O  systemctl start crio

### f. kubelet 配置 ###

添加参数 --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock

测试启动一个 deployment 以注解 ***io.kubernetes.cri-o.TrustedSandbox: "false"***：

> spec:  
>   replicas: 1  
>   selector:  
>     matchLabels:  
>       app: wordpress-hostpath  
>       tier: mysql-hostpath  
>   strategy:  
>     type: Recreate  
>   template:  
>     metadata:  
>       annotations:  
>         ***io.kubernetes.cri-o.TrustedSandbox: "false"***

### crictl image ###

> \# crictl images  
> IMAGE                                 TAG                 IMAGE ID            SIZE  
> docker.io/calico/cni                  v3.7.4              203668d151b2e       139MB  
> docker.io/calico/kube-controllers     v3.7.4              e67ede28cc7e4       46.8MB  
> docker.io/calico/node                 v3.7.4              84b65b552a8f0       159MB  
> docker.io/coredns/coredns             1.5.0               7987f0908caf3       42.7MB  
> docker.io/zhangzhonglin/pause-amd64   3.0                 99e59f495ffaa       753kB

### 发现启动 crio common 进程 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 3][]

### 启动 qemu 进程 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 4][]

### 总结： ###

这个镜像管理比较搓，下载到本地还是不可以，必须是 crio.cfg 配置中的仓库必须能通才可以，containerd 最起码可以管理镜像可以 load

安装 kata-runtime，先验证环境是否支持虚拟化环境，否则就白玩了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 5][]

kata-runtime kata-env，中 VMContainerCapable = true 这个是支持虚拟化环境。

## 2. 使用 containerd 作为 CRI ##

下载地址：https://github.com/containerd/containerd/releases
    解压至目录 /usr/bin 下
    
    sudo systemctl start containerd
    
    sudo systemctl status containerd
    
    containerd config default 生成默认配置，位置/etc/containerd/config.toml

2.1 crictl配置

> \# cat /etc/crictl.yaml   
> runtime-endpoint: unix:///run/containerd/containerd.sock  
> image-endpoint: unix:///run/containerd/containerd.sock  
> timeout: 10  
> debug: true

**    2.2 containerd配置**

containerd config default > /etc/containerd/config.toml

> \# cat /etc/containerd/config.toml 
> 
> root = "/var/lib/containerd"  
> state = "/run/containerd"  
> oom\_score = 0
> 
> \[grpc\]  
>   address = "/run/containerd/containerd.sock"  
>   uid = 0  
>   gid = 0
> 
> .........................................................................

2.2.1 kata-container cri 插件配置，对于 1.5+版本

`plugins.cri.containerd.runtimes.runc`: the runc, and it is the default runtime

`plugins.cri.containerd.runtimes.kata`: The function in containerd (reference [the document here][]) where the dot-connected string `io.containerd.kata.v2` is translated to `containerd-shim-kata-v2`

>     [plugins.cri.containerd]
>           no_pivot = false
>         [plugins.cri.containerd.runtimes]
>           [plugins.cri.containerd.runtimes.runc]
>              runtime_type = "io.containerd.runc.v1"
>              [plugins.cri.containerd.runtimes.runc.options]
>                NoPivotRoot = false
>                NoNewKeyring = false
>                ShimCgroup = ""
>                IoUid = 0
>                IoGid = 0
>                BinaryName = "runc"
>                Root = ""
>                CriuPath = ""
>                SystemdCgroup = false
>           [plugins.cri.containerd.runtimes.kata]
>              runtime_type = "io.containerd.kata.v2"
>           [plugins.cri.containerd.runtimes.katacli]
>              runtime_type = "io.containerd.runc.v1"
>              [plugins.cri.containerd.runtimes.katacli.options]
>                NoPivotRoot = false
>                NoNewKeyring = false
>                ShimCgroup = ""
>                IoUid = 0
>                IoGid = 0
>                BinaryName = "/usr/bin/kata-runtime"
>                Root = ""
>                CriuPath = ""
>                SystemdCgroup = false

参考： [https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md][https_github.com_kata-containers_documentation_blob_master_how-to_containerd-kata.md]

2.3 kubelet配置添加如下：

>   --runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock

镜像下载的问题

ctr cri load xxx.tar

### 参考： ###

>     [https://github.com/kata-containers/runtime][https_github.com_kata-containers_runtime]

# katacontainer 原理架构 #

# Overview #

kata-container 和 runc是平级的。docker 管理容器生命周期的框架，而真正启动容器是 runc，可以换成 kata。kata-container可以当做 docker 的一个插件

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 7][]

### 参考： ###

[https://github.com/kata-containers/documentation/blob/master/design/architecture.md][https_github.com_kata-containers_documentation_blob_master_design_architecture.md]

[https://yq.aliyun.com/articles/692117][https_yq.aliyun.com_articles_692117]

安装 [https://github.com/kata-containers/documentation/blob/master/install/installing-with-kata-doc-to-script.md][https_github.com_kata-containers_documentation_blob_master_install_installing-with-kata-doc-to-script.md]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70]: /images/20220326/3b0303524297467b99e53be9e55802bb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 1]: /images/20220326/2343d45c05cd48d08fb8cc7a124cc000.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 2]: /images/20220326/43ce739e53714ec8aca292f6606da100.png
[20190115113122507.png]: /images/20220326/ee38cbd80f0e4627bfdb2814a482f155.png
[https_github.com_kubernetes-sigs_cri-tools_releases_download_v1.13.0_crictl-v1.13.0-linux-amd64.tar.gz]: https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.13.0/crictl-v1.13.0-linux-amd64.tar.gz
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 3]: /images/20220326/1bb4eed9ab1d44aabf7b5b73824ca35b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 4]: /images/20220326/bc2817c8d4374cc7be114b9e5b16d19c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 5]: /images/20220326/77a323a458604787afe7744d50c62f1f.png
[the document here]: https://github.com/containerd/containerd/tree/master/runtime/v2#binary-naming
[https_github.com_kata-containers_documentation_blob_master_how-to_containerd-kata.md]: https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md
[https_github.com_kata-containers_runtime]: https://github.com/kata-containers/runtime
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 6]: /images/20220326/5a54726724fd4eb09ecf72426938c705.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 7]: /images/20220326/cb4d84bcc6f0484d8f361719abb5b065.png
[https_github.com_kata-containers_documentation_blob_master_design_architecture.md]: https://github.com/kata-containers/documentation/blob/master/design/architecture.md
[https_yq.aliyun.com_articles_692117]: https://yq.aliyun.com/articles/692117
[https_github.com_kata-containers_documentation_blob_master_install_installing-with-kata-doc-to-script.md]: https://github.com/kata-containers/documentation/blob/master/install/installing-with-kata-doc-to-script.md