【kubernetes/k8s概念】k8s csi

清疚 2022-02-14 09:12 410阅读 0赞

## WHY ##

> In-Tree: 需要将后端存储的代码逻辑放到K8S的代码中运行。逻辑代码可能会引起与K8S其他部件之间的相互影响。
> 
> Flexvolume: 调用一个主机的可执行程序包的方式执行存储卷的挂载使用。解决了In-Tree方式的强耦合，不过命令行调用的方式，在主机安全性、部署依赖的容器化、与K8S服务之间的相互扩展性等方面存在不足  
> Flexvolume运行在host 空间，不能使用rbac授权机制访问Kubernetes API，导致其功能极大的受限。

> CSI: CSI标准使K8S和存储提供者之间将彻底解耦，将存储的所有的部件作为容器形式运行在K8S上。

# #

## 目前版本信息： ##

<table> 
 <thead> 
  <tr> 
   <th>Kubernetes</th> 
   <th>CSI Version</th> 
   <th>CSI Status</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>v1.9</td> 
   <td>v0.1</td> 
   <td>Alpha</td> 
  </tr> 
  <tr> 
   <td>v1.10</td> 
   <td>v0.2</td> 
   <td>Beta</td> 
  </tr> 
  <tr> 
   <td>v1.11</td> 
   <td>v0.3</td> 
   <td>Beta</td> 
  </tr> 
  <tr> 
   <td>v1.12</td> 
   <td>v0.3</td> 
   <td>Beta</td> 
  </tr> 
  <tr> 
   <td>v1.13</td> 
   <td>v1.0.0</td> 
   <td>GA</td> 
  </tr> 
 </tbody> 
</table>

**       CSI是Container Storage Interface的缩写。CSI是由来自Kubernetes、Mesos、 Docker等社区的member联合制定的一个行业标准接口规范，旨在将任意存储系统暴露给容器化应用程序。CSI规范定义了存储提供商（SP）实现CSI兼容插件的最小操作集和部署建议。CSI规范的主要焦点是声明插件必须实现的接口**

## 架构图 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70][]

三个独立的外部组件（External Components），即：Driver Registrar、External Provisioner 和 External Attacher，对应的正是从 Kubernetes 项目里面剥离出来的那部分存储管理功能。

External Components 虽然是外部组件，但依然由 Kubernetes 社区来开发和维护。

参考：[http://bamboox.online/k8s-15-%E5%AD%98%E5%82%A8-02.html][http_bamboox.online_k8s-15-_E5_AD_98_E5_82_A8-02.html]

### Driver Registrar ###

将插件注册到 kubelet ，需要请求 CSI 插件的 Identity 服务来获取插件信息

### External Provisioner ###

Watch  APIServer 的 PVC 对象，PVC 被创建时，就会调用 CSI Controller 的 CreateVolume 方法，创建对应 PV

### External Attacher ###

Watch APIServer 的 VolumeAttachment 对象，就会调用 CSI Controller 服务的 ControllerPublish 方法，完成它所对应的 Volume 的 Attach 阶段

### 开发步骤 ###

> 1.  Create a containerized application implementing the *Identity*, *Node*, and optionally the *Controller* services described in the [CSI specification][] (the CSI driver container). See [Developing CSI Driver][] for more information.
> 2.  Unit test it using csi-sanity. See [Driver - Unit Testing][] for more information.
> 3.  Define Kubernetes API YAML files that deploy the CSI driver container along with appropriate sidecar containers. See [Deploying in Kubernetes][] for more information.
> 4.  Deploy the driver on a Kubernetes cluster and run end-to-end functional tests on it. See [Driver - Functional Testing][]

At a minimum, CSI drivers must implement the following CSI services:

>  *  CSI `Identity` service
>     
>      *  Enables callers (Kubernetes components and CSI sidecar containers) to identify the driver and what optional functionality it supports.
>  *  CSI `Node` service
>     
>      *  Only `NodePublishVolume`, `NodeUnpublishVolume`, and `NodeGetCapabilities` are required.
>      *  Required methods enable callers to make a volume available at a specified path and discover what optional functionality the driver supports.

# 一.  Identity 身份服务 #

Node Plugin和Controller Plugin都必须实现这些RPC集。协调kubernetes与csi的版本信息，

负责对外暴露这个插件的信息

service Identity {
      rpc GetPluginInfo(GetPluginInfoRequest)
        returns (GetPluginInfoResponse) {}
    
      rpc GetPluginCapabilities(GetPluginCapabilitiesRequest)
        returns (GetPluginCapabilitiesResponse) {}
    
      rpc Probe (ProbeRequest)
        returns (ProbeResponse) {}
    }

*  PluginCapability\_Service\_CONTROLLER\_SERVICE：代表对 ControllerService 插件提供 RPC 服务，这个是可选的，如果实现了就开启
 *  PluginCapability\_Service\_VOLUME\_ACCESSIBILITY\_CONSTRAINTS：代表 volume 在集群中访问限制

# 二. Controller 控制器服务 #

Controller Plugin必须实现这些RPC集。创建以及管理volume管理卷

service Controller {
      rpc CreateVolume (CreateVolumeRequest)
        returns (CreateVolumeResponse) {}
    
      rpc DeleteVolume (DeleteVolumeRequest)
        returns (DeleteVolumeResponse) {}
    
      rpc ControllerPublishVolume (ControllerPublishVolumeRequest)
        returns (ControllerPublishVolumeResponse) {}
    
      rpc ControllerUnpublishVolume (ControllerUnpublishVolumeRequest)
        returns (ControllerUnpublishVolumeResponse) {}
    
      rpc ValidateVolumeCapabilities (ValidateVolumeCapabilitiesRequest)
        returns (ValidateVolumeCapabilitiesResponse) {}
    
      rpc ListVolumes (ListVolumesRequest)
        returns (ListVolumesResponse) {}
    
      rpc GetCapacity (GetCapacityRequest)
        returns (GetCapacityResponse) {}
    
      rpc ControllerGetCapabilities (ControllerGetCapabilitiesRequest)
        returns (ControllerGetCapabilitiesResponse) {}
    
      rpc CreateSnapshot (CreateSnapshotRequest)
        returns (CreateSnapshotResponse) {}
    
      rpc DeleteSnapshot (DeleteSnapshotRequest)
        returns (DeleteSnapshotResponse) {}
    
      rpc ListSnapshots (ListSnapshotsRequest)
        returns (ListSnapshotsResponse) {}
    
      rpc ControllerExpandVolume (ControllerExpandVolumeRequest)
        returns (ControllerExpandVolumeResponse) {}
    }

*  ControllerServiceCapability\_RPC\_CREATE\_DELETE\_VOLUME：支持动态 volume 执行 provision delete 操作
 *  ControllerServiceCapability\_RPC\_PUBLISH\_UNPUBLISH\_VOLUME：代表实现了 ControllerPublishVolume 和 ControllerUnpublishVolume 方法，对应 kubernetes 中 attach detach volume 操作

# 三. Node 节点服务 #

Node Plugin必须实现这些RPC集。 将volume存储卷挂载到指定目录中，/var/lib/kubelet/plugins/$\{plugin\_name\}/csi.sock

service Node {
      rpc NodeStageVolume (NodeStageVolumeRequest)
        returns (NodeStageVolumeResponse) {}
    
      rpc NodeUnstageVolume (NodeUnstageVolumeRequest)
        returns (NodeUnstageVolumeResponse) {}
    
      rpc NodePublishVolume (NodePublishVolumeRequest)
        returns (NodePublishVolumeResponse) {}
    
      rpc NodeUnpublishVolume (NodeUnpublishVolumeRequest)
        returns (NodeUnpublishVolumeResponse) {}
    
      rpc NodeGetVolumeStats (NodeGetVolumeStatsRequest)
        returns (NodeGetVolumeStatsResponse) {}
    
    
      rpc NodeExpandVolume(NodeExpandVolumeRequest)
        returns (NodeExpandVolumeResponse) {}
    
    
      rpc NodeGetCapabilities (NodeGetCapabilitiesRequest)
        returns (NodeGetCapabilitiesResponse) {}
    
      rpc NodeGetInfo (NodeGetInfoRequest)
        returns (NodeGetInfoResponse) {}
    }

*  NodeServiceCapability\_RPC\_STAGE\_UNSTAGE\_VOLUME：实现了 NodeStageVolume 和 NodeUnstageVolume 方法，对应了 kubernetes 中的 mount / unmount device 操作

# 1. DefaultIdentityServer保存driver对象 #

实现了三个接口 GetPluginInfo，GetPluginCapabilities，Probe

// DefaultIdentityServer stores driver object
    type DefaultIdentityServer struct {
    	Driver *CSIDriver
    }

###      1.1 GetPluginInfo ###

获得plugin 信息，包括插件名字与版本

// GetPluginInfo returns plugin information
    func (ids *DefaultIdentityServer) GetPluginInfo(ctx context.Context, req *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {
    	klog.V(5).Infof("Using default GetPluginInfo")
    
    	if ids.Driver.name == "" {
    		return nil, status.Error(codes.Unavailable, "Driver name not configured")
    	}
    
    	if ids.Driver.version == "" {
    		return nil, status.Error(codes.Unavailable, "Driver is missing version")
    	}
    
    	return &csi.GetPluginInfoResponse{
    		Name:          ids.Driver.name,
    		VendorVersion: ids.Driver.version,
    	}, nil
    }

###     1.2 Probe 用于探测 ###

// Probe returns empty response
    func (ids *DefaultIdentityServer) Probe(ctx context.Context, req *csi.ProbeRequest) (*csi.ProbeResponse, error) {
    	return &csi.ProbeResponse{}, nil
    }

###     1.3 GetPluginCapabilities用于获得插件具有那些特权 ###

// GetPluginCapabilities returns plugin capabilities
    func (ids *DefaultIdentityServer) GetPluginCapabilities(ctx context.Context, req *csi.GetPluginCapabilitiesRequest) (*csi.GetPluginCapabilitiesResponse, error) {
    	klog.V(5).Infof("Using default capabilities")
    	return &csi.GetPluginCapabilitiesResponse{
    		Capabilities: []*csi.PluginCapability{
    			{
    				Type: &csi.PluginCapability_Service_{
    					Service: &csi.PluginCapability_Service{
    						Type: csi.PluginCapability_Service_CONTROLLER_SERVICE,
    					},
    				},
    			},
    		},
    	}, nil
    }

# 2.  DefaultNodeServer实现了NodeServer接口 #

*  NodeStageVolume(ctx context.Context, req \*csi.NodeStageVolumeRequest)
 *  NodeUnstageVolume(ctx context.Context, req \*csi.NodeUnstageVolumeRequest)
 *  NodeGetInfo(ctx context.Context, req \*csi.NodeGetInfoRequest)
 *  NodeGetCapabilities(ctx context.Context, req \*csi.NodeGetCapabilitiesRequest)
 *  NodeGetVolumeStats(ctx context.Context, in \*csi.NodeGetVolumeStatsRequest)

// DefaultNodeServer stores driver object
    type DefaultNodeServer struct {
    	Driver *CSIDriver
    }

# 四. CSIDriver Object #

主要有两个目的：

*  简化了 driver 服务发现：可以通过 Kubectl get CSIDriver 
 *  定制 k8s driver 行为功能：比如默认 Attach / Detach 操作

###     4.1 CSIDriver Object 定义格式 ###

attachRequired:：需要 attach 操作，实现了 **ControllerPublishVolume** 方法

podInfoOnMount：在 mount 操作中需要 pod 的信息，比如 name UID 等，如果设置为 true，**NodePublishVolume** 方法中 volume\_context 如下所示：

*  `"csi.storage.k8s.io/pod.name": pod.Name`
 *  `"csi.storage.k8s.io/pod.namespace": pod.Namespace`
 *  `"csi.storage.k8s.io/pod.uid": string(pod.UID)`

volumeLifecycleModes：k8s v1.16 版本新增，

> apiVersion: storage.k8s.io/v1beta1  
> kind: CSIDriver  
> metadata:  
>   name: hostpath.csi.k8s.io  
> spec:  
>   attachRequired: true  
>   podInfoOnMount: true  
>   volumeLifecycleModes:  
>   - Persistent  
>   - Ephemeral

# 五. CSINode Object #

主要有以下几个目的：

*  影射 k8s node 名字到 CSI node 名字：如果新的 CSI driver 注册，存储信息
 *  driver 可用性
 *  volume topology

###     5.1 CSINode Object 定义 ###

*  `drivers` - list of CSI drivers running on the node and their properties.
 *  `name` - the CSI driver that this object refers to.
 *  `nodeID` - the assigned identifier for the node as determined by the driver.
 *  `topologyKeys` - A list of topology keys assigned to the node as supported by the driver.

> apiVersion: storage.k8s.io/v1beta1  
> kind: CSINode  
> metadata:  
>   name: master-node  
> spec:  
>   drivers:  
>   - name: hostpath.csi.k8s.io  
>     nodeID: master-node  
>     topologyKeys:  
>     - topology.hostpath.csi/node

[node-driver-registrar][] sidecar container 创建 CSINode 对象

# [Kubernetes Changelog][] #

## [Kubernetes 1.14][] ##

### [Breaking Changes][] ###

*  `csi.storage.k8s.io/v1alpha1` `CSINodeInfo` and `CSIDriver` CRDs are no longer supported.

### [Features][] ###

*  New beta features:
    
     *  Topology
     *  Raw block
     *  Skip attach
     *  Pod info on mount
 *  New alpha features:
    
     *  Volume expansion
 *  New `storage.k8s.io/v1beta1` `CSINode` and `CSIDriver` objects were introduced.

## [Kubernetes 1.13][] ##

### [Deprecations][] ###

*  CSI spec 0.2 and 0.3 are deprecated and support will be removed in Kubernetes 1.18.

### [Features][Features 1] ###

*  GA support added for [CSI spec 1.0][].

### 参考： ###

Kubernetes Container Storage Interface (CSI) Documentation

How to write a Container Storage Interface (CSI) plugin

[https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/][https_arslan.io_2018_06_21_how-to-write-a-container-storage-interface-csi-plugin]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70]: /images/20220214/775b7d8fcac84f8c95472a1914835e73.png
[http_bamboox.online_k8s-15-_E5_AD_98_E5_82_A8-02.html]: http://bamboox.online/k8s-15-%E5%AD%98%E5%82%A8-02.html
[CSI specification]: https://github.com/container-storage-interface/spec/blob/master/spec.md#rpc-interface
[Developing CSI Driver]: https://kubernetes-csi.github.io/docs/developing.html
[Driver - Unit Testing]: https://kubernetes-csi.github.io/docs/unit-testing.html
[Deploying in Kubernetes]: https://kubernetes-csi.github.io/docs/deploying.html
[Driver - Functional Testing]: https://kubernetes-csi.github.io/docs/functional-testing.html
[node-driver-registrar]: https://kubernetes-csi.github.io/docs/node-driver-registrar.html
[Kubernetes Changelog]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#kubernetes-changelog
[Kubernetes 1.14]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#kubernetes-114
[Breaking Changes]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#breaking-changes
[Features]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#features
[Kubernetes 1.13]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#kubernetes-113
[Deprecations]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#deprecations
[Features 1]: https://kubernetes-csi.github.io/docs/kubernetes-changelog.html#features-1
[CSI spec 1.0]: https://github.com/container-storage-interface/spec/releases/tag/v1.0.0
[https_arslan.io_2018_06_21_how-to-write-a-container-storage-interface-csi-plugin]: https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/