K8S 配置 storageclass 使用 nfs 动态申领本地磁盘空间

╰半夏微凉° 2023-09-28 11:31 88阅读 0赞

##### 系列文章目录 #####

--------------------

第一章：✨ [k8s入门：裸机部署 k8s 集群][k8s_ k8s]  
第二章：✨ [k8s入门：部署应用到 k8s 集群][k8s_ k8s 1]  
第三章：✨ [k8s入门：service 简单使用][k8s_service]  
第四章：✨ [k8s入门：StatefulSet 简单使用][k8s_StatefulSet]  
第五章：✨ [k8s入门：存储（storage）][k8s_storage]  
第六章：✨ [K8S 配置 storageclass 使用 nfs 动态申领本地磁盘空间][K8S _ storageclass _ nfs]  
第七章：✨ [k8s入门：配置 ConfigMap & Secret][k8s_ ConfigMap _ Secret]  
第八章：✨ [k8s入门：k8s入门：Helm 构建 MySQL][k8s_k8s_Helm _ MySQL]  
第九章：✨ [k8s入门：kubernetes-dashboard 安装][k8s_kubernetes-dashboard]  
第十章：✨ [k8s入门：kube-prometheus-stack 全家桶搭建（Grafana + Prometheus）][k8s_kube-prometheus-stack _Grafana _ Prometheus]

--------------------

#### 文章目录 ####

*   *   *   *  系列文章目录
             *  一、简介
             *  二、配置文件
             *  三、rbac.yaml
             *  四、deployment.yaml
             *  五、class.yaml
             *  六、test-claim.yaml
             *  七、test-pod.yaml
             *  八、动态新增 PV

--------------------

##### 一、简介 #####

本篇文章使用 PersistentVolumeClaim （pvc） 结合资源对象 StorageClass （SC） 去动态申领 PersistentVolume （PV）本地 nfs 共享文件系统存储空间，SC 的根本作用就是根据 pvc 的定义来动态创建pv，不仅节省了我们管理员的时间，还可以封装不同类型的存储供 pvc 选用。

每个 StorageClass 都包含以下三个重要的字段，这些字段会在sc需要动态分配pv时会使用到：

*  每个 StorageClass 都有一个制备器（Provisioner），用来决定使用哪个卷插件制备 PV。 该字段必须指定。
 *  回收策略（reclaimPolicy）可以是 Delete 或者 Retain，如果 StorageClass 对象创建时没有指定 reclaimPolicy，它将默认为 Delete。
 *  PersistentVolume 可以配置为可扩展。将此功能设置为 true 时，允许用户通过编辑相应的 PVC 对象来调整卷大小。

注意： 首先保证你的 nfs 共享文件系统可用，没有安装的可参考 [https://blog.csdn.net/qq\_41538097/article/details/125215989][k8s_storage] 步骤七

配置文件可参考：[https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client/deploy][https_github.com_kubernetes-retired_external-storage_tree_master_nfs-client_deploy]  
StorageClass 详细使用可参考： [https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes][https_kubernetes.io_zh-cn_docs_concepts_storage_storage-classes]

##### 二、配置文件 #####

配置文件可以从该链接获取 [https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes][https_kubernetes.io_zh-cn_docs_concepts_storage_storage-classes]

也可以命令将整个项目拉取下来`git clone https://github.com/kubernetes-incubator/external-storage.git`

我的配置文件如下

[root@master nfs]# pwd
    /root/StorageClass/nfs
    [root@master nfs]# ls
    class.yaml  deployment.yaml  rbac.yaml  test-claim.yaml  test-pod.yaml

##### 三、rbac.yaml #####

创建 rbac（基于角色的访问控制）配置文件 rbac.yaml，就是用户通过角色与权限进行关联。

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: nfs-client-provisioner-runner
    rules:
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["create", "update", "patch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: run-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: ClusterRole
      name: nfs-client-provisioner-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    rules:
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: Role
      name: leader-locking-nfs-client-provisioner
      apiGroup: rbac.authorization.k8s.io

部署 rbac 配置文件

[root@master nfs]# kubectl apply -f rbac.yaml 
    serviceaccount/nfs-client-provisioner created
    clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
    clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
    role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
    rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created

##### 四、deployment.yaml #####

新增 nfs 的 Deployment 配置 deployment.yaml

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nfs-client-provisioner
      labels:
        app: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    spec:
      replicas: 1
      strategy:
        type: Recreate
      selector:
        matchLabels:
          app: nfs-client-provisioner
      template:
        metadata:
          labels:
            app: nfs-client-provisioner
        spec:
          serviceAccountName: nfs-client-provisioner
          containers:
            - name: nfs-client-provisioner
              image: quay.io/external_storage/nfs-client-provisioner:latest
              volumeMounts:
                - name: nfs-client-root
                  mountPath: /persistentvolumes
              env:
                - name: PROVISIONER_NAME
                  value: nfs-diy # StorageClass 三个重要字段之一 Provisioner，名字自己指定
                - name: NFS_SERVER
                  value: 192.168.25.100 # nfs 服务器地址
                - name: NFS_PATH
                  value: /home/data # nfs 共享文件夹
          volumes:
            - name: nfs-client-root
              nfs:
                server: 192.168.25.100 # nfs 服务器地址
                path: /home/data # nfs 共享文件夹

部署 nfs 的 Deployment

[root@master nfs]# kubectl apply -f deployment.yaml 
    deployment.apps/nfs-client-provisioner created
    [root@master nfs]# kubectl get deployment
    NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
    nfs-client-provisioner   1/1     1            1           10s
    [root@master nfs]# kubectl get pod -o wide
    NAME                                      READY   STATUS    RESTARTS        AGE    IP            NODE    NOMINATED NODE   READINESS GATES
    nfs-client-provisioner-6b4c9588c6-9hs2q   1/1     Running   0               23s    10.244.2.46   node2   <none>           <none>

##### 五、class.yaml #####

定义了 存储类 class.yaml

apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: nfs-storage
    provisioner: nfs-diy # 和 deployment.yaml 中 env.PROVISIONER_NAME 保持一致
    reclaimPolicy: Retain
    parameters:
      archiveOnDelete: "false"

部署 SC

[root@master nfs]# kubectl apply -f class.yaml 
    storageclass.storage.k8s.io/nfs-storage created
    [root@master nfs]# kubectl get sc
    NAME          PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
    nfs-storage   nfs-diy       Retain          Immediate           false                  15s

##### 六、test-claim.yaml #####

使用 PVC 申领本地磁盘

kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-claim
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Mi

部署 PVC

[root@master nfs]# kubectl apply -f test-claim.yaml 
    persistentvolumeclaim/test-claim created
    [root@master nfs]# kubectl get pvc
    NAME                 STATUS    VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    test-claim           Pending                                                   nfs-storage    9s
    [root@master nfs]# kubectl describe pvc/test-claim
    Name:          test-claim
    Namespace:     default
    StorageClass:  nfs-storage
    Status:        Pending
    Volume:        
    Labels:        <none>
    Annotations:   volume.beta.kubernetes.io/storage-class: nfs-storage
                   volume.beta.kubernetes.io/storage-provisioner: nfs-diy
                   volume.kubernetes.io/storage-provisioner: nfs-diy
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      
    Access Modes:  
    VolumeMode:    Filesystem
    Used By:       <none>
    Events:
      Type    Reason                Age               From                         Message
      ----    ------                ----              ----                         -------
      Normal  ExternalProvisioning  6s (x3 over 15s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator

如上可发现创建失败，报错 `waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator`

注意：如果是 v1.20 版本以上 apiserver 默认禁止使用 selfLink，需要手动配置`- --feature-gates=RemoveSelfLink=false` 开启。

修改kube-apiserver.yaml 文件：

[root@master nfs]# cat /etc/kubernetes/manifests/kube-apiserver.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      annotations:
        kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.25.100:6443
      creationTimestamp: null
      labels:
        component: kube-apiserver
        tier: control-plane
      name: kube-apiserver
      namespace: kube-system
    spec:
      containers:
      - command:
        - kube-apiserver
        - --advertise-address=192.168.25.100
        - --feature-gates=RemoveSelfLink=false   # 添加这条信息
        - --allow-privileged=true

`systemctl restart kubelet`重启 kubelet 服务，然后查看 PVC，PV 信息，可以看到 PVC 申领 PV 卷成功。  
注意： 如果不成功，仍然报这个错

*  可以使用 `kubectl get pod -n kube-system` 查看 `kube-apiserver` 是否启动成功
 *  查看上面 deployment 创建的 pod 是否启动失败

[root@master nfs]# systemctl restart kubelet
    [root@master nfs]# kubectl describe pvc/test-claim 
    Name:          test-claim
    Namespace:     default
    StorageClass:  nfs-storage
    Status:        Bound
    Volume:        pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
    Labels:        <none>
    Annotations:   pv.kubernetes.io/bind-completed: yes
                   pv.kubernetes.io/bound-by-controller: yes
                   volume.beta.kubernetes.io/storage-class: nfs-storage
                   volume.beta.kubernetes.io/storage-provisioner: nfs-diy
                   volume.kubernetes.io/storage-provisioner: nfs-diy
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      1Mi
    Access Modes:  RWX
    VolumeMode:    Filesystem
    Used By:       <none>
    Events:
      Type    Reason                 Age                From                                                                                  Message
      ----    ------                 ----               ----                                                                                  -------
      Normal  ExternalProvisioning   12s (x5 over 72s)  persistentvolume-controller                                                           waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator
      Normal  Provisioning           5s                 nfs-diy_nfs-client-provisioner-5c58f4c86f-ndffx_15778b3c-f0a6-11ec-894c-ceb09bae6415  External provisioner is provisioning volume for claim "default/test-claim"
      Normal  ProvisioningSucceeded  5s                 nfs-diy_nfs-client-provisioner-5c58f4c86f-ndffx_15778b3c-f0a6-11ec-894c-ceb09bae6415  Successfully provisioned volume pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
    [root@master nfs]# kubectl get pvc
    NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    test-claim           Bound    pvc-c2ae61f8-7c40-4393-9110-114a0455afa1   1Mi        RWX            nfs-storage    107s
    [root@master nfs]# kubectl get pv
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
    pvc-c2ae61f8-7c40-4393-9110-114a0455afa1   1Mi        RWX            Retain           Bound    default/test-claim           nfs-storage             109s
    [root@master nfs]# ls /home/data
    default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1

PVC 动态持久卷在 nfs 服务器持久卷路径被配置为 `共享文件夹目录/${namespace}-${pvcName}-${pvName}`，比如：`/home/data/default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1`

##### 七、test-pod.yaml #####

创建 pod 测试

kind: Pod
    apiVersion: v1
    metadata:
      name: test-pod-nginx
    spec:
      containers:
        - name: test-pod-nginx-containers
          image: nginx:latest
          ports:
            - containerPort: 80
              name: "http-server"
              hostPort: 80
          volumeMounts:
            - name: test-pod-nginx-storage
              mountPath: "/usr/share/nginx/html"
      restartPolicy: "Never"
      volumes:
        - name: test-pod-nginx-storage
          persistentVolumeClaim:
            claimName: test-claim

部署 pod

[root@master nfs]# kubectl apply -f test-pod.yaml 
    pod/test-pod-nginx created
    [root@master nfs]# kubectl get pod -o wide
    NAME                                      READY   STATUS    RESTARTS        AGE     IP            NODE    NOMINATED NODE   READINESS GATES
    test-pod-nginx                            1/1     Running   0               33s     10.244.1.46   node1   <none>           <none>
    [root@master data]# kubectl get pv
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
    pvc-c2ae61f8-7c40-4393-9110-114a0455afa1   1Mi        RWX            Retain           Bound    default/test-claim           nfs-storage             13m
    [root@master nfs]# echo "K8S configures storageclass to dynamically claim local disk space using nfs" > /home/data/default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1/index.html

因为上述 pod 部署到了 node1 节点（192.168.1.101），访问 [http://192.168.25.101/][http_192.168.25.101] 测试成功

![在这里插入图片描述][f7e21215fefe4139a6798296c33ebef5.png]

##### 八、动态新增 PV #####

test-claim2.yaml，通过 nfs-storage 动态申领 100M PV 本地存储

kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-claim2
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 100Mi

部署 PVC，可以看到申请的 PV 卷名字 pvc-9385f11d-2587-469b-bd13-3a756fe61acb，大小为 100 M，挂载目录：`/home/data/default-test-claim2-pvc-9385f11d-2587-469b-bd13-3a756fe61acb`

[root@master nfs]# kubectl apply -f test-claim2.yaml 
    persistentvolumeclaim/test-claim2 created
    [root@master nfs]# kubectl get pvc
    NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    test-claim           Bound    pvc-c2ae61f8-7c40-4393-9110-114a0455afa1   1Mi        RWX            nfs-storage    10h
    test-claim2          Bound    pvc-9385f11d-2587-469b-bd13-3a756fe61acb   100Mi      RWX            nfs-storage    5s
    [root@master nfs]# kubectl get pv
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
    pvc-9385f11d-2587-469b-bd13-3a756fe61acb   100Mi      RWX            Retain           Bound    default/test-claim2          nfs-storage             17s
    pvc-c2ae61f8-7c40-4393-9110-114a0455afa1   1Mi        RWX            Retain           Bound    default/test-claim           nfs-storage             10h

[k8s_ k8s]: https://blog.csdn.net/qq_41538097/article/details/124869179
[k8s_ k8s 1]: https://blog.csdn.net/qq_41538097/article/details/124884400
[k8s_service]: https://blog.csdn.net/qq_41538097/article/details/125175685
[k8s_StatefulSet]: https://blog.csdn.net/qq_41538097/article/details/125189618
[k8s_storage]: https://blog.csdn.net/qq_41538097/article/details/125215989
[K8S _ storageclass _ nfs]: https://blog.csdn.net/qq_41538097/article/details/125380287
[k8s_ ConfigMap _ Secret]: https://blog.csdn.net/qq_41538097/article/details/125387461
[k8s_k8s_Helm _ MySQL]: https://yixiu.blog.csdn.net/article/details/125442282
[k8s_kubernetes-dashboard]: https://blog.csdn.net/qq_41538097/article/details/125561769
[k8s_kube-prometheus-stack _Grafana _ Prometheus]: https://blog.csdn.net/qq_41538097/article/details/125564711
[https_github.com_kubernetes-retired_external-storage_tree_master_nfs-client_deploy]: https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client/deploy
[https_kubernetes.io_zh-cn_docs_concepts_storage_storage-classes]: https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes
[http_192.168.25.101]: http://192.168.25.101/
[f7e21215fefe4139a6798296c33ebef5.png]: https://img-blog.csdnimg.cn/f7e21215fefe4139a6798296c33ebef5.png