K8S通过rook部署rook ceph集群、配置dashboard访问并创建pvc

客官°小女子只卖身不卖艺 2023-09-28 21:51 15阅读 0赞

## Rook概述 ##

### Ceph简介 ###

Ceph是一种高度可扩展的分布式存储解决方案，提供对象、文件和块存储。在每个存储节点上，将找到Ceph存储对象的文件系统和Ceph OSD(对象存储守护程序)进程。在Ceph集群上，还存在Ceph MON(监控)守护程序，它们确保Ceph集群保持高可用性。

### Rook简介 ###

Rook 是一个开源的cloud-native storage编排, 提供平台和框架；为各种存储解决方案提供平台、框架和支持，以便与云原生环境本地集成。目前主要专用于Cloud-Native环境的文件、块、对象存储服务。它实现了一个自我管理的、自我扩容的、自我修复的分布式存储服务。  
Rook支持自动部署、启动、配置、分配(provisioning)、扩容/缩容、升级、迁移、灾难恢复、监控，以及资源管理。为了实现所有这些功能，Rook依赖底层的容器编排平台，例如 kubernetes、CoreOS 等。。  
Rook 目前支持Ceph、NFS、Minio Object Store、Edegefs、Cassandra、CockroachDB 存储的搭建。  
Rook机制：  
Rook 提供了卷插件，来扩展了 K8S 的存储系统，使用 Kubelet 代理程序 Pod 可以挂载 Rook 管理的块设备和文件系统。  
Rook Operator 负责启动并监控整个底层存储系统，例如 Ceph Pod、Ceph OSD 等，同时它还管理 CRD、对象存储、文件系统。  
Rook Agent 代理部署在 K8S 每个节点上以 Pod 容器运行，每个代理 Pod 都配置一个 Flexvolume 驱动，该驱动主要用来跟 K8S 的卷控制框架集成起来，每个节点上的相关的操作，例如添加存储设备、挂载、格式化、删除存储等操作，都有该代理来完成。

## Rook架构 ##

Rook架构如下：

![img][]

环境：

192.168.200.3 master1
    192.168.200.4 master2
    192.168.200.5 master3
    192.168.200.6 node1
    192.168.200.7 node2
    192.168.200.8 node3

1.三台node节点分别添加一块磁盘并输入以下命令识别磁盘(不用重启)

echo "- - -" >/sys/class/scsi_host/host0/scan
    echo "- - -" >/sys/class/scsi_host/host1/scan
    echo "- - -" >/sys/class/scsi_host/host2/scan

2.克隆github项目

git clone https://github.com/rook/rook.git

3.切换到需要的版本分支

cd rook
    git checkout -b release-1.1 remotes/origin/release-1.1
    git branch -a

![img][img 1]

4.使用node节点存储，在master1上需要修改参数

kubectl taint node master1 node-role.kubernetes.io/master="":NoSchedule
    kubectl taint node master2 node-role.kubernetes.io/master="":NoSchedule
    kubectl taint node master3 node-role.kubernetes.io/master="":NoSchedule
    kubectl label nodes {
        node1,node2,node3} ceph-osd=enabled
    kubectl label nodes {
        node1,node2,node3} ceph-mon=enabled
    kubectl label nodes node1 ceph-mgr=enabled

5.进入项目路径安装operator

cd rook/cluster/examples/kubernetes/ceph
    kubectl apply -f common.yaml
    kubectl apply -f operator.yaml

![img][img 2]

6.配置cluster.yaml

apiVersion: ceph.rook.io/v1
    kind: CephCluster
    metadata:
      name: rook-ceph
      namespace: rook-ceph
    spec:
      cephVersion:
        image: ceph/ceph:v14.2.4-20190917
        allowUnsupported: false
      dataDirHostPath: /var/lib/rook
      skipUpgradeChecks: false
      mon:
        count: 3
        allowMultiplePerNode: false
      dashboard:
        enabled: true
        ssl: true
      monitoring:
        enabled: false
        rulesNamespace: rook-ceph
      network:
        hostNetwork: false
      rbdMirroring:
        workers: 0
      placement:
     #    all:
     #      nodeAffinity:
     #        requiredDuringSchedulingIgnoredDuringExecution:
     #          nodeSelectorTerms:
     #          - matchExpressions:
     #            - key: role
     #              operator: In
     #              values:
     #              - storage-node
     #      podAffinity:
     #      podAntiAffinity:
     #      tolerations:
     #      - key: storage-node
     #        operator: Exists
        mon:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: ceph-mon
                  operator: In
                  values:
                  - enabled
          tolerations:
          - key: ceph-mon
            operator: Exists
        ods:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: ceph-osd
                  operator: In
                  values:
                  - enabled
          tolerations:
          - key: ceph-osd
            operator: Exists
        mgr:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: ceph-mgr
                  operator: In
                  values:
                  - enabled
          tolerations:
          - key: ceph-mgr
            operator: Exists   
      annotations:
      resources:
      removeOSDsIfOutAndSafeToRemove: false
      storage: 
        useAllNodes: false                                     #关闭使用所有Node
        useAllDevices: false                                   #关闭使用所有设备
        deviceFilter: sdb
        config:
            metadataDevice: 
            databaseSizeMB: "1024" 
            journalSizeMB: "1024"  
        nodes:
        - name: "node1"                                        #指定存储节点主机
          config:
            storeType: bluestore                               #指定类型为裸磁盘
          devices:                                             #指定磁盘为sdb
          - name: "sdb"
        - name: "node2"
          config:
            storeType: bluestore
          devices: 
          - name: "sdb"
        - name: "node3"
          config:
            storeType: bluestore
          devices:
          - name: "sdb"
      disruptionManagement:
        managePodBudgets: false
        osdMaintenanceTimeout: 30
        manageMachineDisruptionBudgets: false
        machineDisruptionBudgetNamespace: openshift-machine-api

7.安装cluster.yaml

kubectl apply -f cluster.yaml

8.查看状态

kubectl get pod -n rook-ceph

![img][img 3]

9.安装Toolbox

> toolbox是一个rook的工具集容器，该容器中的命令可以用来调试、测试Rook，对Ceph临时测试的操作一般在这个容器内执行。

kubectl apply -f toolbox.yaml
    kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"

![img][img 4]

10.测试Rook

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') sh

*  ceph status

![img][img 5]

*  ceph osd status
 *  ceph df

![img][img 6]

*  rados df
 *  ceph auth ls

![img][img 7]

![img][img 8]

*  ceph version

![img][img 9]

11.创建StorageClass

在提供(Provisioning)块存储之前，需要先创建StorageClass和存储池。K8S需要这两类资源，才能和Rook交互，进而分配持久卷(PV)。  
详解：如下配置文件中会创建一个名为replicapool的存储池，和rook-ceph-block的storageClass。

storageclass.yaml：

apiVersion: ceph.rook.io/v1
    kind: CephBlockPool
    metadata:
      name: replicapool
      namespace: rook-ceph
    spec:
      failureDomain: host
      replicated:
        size: 3
    ---
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
       name: rook-ceph-block
    provisioner: rook-ceph.rbd.csi.ceph.com
    parameters:
        clusterID: rook-ceph
        pool: replicapool
        imageFormat: "2"
        imageFeatures: layering
        csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
        csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
        csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
        csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
        csi.storage.k8s.io/fstype: ext4
    reclaimPolicy: Delete

12.安装storageclass.yaml

kubectl apply -f storageclass.yaml

![img][img 10]

13.创建PVC  
详解：这里创建相应的PVC，storageClassName:为基于rook Ceph集群的rook-ceph-block。

pvc.yaml：

apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: block-pvc
    spec:
      storageClassName: rook-ceph-block
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 200Mi

14.安装pvc.yaml

kubectl apply -f pvc.yaml

![img][img 11]

15.已创建dashboard，但仅使用clusterIP暴露服务，用官方提供的默认yaml可部署外部nodePort方式暴露服务的dashboard。

kubectl apply -f dashboard-external-https.yaml

dashboard-external-https.yaml：

apiVersion: v1
    kind: Service
    metadata:
      name: rook-ceph-mgr-dashboard-external-https
      namespace: rook-ceph
      labels:
        app: rook-ceph-mgr
        rook_cluster: rook-ceph
    spec:
      ports:
      - name: dashboard
        port: 8443
        protocol: TCP
        targetPort: 8443
      selector:
        app: rook-ceph-mgr
        rook_cluster: rook-ceph
      sessionAffinity: None
      type: NodePort

16.查看svc

kubectl get svc -n rook-ceph

![img][img 12]

17.查看密码并登陆

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}'  |  base64 --decode

![img][img 13]

https://ip:port

![img][img 14]

--------------------

参考链接：  
[https://rook.io/docs/rook/latest/Getting-Started/quickstart/][https_rook.io_docs_rook_latest_Getting-Started_quickstart]  
[https://blog.csdn.net/weixin\_40212316/article/details/115400377][https_blog.csdn.net_weixin_40212316_article_details_115400377]

[img]: https://img-blog.csdnimg.cn/img_convert/561a9d5f2aa336bfc21f543dab863e31.png
[img 1]: https://img-blog.csdnimg.cn/img_convert/873e5959a5f47d06ae59f5a9c34d637d.png
[img 2]: https://img-blog.csdnimg.cn/img_convert/a644c7d57719c0401c419ec33645629e.png
[img 3]: https://img-blog.csdnimg.cn/img_convert/37d67df3dfab35c52880b2ce284369c6.png
[img 4]: https://img-blog.csdnimg.cn/img_convert/a23b8b7b284ec595d2c8733c030443f1.png
[img 5]: https://img-blog.csdnimg.cn/img_convert/85cbc73c604b9b1c2e3b0ec3924bb6d2.png
[img 6]: https://img-blog.csdnimg.cn/img_convert/af978f9654df811033e0ec27a023e9f2.png
[img 7]: https://img-blog.csdnimg.cn/img_convert/8648f4c4dc46a6641d9813f334161784.png
[img 8]: https://img-blog.csdnimg.cn/img_convert/e730490e130a3ec83aed4ad73be64b16.png
[img 9]: https://img-blog.csdnimg.cn/img_convert/09283fe2ccab1a4a492fafe55581d602.png
[img 10]: https://img-blog.csdnimg.cn/img_convert/9bdbbeca1ca04c782189b6cbf0b8d69e.png
[img 11]: https://img-blog.csdnimg.cn/img_convert/0dc94357260c8534c83ca23f7a4cdd91.png
[img 12]: https://img-blog.csdnimg.cn/img_convert/f95912b3323abe00b360905848b26a93.png
[img 13]: https://img-blog.csdnimg.cn/img_convert/2999c0d292093be94cd2ed0ce00c0f2a.png
[img 14]: https://img-blog.csdnimg.cn/img_convert/a864102a8a3ffeda25d2ad0c8a0c19eb.png
[https_rook.io_docs_rook_latest_Getting-Started_quickstart]: https://rook.io/docs/rook/latest/Getting-Started/quickstart/
[https_blog.csdn.net_weixin_40212316_article_details_115400377]: https://blog.csdn.net/weixin_40212316/article/details/115400377