使用rook部署ceph_如何使用Rook在Kubernetes中设置Ceph集群

以你之姓@ 2022-11-28 15:26 717阅读 0赞

使用rook部署ceph

*The author selected the [Mozilla Foundation][] to receive a donation as part of the [Write for DOnations][] program.*

*作者选择[Mozilla基金会][Mozilla Foundation]作为[Write for DOnations][]计划的一部分接受捐赠。*

### 介绍 **(**Introduction**)** ###

[Kubernetes][] containers are *stateless* as a core principle, but data must still be managed, preserved, and made accessible to other services. Stateless means that the container is running in isolation without any knowledge of past transactions, which makes it easy to replace, delete, or distribute the container. However, it also means that data will be lost for certain lifecycle events like restart or deletion.

[Kubernetes][]容器是*无状态*的核心原则，但是数据仍然必须被管理，保留并使其可被其他服务访问。 无状态意味着容器正在孤立运行，无需任何有关过去事务的知识，这使得替换，删除或分发容器变得容易。 但是，这也意味着对于某些生命周期事件(如重新启动或删除)，数据将丢失。

[Rook][] is a storage orchestration tool that provides a cloud-native, open source solution for a diverse set of storage providers. Rook uses the power of Kubernetes to turn a storage system into self-managing services that provide a seamless experience for saving Kubernetes application or deployment data.

[Rook][]是一种存储编排工具，可为各种存储提供商提供云原生的开源解决方案。 Rook利用Kubernetes的功能将存储系统转变为自我管理服务，这些服务提供了无缝的体验来保存Kubernetes应用程序或部署数据。

[Ceph][] is a highly scalable distributed-storage solution offering object, block, and file storage. Ceph clusters are designed to run on any hardware using the so-called [*CRUSH algorithm* (Controlled Replication Under Scalable Hashing).][CRUSH algorithm_ _Controlled Replication Under Scalable Hashing_.]

[Ceph][]是一种高度可扩展的分布式存储解决方案，提供对象，块和文件存储。 Ceph集群被设计为使用所谓的[*CRUSH算法* (可伸缩哈希下的受控复制)][CRUSH algorithm_ _Controlled Replication Under Scalable Hashing_.]在任何硬件上运行[。][CRUSH algorithm_ _Controlled Replication Under Scalable Hashing_.]

One main benefit of this deployment is that you get the highly scalable storage solution of Ceph without having to configure it manually using the Ceph command line, because Rook automatically handles it. Kubernetes applications can then mount block devices and filesystems from Rook to preserve and monitor their application data.

这种部署的一个主要好处是，您可以获得Ceph的高度可扩展的存储解决方案，而无需使用Ceph命令行手动配置它，因为Rook会自动处理它。 然后，Kubernetes应用程序可以从Rook挂载块设备和文件系统，以保留和监视其应用程序数据。

In this tutorial, you will set up a Ceph cluster using Rook and use it to persist data for a MongoDB database as an example.

在本教程中，您将使用Rook设置一个Ceph集群，并以其为MongoDB数据库持久化数据。

## 先决条件 **(**Prerequisites**)** ##

Before you begin this guide, you’ll need the following:

在开始本指南之前，您需要满足以下条件：

*  A DigitalOcean Kubernetes cluster with at least three nodes that each have 2 vCPUs and 4 GB of Memory. To create a cluster on DigitalOcean and connect to it, see the [Kubernetes Quickstart][].
    
    一个DigitalOcean Kubernetes集群，至少具有三个节点，每个节点具有2个vCPU和4 GB的内存。 要在DigitalOcean上创建集群并连接到集群，请参见[Kubernetes Quickstart][] 。
 *  The kubectl command-line tool installed on a development server and configured to connect to your cluster. You can read more about installing kubectl in its [official documentation][].
    
    kubectl命令行工具安装在开发服务器上，并配置为连接到集群。 您可以在其[官方文档中][official documentation]阅读有关安装kubectl的更多信息。
 *  A [DigitalOcean block storage][] Volume with at least 100 GB for each node of the cluster you just created—for example, if you have three nodes you will need three Volumes. Select **Manually Format** rather than automatic and then attach your Volume to the Droplets in your node pool. You can follow the [Volumes Quickstart][] to achieve this.
    
    每个刚创建的群集的每个节点至少具有100 GB的[DigitalOcean块存储][DigitalOcean block storage]卷-例如，如果您有三个节点，则需要三个卷。 选择“ **手动格式化”**而不是“自动**格式化”** ，然后将卷附加到节点池中的Droplet。 您可以按照[Volumes Quickstart][]来实现。

## 第1步-设置Rook **(**Step 1 — Setting up Rook**)** ##

After completing the prerequisite, you have a fully functional Kubernetes cluster with three nodes and three Volumes—you’re now ready to set up Rook.

完成前提条件后，您将拥有一个具有三个节点和三个卷的功能齐全的Kubernetes集群，现在您可以设置Rook。

In this section, you will clone the Rook repository, deploy your first Rook *operator* on your Kubernetes cluster, and validate the given deployment status. A Rook operator is a container that automatically bootstraps the storage clusters and monitors the storage daemons to ensure the storage clusters are healthy.

在本节中，您将克隆Rook存储库，在Kubernetes集群上部署第一个Rook *运算符* ，并验证给定的部署状态。 Rook运算符是一个容器，可以自动引导存储集群并监视存储后台驻留程序以确保存储集群运行状况良好。

First, you will clone the Rook repository, so you have all the resources needed to start setting up your Rook cluster:

首先，您将克隆Rook存储库，因此您拥有开始设置Rook集群所需的所有资源：

*  git clone --single-branch --branch release-1.3 https://github.com/rook/rook.git
    
    git clone --single-branch --branch版本-1.3 https://github.com/rook/rook.git

This command will clone the Rook repository from Github and create a folder with the name of `rook` in your directory. Now enter the directory using the following command:

该命令将从Github克隆Rook存储库，并在目录中创建一个名为`rook`的文件夹。 现在，使用以下命令输入目录：

*  cd rook/cluster/examples/kubernetes/ceph
    
    cd rook / cluster / examples / kubernetes / ceph

Next you will continue by creating the common resources you needed for your Rook deployment, which you can do by deploying the Kubernetes config file that is available by default in the directory:

接下来，您将继续创建Rook部署所需的公共资源，这可以通过部署Kubernetes配置文件来完成，该文件默认情况下在目录中可用：

*  kubectl create -f common.yaml
    
    kubectl创建-f common.yaml

The resources you’ve created are mainly [CustomResourceDefinitions][] (CRDs) and define new resources that the operator will later use. They contain resources like the ServiceAccount, Role, RoleBinding, ClusterRole, and ClusterRoleBinding.

您创建的资源主要是[CustomResourceDefinitions][] (CRD)，并定义了操作员以后将使用的新资源。 它们包含诸如ServiceAccount，Role，RoleBinding，ClusterRole和ClusterRoleBinding之类的资源。

**Note:** This standard file assumes that you will deploy the Rook operator and all Ceph daemons in the same namespace. If you want to deploy the operator in a separate namespace, see the comments throughout the `common.yaml` file.

**注意：**此标准文件假定您将Rook运算符和所有Ceph守护程序部署在同一名称空间中。 如果要在单独的命名空间中部署操作员，请参阅`common.yaml`文件中的注释。

After the common resources are created, the next step is to create the Rook operator.

创建公用资源后，下一步是创建Rook运算符。

Before deploying the `operator.yaml` file, you will need to change the `CSI_RBD_GRPC_METRICS_PORT` variable because your DigitalOcean Kubernetes cluster already uses the standard port by default. Open the file with the following command:

在部署`operator.yaml`文件之前，您将需要更改`CSI_RBD_GRPC_METRICS_PORT`变量，因为默认情况下，DigitalOcean Kubernetes集群已使用标准端口。 使用以下命令打开文件：

*  nano operator.yaml
    
    纳米算子

Then search for the `CSI_RBD_GRPC_METRICS_PORT` variable, uncomment it by removing the `#`, and change the value from port `9001` to `9093`:

然后搜索`CSI_RBD_GRPC_METRICS_PORT`变量，通过删除`#`取消注释，并将值从端口`9001`更改为`9093` ：

operator.yaml

运算符

kind: ConfigMap
    apiVersion: v1
    metadata:
      name: rook-ceph-operator-config
      namespace: rook-ceph
    data:
      ROOK_CSI_ENABLE_CEPHFS: "true"
      ROOK_CSI_ENABLE_RBD: "true"
      ROOK_CSI_ENABLE_GRPC_METRICS: "true"
      CSI_ENABLE_SNAPSHOTTER: "true"
      CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
      ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"
      # Configure CSI CSI Ceph FS grpc and liveness metrics port
      # CSI_CEPHFS_GRPC_METRICS_PORT: "9091"
      # CSI_CEPHFS_LIVENESS_METRICS_PORT: "9081"
      # Configure CSI RBD grpc and liveness metrics port
      CSI_RBD_GRPC_METRICS_PORT: "9093"
      # CSI_RBD_LIVENESS_METRICS_PORT: "9080"

Once you’re done, save and exit the file.

完成后，保存并退出文件。

Next, you can deploy the operator using the following command:

接下来，您可以使用以下命令部署操作员：

*  kubectl create -f operator.yaml
    
    kubectl创建-f operator.yaml

The command will output the following:

该命令将输出以下内容：

Output configmap/rook-ceph-operator-config created
    deployment.apps/rook-ceph-operator created

Again, you’re using the `kubectl create` command with the `-f` flag to assign the file that you want to apply. It will take around a couple of seconds for the operator to be running. You can verify the status using the following command:

同样，您使用带有`-f`标志的`kubectl create`命令来分配要应用的文件。 操作员将需要大约几秒钟的时间运行。 您可以使用以下命令来验证状态：

*  kubectl get pod -n rook-ceph
    
    kubectl获取pod -n rook-ceph

You use the `-n` flag to get the pods of a specific Kubernetes namespace (`rook-ceph` in this example).

您可以使用`-n`标志来获取特定Kubernetes名称空间的Pod(在此示例中为`rook-ceph` ceph)。

Once the operator deployment is ready, it will trigger the creation of the DeamonSets that are in charge of creating the `rook-discovery` agents on each worker node of your cluster. You’ll receive output similar to:

一旦操作员部署准备就绪，它将触发DeamonSets的创建，这些DeamonSets负责在集群的每个工作节点上创建`rook-discovery`代理。 您将收到类似于以下内容的输出：

Output NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-599765ff49-fhbz9   1/1     Running   0          92s
    rook-discover-6fhlb                   1/1     Running   0          55s
    rook-discover-97kmz                   1/1     Running   0          55s
    rook-discover-z5k2z                   1/1     Running   0          55s

You have successfully installed Rook and deployed your first operator. Next, you will create a Ceph cluster and verify that it is working.

您已成功安装Rook并部署了您的第一个操作员。 接下来，您将创建一个Ceph集群并验证其是否正常运行。

## 第2步-创建Ceph集群 **(**Step 2 — Creating a Ceph Cluster**)** ##

Now that you have successfully set up Rook on your Kubernetes cluster, you’ll continue by creating a Ceph cluster within the Kubernetes cluster and verifying its functionality.

现在您已经成功在Kubernetes集群上设置了Rook，接下来将在Kubernetes集群中创建一个Ceph集群并验证其功能。

First let’s review the most important Ceph components and their functionality:

首先，让我们回顾最重要的Ceph组件及其功能：

*  *Ceph Monitors*, also known as MONs, are responsible for maintaining the maps of the cluster required for the Ceph daemons to coordinate with each other. There should always be more than one MON running to increase the reliability and availability of your storage service.
    
    *Ceph监视器* ，也称为MON，负责维护Ceph守护程序相互协调所需的集群映射。 为了确保存储服务的可靠性和可用性，应始终运行多个MON。
 *  *Ceph Managers*, also known as MGRs, are runtime daemons responsible for keeping track of runtime metrics and the current state of your Ceph cluster. They run alongside your monitoring daemons (MONs) to provide additional monitoring and an interface to external monitoring and management systems.
    
    *Ceph管理器* (也称为MGR)是运行时守护程序，负责跟踪运行时指标和Ceph集群的当前状态。 它们与您的监视守护程序(MON)一起运行，以提供其他监视以及与外部监视和管理系统的接口。
 *  *Ceph Object Store Devices*, also known as OSDs, are responsible for storing objects on a local file system and providing access to them over the network. These are usually tied to one physical disk of your cluster. Ceph clients interact with OSDs directly.
    
    *Ceph对象存储设备* (也称为OSD)负责将对象存储在本地文件系统上，并通过网络提供对它们的访问。 这些通常绑定到群集的一个物理磁盘。 Ceph客户端直接与OSD交互。

To interact with the data of your Ceph storage, a client will first make contact with the Ceph Monitors (MONs) to obtain the current version of the *cluster map*. The cluster map contains the data storage location as well as the cluster topology. The Ceph clients then use the cluster map to decide which OSD they need to interact with.

为了与您的Ceph存储中的数据进行交互，客户端将首先与Ceph监视器(MONs)联系以获得*集群映射*的当前版本。 群集图包含数据存储位置以及群集拓扑。 然后，Ceph客户端使用群集映射表来确定与之交互的OSD。

Rook enables Ceph storage to run on your Kubernetes cluster. All of these components are running in your Rook cluster and will directly interact with the Rook agents. This provides a more streamlined experience for administering your Ceph cluster by hiding Ceph components like placement groups and storage maps while still providing the options of advanced configurations.

Rook使Ceph存储可以在您的Kubernetes集群上运行。 所有这些组件都在您的Rook集群中运行，并将直接与Rook代理进行交互。 通过隐藏Ceph组件(如放置组和存储图)，同时仍提供高级配置选项，这为管理Ceph集群提供了更为简化的体验。

Now that you have a better understanding of what Ceph is and how it is used in Rook, you will continue by setting up your Ceph cluster.

现在您对Ceph是什么以及在Rook中如何使用有了更好的了解，您将继续设置Ceph集群。

You can complete the setup by either running the example configuration, found in the `examples` directory of the Rook project, or by writing your own configuration. The example configuration is fine for most use cases and provides excellent documentation of optional parameters.

您可以通过运行位于Rook项目的`examples`目录中的示例配置或编写自己的配置来完成设置。 示例配置适合大多数用例，并提供了有关可选参数的出色文档。

Now you’ll start the creation process of a Ceph cluster [Kubernetes Object][].

现在，您将开始创建Ceph集群[Kubernetes Object的过程][Kubernetes Object] 。

First, you need to create a YAML file:

首先，您需要创建一个YAML文件：

*  nano cephcluster.yaml
    
    纳米头孢类

The configuration defines how the Ceph cluster will be deployed. In this example, you will deploy three Ceph Monitors (MON) and enable the Ceph dashboard. The [Ceph dashboard][] is out of scope for this tutorial, but you can use it later in your own individual project for visualizing the current status of your Ceph cluster.

该配置定义了如何部署Ceph集群。 在此示例中，您将部署三个Ceph监视器(MON)并启用Ceph仪表板。 [Ceph仪表板][Ceph dashboard]不在本教程的讨论范围之内，但是您以后可以在自己的单个项目中使用它来可视化Ceph集群的当前状态。

Add the following content to define the `apiVersion` and the Kubernetes Object `kind` as well as the `name` and the `namespace` the Object should be deployed in:

添加以下内容以定义`apiVersion`和Kubernetes对象的`kind`以及应在其中部署对象的`name`和`namespace` ：

cephcluster.yaml

apiVersion: ceph.rook.io/v1
    kind: CephCluster
    metadata:
      name: rook-ceph
      namespace: rook-ceph

After that, add the `spec` key, which defines the model that Kubernetes will use to create your Ceph cluster. You’ll first define the image version you want to use and whether you allow unsupported Ceph versions or not:

之后，添加`spec`密钥，该密钥定义Kubernetes将用来创建Ceph集群的模型。 首先，您要定义要使用的映像版本，以及是否允许使用不受支持的Ceph版本：

cephcluster.yaml

spec:
      cephVersion:
        image: ceph/ceph:v14.2.8
        allowUnsupported: false

Then set the data directory where configuration files will be persisted using the `dataDirHostPath` key:

然后使用`dataDirHostPath`项设置将配置文件`dataDirHostPath`在其中的数据目录：

cephcluster.yaml

dataDirHostPath: /var/lib/rook

Next, you define if you want to skip upgrade checks and when you want to upgrade your cluster using the following parameters:

接下来，使用以下参数定义是否要跳过升级检查以及何时要升级群集：

cephcluster.yaml

skipUpgradeChecks: false
      continueUpgradeAfterChecksEvenIfNotHealthy: false

You configure the number of Ceph Monitors (MONs) using the `mon` key. You also allow the deployment of multiple MONs per node:

您使用`mon`键配置Ceph监视器(MON)的数量。 您还允许每个节点部署多个MON：

cephcluster.yaml

mon:
        count: 3
        allowMultiplePerNode: false

Options for the Ceph dashboard are defined under the `dashboard` key. This gives you options to enable the dashboard, customize the port, and prefix it when using a reverse proxy:

Ceph仪表板的选项在`dashboard`键下定义。 这为您提供了启用仪表板，自定义端口并在使用反向代理时为其添加前缀的选项：

cephcluster.yaml

dashboard:
        enabled: true
        # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
        # urlPrefix: /ceph-dashboard
        # serve the dashboard at the given port.
        # port: 8443
        # serve the dashboard using SSL
        ssl: false

You can also enable monitoring of your cluster with the `monitoring` key (monitoring requires [Prometheus to be pre-installed][]):

您还可以使用`monitoring`键启用对群集的`monitoring` (监视需要[预安装Prometheus][Prometheus to be pre-installed] )：

cephcluster.yaml

monitoring:
        enabled: false
        rulesNamespace: rook-ceph

RDB stands for RADOS (Reliable Autonomic Distributed Object Store) block device, which are thin-provisioned and resizable Ceph block devices that store data on multiple nodes.

RDB代表RADOS(可靠的自主分布对象存储)块设备，它是在多个节点上存储数据的精简配置和可调整大小的Ceph块设备。

RBD images can be asynchronously shared between two Ceph clusters by enabling `rbdMirroring`. Since we’re working with one cluster in this tutorial, this isn’t necessary. The number of workers is therefore set to `0`:

通过启用`rbdMirroring` ，可以在两个Ceph集群之间异步共享RBD图像。 由于我们在本教程中使用的是一个集群，因此这不是必需的。 因此，工人人数设置为`0` ：

cephcluster.yaml

rbdMirroring:
        workers: 0

You can enable the crash collector for the Ceph daemons:

您可以为Ceph守护程序启用崩溃收集器：

cephcluster.yaml

crashCollector:
        disable: false

The cleanup policy is only important if you want to delete your cluster. That is why this option has to be left empty:

仅当您要删除集群时，清理策略才重要。 这就是为什么必须将此选项留空的原因：

cephcluster.yaml

cleanupPolicy:
        deleteDataDirOnHosts: ""
      removeOSDsIfOutAndSafeToRemove: false

The `storage` key lets you define the cluster level storage options; for example, which node and devices to use, the database size, and how many OSDs to create per device:

使用`storage`键，您可以定义集群级别的存储选项。 例如，要使用的节点和设备，数据库大小以及每个设备要创建多少OSD：

cephcluster.yaml

storage:
        useAllNodes: true
        useAllDevices: true
        config:
          # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
          # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
          # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller

You use the `disruptionManagement` key to manage daemon disruptions during upgrade or fencing:

您可以使用`disruptionManagement`键来管理升级或隔离期间的守护程序中断：

cephcluster.yaml

disruptionManagement:
        managePodBudgets: false
        osdMaintenanceTimeout: 30
        manageMachineDisruptionBudgets: false
        machineDisruptionBudgetNamespace: openshift-machine-api

These configuration blocks will result in the final following file:

这些配置块将生成最终的以下文件：

cephcluster.yaml

apiVersion: ceph.rook.io/v1
    kind: CephCluster
    metadata:
      name: rook-ceph
      namespace: rook-ceph
    spec:
      cephVersion:
        image: ceph/ceph:v14.2.8
        allowUnsupported: false
      dataDirHostPath: /var/lib/rook
      skipUpgradeChecks: false
      continueUpgradeAfterChecksEvenIfNotHealthy: false
      mon:
        count: 3
        allowMultiplePerNode: false
      dashboard:
        enabled: true
        # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
        # urlPrefix: /ceph-dashboard
        # serve the dashboard at the given port.
        # port: 8443
        # serve the dashboard using SSL
        ssl: false
      monitoring:
        enabled: false
        rulesNamespace: rook-ceph
      rbdMirroring:
        workers: 0
      crashCollector:
        disable: false
      cleanupPolicy:
        deleteDataDirOnHosts: ""
      removeOSDsIfOutAndSafeToRemove: false
      storage:
        useAllNodes: true
        useAllDevices: true
        config:
          # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
          # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
          # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller
      disruptionManagement:
        managePodBudgets: false
        osdMaintenanceTimeout: 30
        manageMachineDisruptionBudgets: false
        machineDisruptionBudgetNamespace: openshift-machine-api

Once you’re done, save and exit your file.

完成后，保存并退出文件。

You can also customize your deployment by, for example changing your database size or defining a custom port for the dashboard. You can find more options for your cluster deployment in the [cluster example][] of the Rook repository.

您还可以通过例如更改数据库大小或为仪表板定义自定义端口来自定义部署。 您可以在Rook存储库的[集群示例][cluster example]中找到更多用于集群部署的选项。

Next, apply this manifest in your Kubernetes cluster:

接下来，将此清单应用于您的Kubernetes集群：

*  kubectl apply -f cephcluster.yaml
    
    kubectl应用-f cephcluster.yaml

Now check that the pods are running:

现在检查Pod是否正在运行：

*  kubectl get pod -n rook-ceph
    
    kubectl获取pod -n rook-ceph

This usually takes a couple of minutes, so just refresh until your output reflects something like the following:

这通常需要几分钟，因此请刷新直到输出反映如下内容：

Output NAME                                                   READY   STATUS    RESTARTS   AGE
    csi-cephfsplugin-lz6dn                                 3/3     Running   0          3m54s
    csi-cephfsplugin-provisioner-674847b584-4j9jw          5/5     Running   0          3m54s
    csi-cephfsplugin-provisioner-674847b584-h2cgl          5/5     Running   0          3m54s
    csi-cephfsplugin-qbpnq                                 3/3     Running   0          3m54s
    csi-cephfsplugin-qzsvr                                 3/3     Running   0          3m54s
    csi-rbdplugin-kk9sw                                    3/3     Running   0          3m55s
    csi-rbdplugin-l95f8                                    3/3     Running   0          3m55s
    csi-rbdplugin-provisioner-64ccb796cf-8gjwv             6/6     Running   0          3m55s
    csi-rbdplugin-provisioner-64ccb796cf-dhpwt             6/6     Running   0          3m55s
    csi-rbdplugin-v4hk6                                    3/3     Running   0          3m55s
    rook-ceph-crashcollector-pool-33zy7-68cdfb6bcf-9cfkn   1/1     Running   0          109s
    rook-ceph-crashcollector-pool-33zyc-565559f7-7r6rt     1/1     Running   0          53s
    rook-ceph-crashcollector-pool-33zym-749dcdc9df-w4xzl   1/1     Running   0          78s
    rook-ceph-mgr-a-7fdf77cf8d-ppkwl                       1/1     Running   0          53s
    rook-ceph-mon-a-97d9767c6-5ftfm                        1/1     Running   0          109s
    rook-ceph-mon-b-9cb7bdb54-lhfkj                        1/1     Running   0          96s
    rook-ceph-mon-c-786b9f7f4b-jdls4                       1/1     Running   0          78s
    rook-ceph-operator-599765ff49-fhbz9                    1/1     Running   0          6m58s
    rook-ceph-osd-prepare-pool-33zy7-c2hww                 1/1     Running   0          21s
    rook-ceph-osd-prepare-pool-33zyc-szwsc                 1/1     Running   0          21s
    rook-ceph-osd-prepare-pool-33zym-2p68b                 1/1     Running   0          21s
    rook-discover-6fhlb                                    1/1     Running   0          6m21s
    rook-discover-97kmz                                    1/1     Running   0          6m21s
    rook-discover-z5k2z                                    1/1     Running   0          6m21s

You have now successfully set up your Ceph cluster and can continue by creating your first storage block.

现在，您已经成功设置了Ceph集群，并且可以通过创建第一个存储块来继续。

## 步骤3 —添加块存储 **(**Step 3 — Adding Block Storage**)** ##

Block storage allows a single pod to mount storage. In this section, you will create a storage block that you can use later in your applications.

块存储允许单个容器安装存储。 在本节中，您将创建一个存储块，以后可以在您的应用程序中使用它。

Before Ceph can provide storage to your cluster, you first need to create a `storageclass` and a `cephblockpool`. This will allow Kubernetes to interoperate with Rook when creating [persistent volumes][]:

在Ceph可以为您的集群提供存储之前，您首先需要创建一个`storageclass`和一个`cephblockpool` 。 创建[持久卷][persistent volumes]时，这将允许Kubernetes与Rook进行互操作：

*  kubectl apply -f ./csi/rbd/storageclass.yaml
    
    kubectl apply -f ./csi/rbd/storageclass.yaml

The command will output the following:

该命令将输出以下内容：

Output cephblockpool.ceph.rook.io/replicapool created
    storageclass.storage.k8s.io/rook-ceph-block created

**Note:** If you’ve deployed the Rook operator in a namespace other than `rook-ceph` you need to change the prefix in the provisioner to match the namespace you use.

**注意：**如果您已将Rook运算符部署在`rook-ceph`以外的名称空间中，则需要在`rook-ceph`中更改前缀以匹配您使用的名称空间。

After successfully deploying the `storageclass` and `cephblockpool`, you will continue by defining the [PersistentVolumeClaim (PVC)][PersistentVolumeClaim _PVC] for your application. A PersistentVolumeClaim is a resource used to request storage from your cluster.

成功部署`storageclass`和`cephblockpool` ，您将继续为应用程序定义[PersistentVolumeClaim(PVC)][PersistentVolumeClaim _PVC] 。 PersistentVolumeClaim是用于从群集请求存储的资源。

For that, you first need to create a YAML file:

为此，您首先需要创建一个YAML文件：

*  nano pvc-rook-ceph-block.yaml
    
    纳米pvc-rook-ceph-block.yaml

Add the following for your PersistentVolumeClaim:

为您的PersistentVolumeClaim添加以下内容：

pvc-rook-ceph-block.yaml

apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: mongo-pvc
    spec:
      storageClassName: rook-ceph-block
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi

First, you need to set an `apiVersion` (`v1` is the current stable version). Then you need to tell Kubernetes which type of resource you want to define using the `kind` key (`PersistentVolumeClaim` in this case).

首先，您需要设置一个`apiVersion` ( `v1`是当前的稳定版本)。 然后，您需要使用`kind`键(在这种情况下为`PersistentVolumeClaim` )告诉Kubernetes您要定义哪种资源类型。

The `spec` key defines the model that Kubernetes will use to create your PersistentVolumeClaim. Here you need to select the storage class you created earlier: `rook-ceph-block`. You can then define the access mode and limit the resources of the claim. `ReadWriteOnce` means the volume can only be mounted by a single node.

`spec`键定义了Kubernetes将用来创建PersistentVolumeClaim的模型。 在这里，您需要选择之前创建的存储类： `rook-ceph-block` 。 然后，您可以定义访问方式并限制索赔的资源。 `ReadWriteOnce`表示该卷只能由单个节点装入。

Now that you have defined the PersistentVolumeClaim, it is time to deploy it using the following command:

现在您已经定义了PersistentVolumeClaim，是时候使用以下命令来部署它了：

*  kubectl apply -f pvc-rook-ceph-block.yaml
    
    kubectl应用-f pvc-rook-ceph-block.yaml

You will receive the following output:

您将收到以下输出：

Output persistentvolumeclaim/mongo-pvc created

You can now check the status of your PVC:

现在，您可以检查PVC的状态：

*  kubectl get pvc
    
    kubectl获取pvc

When the PVC is bound, you are ready:

绑定PVC后，您就可以准备：

Output NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
    mongo-pvc   Bound    pvc-ec1ca7d1-d069-4d2a-9281-3d22c10b6570   5Gi        RWO            rook-ceph-block   16s

You have now successfully created a storage class and used it to create a `PersistenVolumeClaim` that you will mount to a application to persist data in the next section.

现在，您已经成功创建了一个存储类，并使用它创建了一个`PersistenVolumeClaim` ，将其安装到应用程序以在下一部分中保留数据。

## 步骤4 —使用rook-ceph-block创建MongoDB部署 **(**Step 4 — Creating a MongoDB Deployment with a rook-ceph-block**)** ##

Now that you have successfully created a storage block and a persistent volume, you will put it to use by implementing it in a MongoDB application.

现在，您已经成功创建了存储块和持久卷，您将通过在MongoDB应用程序中实现它来使用它。

The configuration will contain a few things:

该配置将包含一些内容：

*  A single container deployment based on the latest version of the `mongo` image.
    
    基于最新版本的`mongo`映像的单个容器部署。
 *  A persistent volume to preserve the data of the MongoDB database.
    
    用于保留MongoDB数据库数据的持久卷。
 *  A service to expose the MongoDB port on port `31017` of every node so you can interact with it later.
    
    公开每个节点的端口`31017`上的MongoDB端口的服务，以便您以后可以与其进行交互。

First open the configuration file:

首先打开配置文件：

*  nano mongo.yaml
    
    纳米mongo.yaml

Start the manifest with the `Deployment` resource:

使用`Deployment`资源启动清单：

mongo.yaml

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: mongo
    spec:
      selector:
        matchLabels:
          app: mongo
      template:
        metadata:
          labels:
            app: mongo
        spec:
          containers:
          - image: mongo:latest
            name: mongo
            ports:
            - containerPort: 27017
              name: mongo
            volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
          volumes:
          - name: mongo-persistent-storage
            persistentVolumeClaim:
              claimName: mongo-pvc
    
    ...

For each resource in the manifest, you need to set an `apiVersion`. For deployments and services, use `apiVersion: apps/v1`, which is a stable version. Then, tell Kubernetes which resource you want to define using the `kind` key. Each definition should also have a name defined in `metadata.name`.

对于清单中的每个资源，您需要设置一个`apiVersion` 。 对于部署和服务，请使用`apiVersion: apps/v1` ，这是一个稳定的版本。 然后，使用`kind`键告诉Kubernetes您要定义哪个资源。 每个定义还应该在`metadata.name`定义一个名称。

The `spec` section tells Kubernetes what the desired state of your final state of the deployment is. This definition requests that Kubernetes should create one pod with one replica.

`spec`部分告诉Kubernetes您最终部署状态的期望状态是什么。 该定义要求Kubernetes应该使用一个副本创建一个Pod。

Labels are key-value pairs that help you organize and cross-reference your Kubernetes resources. You can define them using `metadata.labels` and you can later search for them using `selector.matchLabels`.

标签是键值对，可帮助您组织和交叉引用Kubernetes资源。 您可以使用`metadata.labels`定义它们，以后可以使用`selector.matchLabels`搜索它们。

The `spec.template` key defines the model that Kubernetes will use to create each of your pods. Here you will define the specifics of your pod’s deployment like the image name, container ports, and the volumes that should be mounted. The image will then automatically be pulled from an image registry by Kubernetes.

`spec.template`键定义了Kubernetes将用来创建每个Pod的模型。 在这里，您将定义Pod部署的细节，例如映像名称，容器端口和应挂载的卷。 然后，Kubernetes将自动从映像注册表中提取该映像。

Here you will use the PersistentVolumeClaim you created earlier to persist the data of the `/data/db` directory of the pods. You can also specify extra information like environment variables that will help you with further customizing your deployment.

在这里，您将使用之前创建的PersistentVolumeClaim来持久保存Pod的`/data/db`目录中的`/data/db` 。 您还可以指定其他信息，例如环境变量，这些信息将帮助您进一步自定义部署。

Next, add the following code to the file to define a Kubernetes `Service` that exposes the MongoDB port on port `31017` of every node in your cluster:

接下来，将以下代码添加到文件中以定义Kubernetes `Service` ，该`Service`公开集群中每个节点的端口`31017`上的MongoDB端口：

mongo.yaml

...
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: mongo
      labels:
        app: mongo
    spec:
      selector:
        app: mongo
      type: NodePort
      ports:
        - port: 27017
          nodePort: 31017

Here you also define an `apiVersion`, but instead of using the `Deployment` type, you define a `Service`. The service will receive connections on port `31017` and forward them to the pods’ port `27017`, where you can then access the application.

在这里，您还定义了一个`apiVersion` ，但是没有使用`Deployment`类型，而是定义了`Service` 。 该服务将在端口`31017`上接收连接，并将它们转发到Pod的端口`27017` ，然后您可以在其中访问应用程序。

The service uses `NodePort` as the service type, which will expose the `Service` on each Node’s IP at a static port between `30000` and `32767` (`31017` in this case).

该服务使用`NodePort`作为服务类型，这将在`30000`到`32767`之间的静态端口(在本例中为`31017` )的每个节点的IP上公开该`Service` 。

Now that you have defined the deployment, it is time to deploy it:

现在，您已经定义了部署，是时候部署它了：

*  kubectl apply -f mongo.yaml
    
    kubectl应用-f mongo.yaml

You will see the following output:

您将看到以下输出：

Output deployment.apps/mongo created
    service/mongo created

You can check the status of the deployment and service:

您可以检查部署和服务的状态：

*  kubectl get svc,deployments
    
    kubectl获取svc，部署

The output will be something like this:

输出将是这样的：

Output NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
    service/kubernetes   ClusterIP   10.245.0.1       <none>        443/TCP           33m
    service/mongo        NodePort    10.245.124.118   <none>        27017:31017/TCP   4m50s
    
    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/mongo   1/1     1            1           4m50s

After the deployment is ready, you can start saving data into your database. The easiest way to do so is by using the MongoDB shell, which is included in the MongoDB pod you just started. You can open it using kubectl.

部署准备就绪后，您可以开始将数据保存到数据库中。 最简单的方法是使用MongoDB shell，该外壳包含在刚启动的MongoDB pod中。 您可以使用kubectl打开它。

For that you are going to need the name of the pod, which you can get using the following command:

为此，您将需要吊舱的名称，可以使用以下命令获得该吊舱的名称：

*  kubectl get pods
    
    kubectl得到豆荚

The output will be similar to this:

输出将类似于以下内容：

Output NAME                     READY   STATUS    RESTARTS   AGE
    mongo-7654889675-mjcks   1/1     Running   0          13m

Now copy the name and use it in the `exec` command:

现在复制名称，并在`exec`命令中使用它：

*  kubectl exec -it your\_pod\_name mongo
    
    kubectl exec -it your\_pod\_name mongo

Now that you are in the MongoDB shell let’s continue by creating a database:

现在您已经在MongoDB Shell中，让我们继续创建数据库：

*  use test
    
    使用测试

The `use` command switches between databases or creates them if they don’t exist.

`use`命令在数据库之间切换或创建数据库(如果不存在)。

Output switched to db test

Then insert some data into your new `test` database. You use the `insertOne()` method to insert a new document in the created database:

然后将一些数据插入新的`test`数据库。 您可以使用`insertOne()`方法在创建的数据库中插入新文档：

*  db.test.insertOne( \{name: "test", number: 10 \})
    
    db.test.insertOne(\{name：“ test”，number：10\})

Output {
        "acknowledged" : true,
        "insertedId" : ObjectId("5f22dd521ba9331d1a145a58")
    }

The next step is retrieving the data to make sure it is saved, which can be done using the `find` command on your collection:

下一步是检索数据以确保已保存，这可以使用集合上的`find`命令来完成：

*  db.getCollection("test").find()
    
    db.getCollection(“ test”)。find()

The output will be similar to this:

输出将类似于以下内容：

Output NAME                     READY   STATUS    RESTARTS   AGE
    { "_id" : ObjectId("5f1b18e34e69b9726c984c51"), "name" : "test", "number" : 10 }

Now that you have saved some data into the database, it will be persisted in the underlying Ceph volume structure. One big advantage of this kind of deployment is the *dynamic provisioning* of the volume. Dynamic provisioning means that applications only need to request the storage and it will be automatically provided by Ceph instead of developers creating the storage manually by sending requests to their storage providers.

现在您已将一些数据保存到数据库中，这些数据将保留在基础Ceph卷结构中。 这种部署的一大优势是*动态配置*卷。 动态预配置意味着应用程序只需要请求存储，它将由Ceph自动提供，而不是开发人员通过将请求发送到其存储提供程序来手动创建存储。

Let’s validate this functionality by restarting the pod and checking if the data is still there. You can do this by deleting the pod, because it will be restarted to fulfill the state defined in the deployment:

让我们通过重新启动Pod并检查数据是否仍然存在来验证此功能。 您可以通过删除pod来完成此操作，因为它将重新启动以实现部署中定义的状态：

*  kubectl delete pod -l app=mongo
    
    kubectl删除pod -l app = mongo

Now let’s validate that the data is still there by connecting to the MongoDB shell and printing out the data. For that you first need to get your pod’s name and then use the `exec` command to open the MongoDB shell:

现在，通过连接到MongoDB shell并打印出数据来验证数据是否仍然存在。 为此，您首先需要获取pod的名称，然后使用`exec`命令打开MongoDB shell：

*  kubectl get pods
    
    kubectl得到豆荚

The output will be similar to this:

输出将类似于以下内容：

Output NAME                     READY   STATUS    RESTARTS   AGE
    mongo-7654889675-mjcks   1/1     Running   0          13m

Now copy the name and use it in the `exec` command:

现在复制名称并在`exec`命令中使用它：

*  kubectl exec -it your\_pod\_name mongo
    
    kubectl exec -it your\_pod\_name mongo

After that, you can retrieve the data by connecting to the database and printing the whole collection:

之后，您可以通过连接到数据库并打印整个集合来检索数据：

*  use test
    
    使用测试
 *  db.getCollection("test").find()
    
    db.getCollection(“ test”)。find()

The output will look similar to this:

输出将类似于以下内容：

Output NAME                     READY   STATUS    RESTARTS   AGE
    { "_id" : ObjectId("5f1b18e34e69b9726c984c51"), "name" : "test", "number" : 10 }

As you can see the data you saved earlier is still in the database even though you restarted the pod. Now that you have successfully set up Rook and Ceph and used them to persist the data of your deployment, let’s review the Rook toolbox and what you can do with it.

如您所见，即使重新启动Pod，先前保存的数据仍在数据库中。 既然您已经成功设置了Rook和Ceph并使用它们来保存部署数据，那么让我们回顾一下Rook工具箱及其处理方法。

## 步骤5 —运行Rook工具箱 **(**Step 5 — Running the Rook Toolbox**)** ##

The [Rook Toolbox][] is a tool that helps you get the current state of your Ceph deployment and troubleshoot problems when they arise. It also allows you to change your Ceph configurations like enabling certain modules, creating users, or pools.

[Rook工具箱][Rook Toolbox]是一个工具，可以帮助您获取Ceph部署的当前状态并在出现问题时进行故障排除。 它还允许您更改Ceph配置，例如启用某些模块，创建用户或池。

In this section, you will install the Rook Toolbox and use it to execute basic commands like getting the current Ceph status.

在本节中，您将安装Rook工具箱并使用它执行基本命令，例如获取当前的Ceph状态。

The toolbox can be started by deploying the `toolbox.yaml` file, which is in the `examples/kubernetes/ceph` directory:

可以通过部署`examples/kubernetes/ceph`目录中的`toolbox.yaml`文件来启动工具箱：

*  kubectl apply -f toolbox.yaml
    
    kubectl套用-f toolbox.yaml

You will receive the following output:

您将收到以下输出：

Output deployment.apps/rook-ceph-tools created

Now check that the pod is running:

现在检查pod是否正在运行：

*  kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
    
    kubectl -n rook-ceph获取pod -l“ app = rook-ceph-tools”

Your output will be similar to this:

您的输出将类似于以下内容：

Output NAME                               READY   STATUS    RESTARTS   AGE
    rook-ceph-tools-7c5bf67444-bmpxc   1/1     Running   0          9s

Once the pod is running you can connect to it using the `kubectl exec` command:

在pod运行之后，您可以使用`kubectl exec`命令连接到它：

*  kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='\{.items\[0\].metadata.name\}') bash
    
    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l“ app = rook-ceph-tools” -o jsonpath ='\{。items \[0\] .metadata.name\}')bash

Let’s break this command down for better understanding:

为了更好的理解，让我们分解一下此命令：

1.  The `kubectl exec` command lets you execute commands in a pod; like setting an environment variable or starting a service. Here you use it to open the BASH terminal in the pod. The command that you want to execute is defined at the end of the command.
    
    使用`kubectl exec`命令可以在Pod中执行命令。 例如设置环境变量或启动服务。 在这里，您可以用它来打开Pod中的BASH终端。 要执行的命令在命令末尾定义。
2.  You use the `-n` flag to specify the Kubernetes namespace the pod is running in.
    
    您可以使用`-n`标志来指定运行Pod的Kubernetes命名空间。
3.  The `-i` (interactive) and `-t` ([`tty`][tty]) flags tell Kubernetes that you want to run the command in interactive mode with `tty` enabled. This lets you interact with the terminal you open.
    
    `-i` (interactive)和`-t` ( [`tty`][tty] )标志告诉Kubernetes您要在启用`tty`交互模式下运行命令。 这使您可以与打开的终端进行交互。
4.  `$()` lets you define an expression in your command. That means that the expression will be evaluated (executed) before the main command and the resulting value will then be passed to the main command as an argument. Here we define another Kubernetes command to get a pod where the label `app=rook-ceph-tool` and read the name of the pod using `jsonpath`. We then use the name as an argument for our first command.
    
    `$()`使您可以在命令中定义表达式。 这意味着将在主命令之前对表达式进行求值(执行)，然后将所得值作为参数传递给主命令。 在这里，我们定义了另一个Kubernetes命令来获取标签为`app=rook-ceph-tool`的容器，并使用`jsonpath`读取容器的名称。 然后，将名称用作第一个命令的参数。

**Note:** As already mentioned this command will open a terminal in the pod, so your prompt will change to reflect this.

**注意：**如前所述，此命令将在Pod中打开一个终端，因此您的提示将更改以反映这一点。

Now that you are connected to the pod you can execute Ceph commands for checking the current status or troubleshooting error messages. For example the `ceph status` command will give you the current health status of your Ceph configuration and more information like the running MONs, the current running data pools, the available and used storage, and the current I/O operations:

现在您已连接到Pod，可以执行Ceph命令来检查当前状态或对错误消息进行故障排除。 例如， `ceph status`命令将为您提供Ceph配置的当前健康状态以及更多信息，例如正在运行的MON，当前正在运行的数据池，可用和已使用的存储以及当前的I / O操作：

*  ceph status
    
    头孢状态

Here is the output of the command:

这是命令的输出：

Output   cluster:
        id:     71522dde-064d-4cf8-baec-2f19b6ae89bf
        health: HEALTH_OK
    
      services:
        mon: 3 daemons, quorum a,b,c (age 23h)
        mgr: a(active, since 23h)
        osd: 3 osds: 3 up (since 23h), 3 in (since 23h)
    
      data:
        pools:   1 pools, 32 pgs
        objects: 61 objects, 157 MiB
        usage:   3.4 GiB used, 297 GiB / 300 GiB avail
        pgs:     32 active+clean
    
      io:
        client:   5.3 KiB/s wr, 0 op/s rd, 0 op/s wr

You can also query the status of specific items like your OSDs using the following command:

您还可以使用以下命令查询OSD等特定项目的状态：

*  ceph osd status
    
    ceph osd状态

This will print information about your OSD like the used and available storage and the current state of the OSD:

这将打印有关OSD的信息，例如已使用和可用的存储以及OSD的当前状态：

Output +----+------------+-------+-------+--------+---------+--------+---------+-----------+
    | id |    host    |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
    +----+------------+-------+-------+--------+---------+--------+---------+-----------+
    | 0  | node-3jis6 | 1165M | 98.8G |    0   |     0   |    0   |     0   | exists,up |
    | 1  | node-3jisa | 1165M | 98.8G |    0   |  5734   |    0   |     0   | exists,up |
    | 2  | node-3jise | 1165M | 98.8G |    0   |     0   |    0   |     0   | exists,up |
    +----+------------+-------+-------+--------+---------+--------+---------+-----------+

More information about the available commands and how you can use them to debug your Ceph deployment can be found in the [official documentation][official documentation 1].

有关可用命令以及如何使用它们调试Ceph部署的更多信息，请参见[官方文档][official documentation 1] 。

You have now successfully set up a complete Rook Ceph cluster on Kubernetes that helps you persist the data of your deployments and share their state between the different pods without having to use some kind of external storage or provision storage manually. You also learned how to start the Rook Toolbox and use it to debug and troubleshoot your Ceph deployment.

现在，您已经在Kubernetes上成功建立了完整的Rook Ceph集群，可帮助您持久保存部署数据并在不同的Pod之间共享它们的状态，而无需手动使用某种外部存储或置备存储。 您还学习了如何启动Rook工具箱，并使用它来调试和调试Ceph部署。

## 结论 **(**Conclusion**)** ##

In this article, you configured your own Rook Ceph cluster on Kubernetes and used it to provide storage for a MongoDB application. You extracted useful terminology and became familiar with the essential concepts of Rook so you can customize your deployment.

在本文中，您在Kubernetes上配置了自己的Rook Ceph集群，并使用它为MongoDB应用程序提供了存储。 您提取了有用的术语，并熟悉Rook的基本概念，因此可以自定义部署。

If you are interested in learning more, consider checking out the official [Rook documentation][] and the example configurations provided in the repository for more configuration options and parameters.

如果您想了解更多信息，请考虑查看[Rook][Rook documentation]官方[文档][Rook documentation]以及存储库中提供的示例配置，以获取更多配置选项和参数。

You can also try out the other kinds of storage Ceph provides like [shared file systems][] if you want to mount the same volume to multiple pods at the same time.

如果您想同时将同一卷安装到多个Pod，您还可以尝试使用Ceph提供的其他类型的存储，例如[共享文件系统][shared file systems] 。

> 翻译自: [https://www.digitalocean.com/community/tutorials/how-to-set-up-a-ceph-cluster-within-kubernetes-using-rook][https_www.digitalocean.com_community_tutorials_how-to-set-up-a-ceph-cluster-within-kubernetes-using-rook]

使用rook部署ceph

[Mozilla Foundation]: https://www.brightfunds.org/organizations/mozilla-foundation
[Write for DOnations]: https://do.co/w4do-cta
[Kubernetes]: https://kubernetes.io/
[Rook]: https://rook.io/
[Ceph]: https://docs.ceph.com/docs/master/
[CRUSH algorithm_ _Controlled Replication Under Scalable Hashing_.]: https://docs.ceph.com/docs/jewel/rados/operations/crush-map/
[Kubernetes Quickstart]: https://www.digitalocean.com/docs/kubernetes/how-to/
[official documentation]: https://kubernetes.io/docs/tasks/tools/install-kubectl/
[DigitalOcean block storage]: https://www.digitalocean.com/products/block-storage/
[Volumes Quickstart]: https://www.digitalocean.com/docs/volumes/how-to/create/
[CustomResourceDefinitions]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions
[Kubernetes Object]: https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes#kubernetes-objects-and-workloads
[Ceph dashboard]: https://docs.ceph.com/docs/master/mgr/dashboard/
[Prometheus to be pre-installed]: https://docs.ceph.com/docs/master/mgr/prometheus/
[cluster example]: https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/cluster.yaml
[persistent volumes]: https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes#other-kubernetes-components
[PersistentVolumeClaim _PVC]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims
[Rook Toolbox]: https://rook.io/docs/rook/v1.3/ceph-toolbox.html
[tty]: https://en.wikipedia.org/wiki/Tty_%28unix%29
[official documentation 1]: https://docs.ceph.com/docs/giant/rados/operations/control/
[Rook documentation]: https://rook.io/docs/rook/v1.3/
[shared file systems]: https://rook.io/docs/rook/v1.3/ceph-filesystem.html
[https_www.digitalocean.com_community_tutorials_how-to-set-up-a-ceph-cluster-within-kubernetes-using-rook]: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-ceph-cluster-within-kubernetes-using-rook