ceph简介及高可用集群搭建

绝地灬酷狼 2023-03-06 03:39 26阅读 0赞

## 微信公众号搜索 DevOps和k8s全栈技术 ，即可关注公众号，也可扫描文章最后的二维码关注公众号，每天会分享技术文章供大家阅读参考哈~ ##

## Ceph简介 ##

Ceph是一个PB，EB级别的分布式存储系统，可以提供文件存储，对象存储、和块存储，它可靠性高，易扩展，管理简便，其中对象存储和块存储可以和其他云平台集成。一个Ceph集群中有Monitor节点、MDS节点（用于文件存储）、OSD守护进程。

Ceph组件

**Ceph OSD**：OSD守护进程，用于存储数据、数据恢复、数据回滚、平衡数据，并通过心跳程序向Monitor提供数据相关的监控信息。一个Ceph集群中至少需要两个OSD守护进程，这两个osd需要在两个物理服务器上，而不是在一台服务器上启动两个osd。  
**Monitor**：维护集群状态的映射信息，包括monmap，osdmap，pgmap；至少需要三个monitor来实现高可用

**MDS**：存储Ceph文件系统的元数据。

**Managers**：Ceph Manager守护程序（ceph-mgr）负责跟踪运行时指标和Ceph集群的当前状态，包括存储利用率，当前性能指标和系统负载。CephManager守护进程还托管基于python的模块来管理和公开Ceph集群信息，包括基于Web的Ceph Dashboard和RESTAPI。高可用性通常至少需要两个

安装环境准备

准备机器，osd节点机器需要两个硬盘，配置4GiB/4vCPU/60G x2

监控节点：

monitor1：192.168.85.128

monitor2：192.168.85.130

monitor3：192.168.85.131

osd节点：

osd1：192.168.85.133

osd2：192.168.85.134

初始化机器

**1.****修改主机名**

在monitor1上操作：

hostnamectl set-hostname monitor1

在monitor2上操作：

hostnamectl set-hostname monitor2

在monitor3上操作：

hostnamectl set-hostname monitor3

在osd1上操作：

hostnamectl set-hostname osd1

在osd2上操作：

hostnamectl set-hostname osd2

**2.****修改host文件，在各个节点操作**

修改各个主机节点hosts文件

/etc/hosts文件如下

192.168.85.128 monitor1
    192.168.85.130 monitor2
    192.168.85.131 monitor3
    192.168.85.133 osd1
    192.168.85.134 osd2

**3.****配置master1到三个节点无密码登陆，在master1上操作**

ssh-keygen -t rsa   \#一直回车就可以

ssh-copy-id -i .ssh/id\_rsa.pub monitor1

ssh-copy-id -i .ssh/id\_rsa.pub monitor2

ssh-copy-id -i .ssh/id\_rsa.pub monitor3

ssh-copy-id -i .ssh/id\_rsa.pub osd1

ssh-copy-id -i .ssh/id\_rsa.pub osd2

**4.****初始化机器（各个节点操作）**

yum -y install wget net-tools nfs-utils lrzsz gccgcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntplibaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-develepel-release lrzsz openssh-server

yum install deltarpm -y

**5.****关闭firewalld防火墙（****各个节点操作****）**

停止firewalld服务 停止防火墙，并禁用这个服务

systemctl stop firewalld.service && systemctldisable firewalld.service

安装iptables 如果你要改用iptables的话，需要安装iptables服务

\#安装

yum install iptables-services -y

禁用iptables

service iptables stop && systemctl disableiptables

**6.****时间同步**

在master1上

ntpdate cn.pool.ntp.org
    systemctl start ntpd &&systemctl enable ntpd

在除了master1的其他节点上

ntpdate monitor1

计划任务：

* */1 * * * /usr/sbin/ntpdate monitor1

安装ceph-deploy

**1.****安装epel源，在各个节点上操作**

yum install -y yum-utils&& sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/&& sudo yum install --nogpgcheck -y epel-release && sudo rpm--import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm/etc/yum.repos.d/dl.fedoraproject.org*

**2.****配置ceph的yum源，在各个节点上操作**

cat /etc/yum.repos.d/ceph.repo

[Ceph]
    name=Ceph packages for$basearch
    baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/
    enabled=1
    gpgcheck=0
    type=rpm-md
    gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
    priority=1
    [Ceph-noarch]
    name=Ceph noarch packages
    baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/
    enabled=1
    gpgcheck=0
    type=rpm-md
    gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
    priority=1
    [ceph-source]
    name=Ceph source packages
    baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS/
    enabled=1
    gpgcheck=0
    type=rpm-md
    gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
    priority=1

**3.****安装ceph-deploy**

yum update -y        #在各个节点上操作

yum install ceph-deploy ceph -y     #在master1上操作

yum installyum-plugin-priorities -y #在master1上操作

在除了master1的其他节点上操作

yum install ceph -y

**4.****创建ceph monitor**

**（1）创建一个目录，用于保存ceph-deploy 生成的配置文件和密钥信息的，在monitor1上操作，**

mkdir /root/ceph-deploy&& cd /root/ceph-deploy

**（2） 初始化monitor节点，在monitor1上操作，**

ceph-deploy new monitor1 monitor2 monitor3

ls可发现在ceph-deploy目录下多了几个文件，如下面这些

ceph.conf ceph-deploy-ceph.log ceph.mon.keyring
    ceph.conf            #ceph集群配置文件
    ceph-deploy-ceph.log #ceph-deploy命令执行结果的日志
    ceph.mon.keyring     #ceph mon角色的key

**（3）修改ceph配置文件，在各个monitor节点操作**

cat ceph.conf

[global]
    fsid = dc97ea52-e32c-404d-9bbc-9d0a90e39e02
    mon_initial_members = monitor1, monitor2,monitor3
    mon_host =192.168.85.128,192.168.85.129,192.168.85.130
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
    osd_pool_default_size = 2

因为我们环境中有两个OSD，Ceph模式的默认osd副本个数为3，因此需要修改配置文件Ceph.conf，在\[global\]部分增加如下配置：

osd pool default size = 2

**（4）安装ceph集群，在monitor1上操作**

ceph-deploy install monitor1 monitor2monitor3 osd1 osd2

上面使用ceph-deploy来安装ceph程序，我们之前通过yum手动在各个节点安装过ceph，为了保证ceph安装成功，这两种方法都需要使用

**（5）部署初始监视器并收集密钥，在monitor1上操作**

ceph-deploy mon create-initial

ll /root/ceph-deploy 会看到生成很多密钥

ceph.bootstrap-mds.keyring
    ceph.bootstrap-mgr.keyring
    ceph.bootstrap-osd.keyring
    ceph.bootstrap-rgw.keyring
    ceph.client.admin.keyring
    ceph.bootstrap-mds.keyring   #MDS启动key
    ceph.bootstrap-osd.keyring   #OSD启动key 
    ceph.client.admin.keyring    #管理员key

**（6）将配置文件和管理密钥分发到各个ceph节点，在monitor1上操作**

ceph-deploy admin monitor1 monitor2 monitor3 osd1 osd2

**（7）授权，在各个节点操作**

chmod +r/etc/ceph/ceph.client.admin.keyring

配置osd，下面步骤均在monitor1上操作

**1.****给ceph添osd，并完成激活**

在osd节点osd1和osd2上操作

mkdir /data/osd1 -p
    chmod 777 /data/osd1/
    chown -R ceph.ceph /data/osd1

**2.回到master1上操作，准备osd**

ceph-deploy osd prepare osd1:/data/osd1osd2:/data/osd1

**3.激活osd：**

ceph-deploy osd activate osd1:/data/osd1osd2:/data/osd1

ceph health   \#查看ceph集群状态，显示如下，在health处看到HEALTH\_OK说明集群正常

cluster a53e18df-5c06-46bb-b4db-7c44f64a367e
         health HEALTH_OK
         monmap e1: 3 mons at {monitor1=192.168.85.128:6789/0,monitor2=192.168.85.130:6789/0,monitor3=192.168.85.131:6789/0}
                election epoch 6, quorum 0,1,2 monitor1,monitor2,monitor3
         osdmap e14: 2 osds: 2 up, 2 in
                flags sortbitwise,require_jewel_osds
          pgmap v618: 320 pgs, 2 pools, 14752 kB data, 23 objects
                14146 MB used, 64246 MB / 78393 MB avail
                     320 active+clean

**4.创建ceph pool池**

ceph osd pool create testpool 256

**5.创建****rbd**

rbd create testpool/myrbd --size 10240

**6.映射块设备到自己机器**

rbd feature disable testpool/myrbd object-map fast-diff deep-flatten
    #禁用一些map，否则客户端挂载不成功
    rbd map testpool/myrbd 
    显示/dev/rbd0，说明映射成功

**7.测试客户端挂载使用**

mkdir /mnt/firstrbd 
    mkfs.xfs /dev/rbd0 
    mount /dev/rbd0 /mnt/firstrbd

**8.kubernetes对接ceph**

**可看视频了解，视频链接如下：**

https://edu.51cto.com/sd/d7dff

常见错误排查

**1.too many PGs per OSD**

**上面测试客户端挂载之后，我们在monitor1上用ceph -s再次查看集群状态**

ceph -s

可看到有警告health HEALTH\_WARN

too many PGs per OSD (320 > max300)

clustera53e18df-5c06-46bb-b4db-7c44f64a367e
         health HEALTH_WARN
                too many PGs per OSD (320 > max300)
         monmap e1: 3 mons at {monitor1=192.168.85.128:6789/0,monitor2=192.168.85.130:6789/0,monitor3=192.168.85.131:6789/0}
                election epoch 4, quorum 0,1,2monitor1,monitor2,monitor3
         osdmap e14: 2 osds: 2 up, 2 in
                flagssortbitwise,require_jewel_osds
          pgmap v253: 320 pgs, 2 pools, 14752 kBdata, 23 objects
                14158 MB used, 64235 MB / 78393 MBavail
                     320 active+clean

这个警告是因为ceph集群中osd 数量较少，创建了pool，每个pool要占用一些pg ，ceph集群默认每个osd 能有300个pgs，这个默认值是可以调整的，但调整会对集群的性能产生一定影响。如果是测试环境，可以消除这种警告，当然这种警告也可以不用理会。

查询每个osd下最大的pg阈值**：**

ceph --show-config | grep mon_pg_warn_max_per_osd

\#可以看到每个osd默认有300个pg

mon_pg_warn_max_per_osd = 300

**解决方法如下：**

**在monitor1，monitor2，monitor3上修改ceph配置文件，把osd上的pg数量修改成600**，**cat /etc/ceph/ceph.conf**

[global]
    ...
    mon_pg_warn_max_per_osd = 600

**上面修改好之后重启monitor服务，在monitor1上操作**

systemctl restart ceph-mon@monitor1
    systemctl restart ceph-mon@monitor2
    systemctl restart ceph-mon@monitor3

**ceph -s    看到下面的ceph集群状态正常了**

**2.重启monitor服务报错**

systemctl restart ceph-mon@monitor1
    systemctl restart ceph-mon@monitor2
    systemctl restart ceph-mon@monitor3

如果启动报错，按如下排查

systemctl status ceph-mon@monitor1

报错如下：

Sep 05 22:55:30 monitor1 systemd[1]: start request repeated tooquickly for ceph-mon@monitor1.service
    Sep 05 22:55:30 monitor1 systemd[1]: Failed to start Ceph clustermonitor daemon.

解决方法：

在各个节点修改配置文件

vim /usr/lib/systemd/system/ceph-mon@.service
    vim /usr/lib/systemd/system/ceph-mon@.service
    vim /usr/lib/systemd/system/ceph-mon@.service

把StartLimitInterval=30min加上注释

在各个节点执行如下，重新加载配置文件

systemctl daemon-reload

在monitor1上执行如下，重启，可启动成功

systemctl restart ceph-mon@monitor1 
    systemctl restart ceph-mon@monitor2 
    systemctl restart ceph-mon@monitor3

**3.全都重新部署**

**当你在部署时遇到问题，排查过程困难重重的话，那么不妨用如下方法，把集群清空，重新部署**，**可以执行以下清除Ceph包之后重新开始安装：**

在monitor1上操作

cd /root/ceph-deploy
    ceph-deploy purge monitor1 monitor2 monitor3 osd1 osd2
    ceph-deploy purgedata monitor1 monitor2 monitor3 osd1 osd2ceph-deployforgetkeys
    rm ceph.*

往期精彩文章

[kubernetes全栈技术+企业案例演示【带你快速掌握和使用k8s】][kubernetes_k8s]

[kubernetes面试题汇总][kubernetes]

[DevOps视频和资料免费领取][DevOps]

[kubernetes技术分享-可用于企业内部培训][kubernetes_-]

[谈谈我的IT发展之路][IT]

[kubernetes系列文章第一篇-k8s基本介绍][kubernetes_-k8s]

[kubernetes系列文章第二篇-kubectl][kubernetes_-kubectl]

[了解pod和pod的生命周期-这一篇文章就够了  
][pod_pod_-_]

[Kubernetes中部署MySQL高可用集群][Kubernetes_MySQL]

[Prometheus+Grafana+Alertmanager搭建全方位的监控告警系统-超详细文档][Prometheus_Grafana_Alertmanager_-]

[k8s1.18多master节点高可用集群安装-超详细中文官方文档][k8s1.18_master_-]

[k8s中蓝绿部署、金丝雀发布、滚动更新汇总][k8s]

[运维常见问题汇总-tomcat篇][-tomcat]

[关于linux内核参数的调优，你需要知道][linux]

[kubernetes持久化存储volume][kubernetes_volume]

[kubernetes挂载ceph rbd和cephfs][kubernetes_ceph rbd_cephfs]

[报警神器Alertmanager发送报警到多个渠道][Alertmanager]

[jenkins+kubernetes+harbor+gitlab构建企业级devops平台][jenkins_kubernetes_harbor_gitlab_devops]

[kubernetes网络插件-flannel篇][kubernetes_-flannel]

[kubernetes网络插件-calico篇][kubernetes_-calico]

[kubernetes认证、授权、准入控制][kubernetes 1]

[限制不同的用户操作k8s资源][k8s 1]

[面试真题&技术资料免费领取-覆盖面超全~][-]

[Prometheus监控MySQL][Prometheus_MySQL]

[Prometheus监控Nginx][Prometheus_Nginx]

[Prometheus监控Tomcat][Prometheus_Tomcat]

[linux面试题汇总][linux 1]

[测试通过storageclass动态生成pv][storageclass_pv]

[通过编写k8s的资源清单yaml文件部署gitlab服务][k8s_yaml_gitlab]

[helm安装和使用-通过helm部署k8s应用][helm_-_helm_k8s]

[Prometheus Operator-上篇-安装和使用篇][Prometheus Operator-_-]

[Prometheus Operator-下篇  
][Prometheus Operator-_]

[通过kubeconfig登陆k8s的dashboard ui界面][kubeconfig_k8s_dashboard ui]

[通过token令牌登陆k8s dashboard ui界面][token_k8s dashboard ui]

[kubernetes集群的etcd数据库详细介绍][kubernetes_etcd]

[Linux网络流量监控工具][Linux]

[kubernetes搭建EFK日志管理系统][kubernetes_EFK]

[prometheus operator监控k8s集群之外的haproxy组件][prometheus operator_k8s_haproxy]

[kubernetes ConfigMap存储卷][kubernetes ConfigMap]

## 技术交流群    ##

学无止境，了解更多关于kubernetes/docker/devops/openstack/openshift/linux/IaaS/PaaS相关内容，想要获取更多资料和免费视频，可按如下方式进入技术交流群

**扫码加群????**

![format_png][]

**微信：**luckylucky421302

## 微信公众号    ##

**                                    ** **长按指纹关注公众号????**

![format_png 1][]

**                                       点击在看少个 bug????**

[kubernetes_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638a31cc14032772d778f1a8fa5dc73f9aba6c99546c9ab25d9f91f706be3cff34572f0c1a&idx=1&mid=2247484708&scene=21&sn=2a7230ade0ce0bd9059e92752e03ba18#wechat_redirect
[kubernetes]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b9ccc14028a3220533e5e05f81b5b434cdc14f1cb4be6771a8afdb943f7f5f940704c61&idx=1&mid=2247484553&scene=21&sn=141700cdd2e13b87d5b756f8965a30c2#wechat_redirect
[DevOps]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b81cc140297e292675f1e8ea21d3a9f05e01e488e642bcc1113d69d06d0afbd40446c0e&idx=1&mid=2247484564&scene=21&sn=3ec4b58056aec30acf82e84cfd9daa52#wechat_redirect
[kubernetes_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b4acc14025c10c5b87f75ff8e95c89c97036f1f73250a64e2a829df191c95c34ed5b8b3&idx=1&mid=2247484511&scene=21&sn=7900c5a4eac3535be97315797d42d64b#wechat_redirect
[IT]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c29cc14053fafcb62be244f5a279052b8d159c9c530e73495cfd8ab97205988780d4aaa&idx=1&mid=2247484220&scene=21&sn=72ae437836108fbecbeb86752aad888c#wechat_redirect
[kubernetes_-k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c42cc1405547ea583833c46878b6c120f5f39fb3746f89a8d2e515d432422b3019ded64&idx=1&mid=2247484247&scene=21&sn=c8986ed491e6c48cb6faab087fe8472c#wechat_redirect
[kubernetes_-kubectl]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638cbacc1405ac390b6c0fc7535e1cc805877a408c5f4a94a7148a47c186a57ff4f0c8ea32&idx=3&mid=2247484335&scene=21&sn=7e5ae586dae2016f6c0f5fcb60dd7317#wechat_redirect
[pod_pod_-_]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c30cc1405263cbd23e3fbe23f9ca9e83c1eb1800216a0e4e26e7c27d4234598def46bb1&idx=1&mid=2247484197&scene=21&sn=36ece95b266609bd769cf6a1fcf9d723#wechat_redirect
[Kubernetes_MySQL]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c73cc1405657ed02b3126423259cb8563c18d268fa7ffd4a516327cd35148ef61d8d762&idx=1&mid=2247484262&scene=21&sn=de970288426cdca05cf6ca0f5df4ff4c#wechat_redirect
[Prometheus_Grafana_Alertmanager_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b08cc14021e5e025f2f0b89d18ec5ef19f9dec4429a56f356e8b596d01095e313dab3e3&idx=1&mid=2247484445&scene=21&sn=bf8928e7aae815b8e7f9ac56a67289ce#wechat_redirect
[k8s1.18_master_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c7dcc14056b9f0261e05e4d36a057c4511fb5806222f71b7965c3573fed86672b2c93b7&idx=1&mid=2247484264&scene=21&sn=4d74de0cb117c07ba7c73ea21c42bf94#wechat_redirect
[k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c36cc140520e6905db5923afe163d7babb5d9eb6c5e8045a795c37b33a2a2e5541e3efd&idx=1&mid=2247484195&scene=21&sn=b841f2ea305acfa2996a667d4ff4d99e#wechat_redirect
[-tomcat]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c21cc140537d10eeb8b4eb37e1451392123dd74513d704af00506c0576ba11da9907735&idx=1&mid=2247484212&scene=21&sn=923b044e9d2b4e5a612ac7c1307d166c#wechat_redirect
[linux]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c5ecc14054862495a8f77d58d7483c34310839ea3ed93d8b44875a1685b3412d381f23a&idx=1&mid=2247484235&scene=21&sn=12e2e80187dbf217efff6c59da778f7c#wechat_redirect
[kubernetes_volume]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638ba8cc1402be9f8feca65582441f2d1b9ef6ceded048368ea6ad2a6eca3f8ae2f44a025f&idx=1&mid=2247484605&scene=21&sn=dfc6dd544423a766b4cd1b06fa59c8e1#wechat_redirect
[kubernetes_ceph rbd_cephfs]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638a07cc140311e6e111b9bb89e3613c495a26b6b496fe97c838787568c72dd1bb23088520&idx=1&mid=2247484690&scene=21&sn=32deb33cef9d91518cd8dea47da073c5#wechat_redirect
[Alertmanager]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638ab7cc1403a1f6b07cbe682548595fe683f1ffdf56d2b4d31382003907a41da1e03c1408&idx=1&mid=2247484834&scene=21&sn=7f407f5a27b03b4757c8892f1b5d6234#wechat_redirect
[jenkins_kubernetes_harbor_gitlab_devops]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638d2fcc14043990967471ba8a2aaf389ab483594fd81395b5afda1c23a9eff4371df6bca4&idx=1&mid=2247483962&scene=21&sn=595eac335f7fb523bbb60cbbede109ca#wechat_redirect
[kubernetes_-flannel]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c005de5f4913b3e81d32cf98dc4f237be04b0c19e02f615650538a9aa0aabdc025604918&idx=1&mid=2247483855&scene=21&sn=26587a359cca53d129ab7e8a7fa394e0#wechat_redirect
[kubernetes_-calico]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c073de5f49658fdaed40affb1a9ea3f059e7df3cc90ae24b194a8698a7f73089d3964cab&idx=1&mid=2247483833&scene=21&sn=c00c2b69db7d06eacaf6206805d2955a#wechat_redirect
[kubernetes 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c05fde5f494917570591168d9e71f507c2d124312c51dfdb6cfb30eb03eb48b73a3e0d9b&idx=1&mid=2247483797&scene=21&sn=bc208ee075c41b81ffabaefe509538ba#wechat_redirect
[k8s 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0b4de5f49a2eef8cc91afa81095b71eb093e94dce6eddbc7059583e819b1ac51c596ebb&idx=1&mid=2247483774&scene=21&sn=dfc3babde045fe99da9fa168cc9f0595#wechat_redirect
[-]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0bfde5f49a9990c672e4434f5ae7b74004dc1982cc44801c524b58067d9b245e5ba5b71&idx=1&mid=2247483765&scene=21&sn=b8444e82a93535425349b509ded4e4b6#wechat_redirect
[Prometheus_MySQL]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0aade5f49bc7df29ecb2280ab9e1a8aa905b31e36856bf8dc19b05ef60c5e7be0333b88&idx=1&mid=2247483744&scene=21&sn=1035d73f8cf39caf8534766435497590#wechat_redirect
[Prometheus_Nginx]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c095de5f4983c249b1b556fa6f8f60a0bb6991d2872079a9d5c28ce2ee17b8fc407eeb2c&idx=1&mid=2247483743&scene=21&sn=3ad15cd693685479dc7bf5dd628796d2#wechat_redirect
[Prometheus_Tomcat]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c086de5f499011eb542c6da8ead32b538c0bf06340ebe56ad439e2649e0b63efbcb4a611&idx=1&mid=2247483724&scene=21&sn=dd390f05270c6ac7a9c2b51fdc22f0ee#wechat_redirect
[linux 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0dede5f49c878898e04329f08d8adf554397476283aaa3f2554f71a8bf20986755fbcc9&idx=1&mid=2247483668&scene=21&sn=a7d0768b7eb034ef2e2f23e25b1b1845#wechat_redirect
[storageclass_pv]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638837cc140121578456f6a5d679ae481421f2f3f1d5099bb7f2c56d00174fe61b2016061d&idx=1&mid=2247485218&scene=21&sn=ba698d444a029a2659091bfd81133697#wechat_redirect
[k8s_yaml_gitlab]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638814cc1401028c7ab321d2ac6579f5c6d2b029fadc5687874e41470bf95aa09c01c33d45&idx=1&mid=2247485185&scene=21&sn=2b5646119a008d69948c5a84d3724c95#wechat_redirect
[helm_-_helm_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638bbbcc1402ad16d4b65b3db26cbd324c85819934077caa41969f0c9d573b71b21e2c51ff&idx=1&mid=2247484590&scene=21&sn=2c23098269c5df1e793aa47198a7b1e6#wechat_redirect
[Prometheus Operator-_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638626cc140f30c8d35e45239e6b559f7369b8c58bacd157a24e7b62313c826baec75f299f&idx=2&mid=2247485747&scene=21&sn=3da14a91d75f745b26c3aa8d10361d23#wechat_redirect
[Prometheus Operator-_]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63865bcc140f4d8966d1c3745c8f69c6384337c43a1a8eda6e1ae488bd78f08e44c9553b19&idx=1&mid=2247485774&scene=21&sn=1b0301dadc4b737e19b1066219813db7#wechat_redirect
[kubeconfig_k8s_dashboard ui]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638679cc140f6f4e024f2e7739b995fc235b6f4e97983a583bc9c7e82da88cf0fbf8736fe7&idx=1&mid=2247485804&scene=21&sn=c157f0596f130bd97ec745b278546c50#wechat_redirect
[token_k8s dashboard ui]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638687cc140f918f04ca60ed188da26cd3ab37a79e6fdbf64531efd11ae60dac0a77788f7c&idx=2&mid=2247485842&scene=21&sn=163174142218e8dca3bb9a94ed173694#wechat_redirect
[kubernetes_etcd]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63853bcc140c2d4c4da1b59a4865c193383392d92075cee80530866892d8a2a365e2b4e565&idx=1&mid=2247485998&scene=21&sn=9889882d1820ac005e5512eccb67c0f1#wechat_redirect
[Linux]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638538cc140c2e581b8c673a9c529359042a2262ec112f8c3476c7408f25e04856f0b37bbd&idx=2&mid=2247485997&scene=21&sn=ac694beaf60cbf18d133eb6ebbc7a345#wechat_redirect
[kubernetes_EFK]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63851fcc140c09d17fe6e270524da6c3ec3c5f3969f6f4f61a54c54724168801d93b2cda9d&idx=1&mid=2247485962&scene=21&sn=87dfde054f2ff489a1ffd28dfe02a435#wechat_redirect
[prometheus operator_k8s_haproxy]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6386f0cc140fe6dea923cb07c2164db1c894c87d8c85d828aa269cdf4ae2d92eeb201adb67&idx=1&mid=2247485925&scene=21&sn=5ae9bc969963775e347c3a1f279d541c#wechat_redirect
[kubernetes ConfigMap]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63853acc140c2ca1dc3ca813b133ca7158c34df5cfe59cebf7621851ead91490eaabc74a1b&idx=1&mid=2247485999&scene=21&sn=f6b03745426149e498a8c7065b060e24#wechat_redirect
[format_png]: /images/20230208/e3d93b5c383f4735a18aa020c4aa7303.png
[format_png 1]: /images/20230208/6a677a5482034fa5a94ef68b8921b0fa.png