Kubernetes 中完美部署RabbitMQ Cluster

╰+攻爆jí腚メ 2023-03-03 11:21 34阅读 0赞

### 1. rabbitmq-autocluster(已被淘汰) ###

Kubernetes中可以使用rabbitmq-autocluster这个docker镜像来创建rabbitmq cluster。rabbitmq-autocluster使用autocluster插件去调用kubernetes apiserver获取rabbitmq服务的endpoints获取node节点信息，并自动加入集群。

去docker-hub中看到这货不是官方出的，并且已经3年没更新了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70][]

去github中看到已经被RabbitMQ官方Team接管

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 1][]

结果到rabbitmq官方github中看到也是3年没更新了，并且只支持3.6.x版本。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 2][]

但是在README中看到，对于3.7.x或更新的版本开始使用 peer discovery subsystem来构建rabbitmq-cluster，并且支持kubernetes。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 3][]

### 2. Rabbitmq-peer-discovery-k8s ###

Github地址[    https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s][https_github.com_rabbitmq_rabbitmq-peer-discovery-k8s]

参考官方给出的minikube示例即可

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 4][]

### 2.1 ConfigMap ###

>     ---
>     # RabbitMQ Config
>     apiVersion: v1
>     kind: ConfigMap
>     metadata:
>       name: rabbitmq-config
>     data:
>       enabled_plugins: |
>           [rabbitmq_management,rabbitmq_peer_discovery_k8s].
>       rabbitmq.conf: |
>           cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
>           cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
>           cluster_formation.k8s.address_type = hostname
>           cluster_formation.node_cleanup.interval = 30
>           cluster_formation.node_cleanup.only_log_warning = true
>           cluster_partition_handling = autoheal
>           queue_master_locator=min-masters
>           loopback_users.guest = false
>     ---

### 2.2 RBAC ###

>     ---
>     # RabbitMQ ServiceAccount
>     apiVersion: v1
>     kind: ServiceAccount
>     metadata:
>       name: rabbitmq
>     ---
>     # RabbitMQ Role
>     apiVersion: rbac.authorization.k8s.io/v1
>     kind: Role
>     metadata:
>       name: endpoint-reader
>     rules:
>       - apiGroups: [""]
>         resources: ["endpoints"]
>         verbs: ["get"]
>     ---
>     # RabbitMQ RoleBinding
>     apiVersion: rbac.authorization.k8s.io/v1
>     kind: RoleBinding
>     metadata:
>       name: endpoint-reader
>     subjects:
>       - kind: ServiceAccount
>         name: rabbitmq
>     roleRef:
>       apiGroup: rbac.authorization.k8s.io
>       kind: Role
>       name: endpoint-reader

### 2.3 Service ###

>     ---
>     # RabbitMQ Service
>     apiVersion: v1
>     kind: Service
>     metadata:
>       name: rabbitmq-cluster
>       labels:
>         app: rabbitmq-cluster
>         type: LoadBalancer
>     spec:
>       selector:
>         app: rabbitmq-cluster
>       ports:
>         - name: amqp-port
>           port: 5672
>           targetPort: 5672
>           protocol: TCP
>         - name: mgmt-port
>           port: 15672
>           targetPort: 15672
>           protocol: TCP
>     ---
>     # RabbitMQ NodePort Service
>     apiVersion: v1
>     kind: Service
>     metadata:
>       name: rabbitmq-nodeport
>       labels:
>         app: rabbitmq-nodeport
>         type: LoadBalancer
>     spec:
>       type: NodePort
>       selector:
>         app: rabbitmq-cluster
>       ports:
>         - name: amqp-port
>           nodePort: 30001
>           port: 5672
>           targetPort: 5672
>           protocol: TCP
>         - name: mgmt-port
>           nodePort: 30002
>           port: 15672
>           targetPort: 15672
>           protocol: TCP

### 2.4 StatefulSet ###

>     # RabbitMQ-Cluster StatefulSet
>     apiVersion: apps/v1
>     kind: StatefulSet
>     metadata:
>       name: rabbitmq-cluster
>     spec:
>       replicas: 3
>       selector:
>         matchLabels:
>           app: rabbitmq-cluster
>       serviceName: rabbitmq-internal
>       template:
>         metadata:
>           labels:
>             app: rabbitmq-cluster
>         spec:
>           serviceAccountName: rabbitmq
>           containers:
>             - name: rabbitmq
>               image: rabbitmq:3
>               livenessProbe:
>                 exec:
>                   # Stage 2 check, more detail at https://www.rabbitmq.com/monitoring.html#health-checks
>                   command: ["rabbitmq-diagnostics", "status"]
>                 initialDelaySeconds: 60
>                 periodSeconds: 60
>                 timeoutSeconds: 10
>               readinessProbe:
>                 exec:
>                   # Stage 2 check, more detail at https://www.rabbitmq.com/monitoring.html#health-checks
>                   command: ["rabbitmq-diagnostics", "status"]
>                 initialDelaySeconds: 60
>                 periodSeconds: 60
>                 timeoutSeconds: 10
>               ports:
>                 - containerPort: 5672
>                   protocol: TCP
>                 - containerPort: 15672
>                   protocol: TCP
>               env:
>                 - name: MY_POD_NAME
>                   valueFrom:
>                     fieldRef:
>                       fieldPath: metadata.name  # get pod.metadata.name, e.g. rabbitmq-cluster-0
>                 - name: MY_POD_NAMESPACE
>                   valueFrom:
>                     fieldRef:
>                       fieldPath: metadata.namespace  # get pod.metadata.namespace
>                 - name: RABBITMQ_DEFAULT_USER
>                   value: "admin"
>                 - name: RABBITMQ_DEFAULT_PASS
>                   value: "admin"
>                 - name: RABBITMQ_USE_LONGNAME
>                   value: "true"
>                 - name: K8S_SERVICE_NAME
>                   value: "rabbitmq-internal"
>                 - name: RABBITMQ_NODENAME
>                   value: "rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local"
>                 - name: K8S_HOSTNAME_SUFFIX
>                   value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
>                 - name: RABBITMQ_ERLANG_COOKIE
>                   value: "SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY="    # generator by: echo $(openssl rand -base64 32)
>               volumeMounts:
>                 - name: config-volume
>                   mountPath: /etc/rabbitmq
>           volumes:
>             - name: config-volume
>               configMap:
>                 name: rabbitmq-config
>                 items:
>                   - key: rabbitmq.conf
>                     path: rabbitmq.conf
>                   - key: enabled_plugins
>                     path: enabled_plugins

### 3. 运行效果 ###

部署了3个pod

![20200731113424353.png][]

查看Pod日志可以看到，RabbitMQ自动发现节点并加入集群

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 5][]

查看RabbitMQ Cluster状态

> root@rabbitmq-cluster-0:/\# rabbitmqctl cluster\_status  
> Cluster status of node rabbit@rabbitmq-cluster-0.rabbitmq-internal.demo.svc.cluster.local ...  
> Basics
> 
> Cluster name: rabbit@rabbitmq-cluster-0.rabbitmq-internal.demo.svc.cluster.local
> 
> Disk Nodes
> 
> rabbit@rabbitmq-cluster-0.rabbitmq-internal.demo.svc.cluster.local  
> rabbit@rabbitmq-cluster-1.rabbitmq-internal.demo.svc.cluster.local  
> rabbit@rabbitmq-cluster-2.rabbitmq-internal.demo.svc.cluster.local
> 
> Running Nodes
> 
> rabbit@rabbitmq-cluster-0.rabbitmq-internal.demo.svc.cluster.local  
> rabbit@rabbitmq-cluster-1.rabbitmq-internal.demo.svc.cluster.local  
> rabbit@rabbitmq-cluster-2.rabbitmq-internal.demo.svc.cluster.local
> 
> Versions
> 
> rabbit@rabbitmq-cluster-0.rabbitmq-internal.demo.svc.cluster.local: RabbitMQ 3.8.5 on Erlang 23.0.3  
> rabbit@rabbitmq-cluster-1.rabbitmq-internal.demo.svc.cluster.local: RabbitMQ 3.8.5 on Erlang 23.0.3  
> rabbit@rabbitmq-cluster-2.rabbitmq-internal.demo.svc.cluster.local: RabbitMQ 3.8.5 on Erlang 23.0.3
> 
> Alarms
> 
> (none)
> 
> Network Partitions
> 
> (none)
> 
> Listeners

登录web管理界面查看

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 6][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70]: /images/20230208/182a0d58029d4f53996596cfbd8a7450.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 1]: /images/20230208/1b472d728f3f4f1fa2ddddb9a47bf21f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 2]: /images/20230208/060e4bf123954f1ea3e9abb42c68c170.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 3]: /images/20230208/cc2eb4115d704dec9df190f72bcf673b.png
[https_github.com_rabbitmq_rabbitmq-peer-discovery-k8s]: https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 4]: /images/20230208/5858f7cfa68b4197981111565de5bc6d.png
[20200731113424353.png]: /images/20230208/d7dbefae31c84d068c9a0c8fd8311712.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 5]: /images/20230208/eda6610d06124a99956da96be280e5d9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NvY28zODQ4_size_16_color_FFFFFF_t_70 6]: /images/20230208/ac2eca1f1ff94704ac012339485dfe26.png