CentOS7上通过HAproxy+Keepalived部署RabbitMQ集群

Love The Way You Lie 2022-03-10 05:06 400阅读 0赞

参考：

[https://blog.csdn.net/sj349781478/article/details/78845852][https_blog.csdn.net_sj349781478_article_details_78845852]

[https://blog.csdn.net/sj349781478/article/details/78862315][https_blog.csdn.net_sj349781478_article_details_78862315]

[https://www.jianshu.com/p/6376936845ff][https_www.jianshu.com_p_6376936845ff]

两台机器，IP地址规划：

Master：192.168.174.140

Backup: 192.168.174.141

虚拟IP【VIP】：192.168.174.150

1、在两台机器上分别执行如下命令，安装RabbiMQ服务：

官方网站强烈建议使用Package Cloud或Bintray这两种方式安装

这里采用第一种方式即Package Cloud方式，安装官网：[http://www.rabbitmq.com/install-rpm.html\#package-cloud][http_www.rabbitmq.com_install-rpm.html_package-cloud]

分别执行如下命令导入相关的认证key：

# import the new PackageCloud key that will be used starting December 1st, 2018 (GMT)
    rpm --import https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
    
    # import the old PackageCloud key that will be discontinued on December 1st, 2018 (GMT)
    rpm --import https://packagecloud.io/gpg.key

然后安装rabbitMQ的yum源，地址：[https://packagecloud.io/rabbitmq/rabbitmq-server/install\#bash-rpm][https_packagecloud.io_rabbitmq_rabbitmq-server_install_bash-rpm]

curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash

完成后末尾输出如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70][]

然而此时并不能安装成功，因为RabbitMQ依赖于Erlang

![20190309180002543.png][]

所以还需要安装Erlang

执行如下命令，导入Erlang的yum源，地址：[https://packagecloud.io/rabbitmq/erlang/install\#bash-rpm][https_packagecloud.io_rabbitmq_erlang_install_bash-rpm]

curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash

完成后末尾输出：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 1][]

上面全部都准备就绪可以安装Erlang，再安装RabbitMQ了

yum -y install erlang
    yum -y install rabbitmq-server

![20190309180323201.png][]

![20190309180305495.png][]

添加RabbitMQ开机启动：

chkconfig rabbitmq-server on
    或者
    systemctl enable rabbitmq-server

查看是否开启开机启动：

systemctl list-unit-files | grep enable

![20190309185243666.png][]

启动RabbitMQ Server或者关闭

systemctl start rabbitmq-server.service
    或者
    rabbitmqctl start_app
    
    对应的关闭
    systemctl stop rabbitmq-server.service
    或者
    rabbitmqctl stop_app

查看RabbitMQ的状态

rabbitmqctl  status

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 2][]

RabbitMQ提供了一个Web界面的控制台，通过 IP:15672 访问，这里遇到一个坑，死活访问不了，两种情况

1：没有启动Web界面的插件

![20190309183202380.png][]

启用插件命令：

rabbitmq-plugins enable rabbitmq_management

如果发现还是访问不了，那就是防火墙问题了，LZ在这里查了老久【建议在学习的时候，使用Linux，安装好第一件事情就是把防火墙关了，同时设置为开机不重启】

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 3][]

关闭防火墙并设置开机禁用：

systemctl stop firewalld
    systemctl disable firewalld

然后再去访问：192.168.174.140:15672

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 4][]

然后添加用户，配置权限和角色【 默认账户 guest/guest】可以在命令行通过rabbitmqctl  list\_users查看

#添加用户，后面两个参数分别是用户名和密码
    rabbitmqctl add_user cuit cuit  
    #添加权限
    rabbitmqctl set_permissions cuit".*"".*"".*" 
    #修改用户角色
    rabbitmqctl set_user_tags cuit administrator

然后登录

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 5][]

然后以同样的步骤安装第二台机器

分别修改一下两台机器的主机映射,使用vi编辑/etc/hosts文件，保存退出

vi /etc/hosts

192.168.174.140 rmq-master-node

192.168.174.141 rmq-backup-node

![20190309195116381.png][]

RabbitMQ的集群是依附于Erlang的集群来工作的,所以必须先构建起Erlang的集群镜像，各个节点的这个文件必须要保持一致。

Erlang Cookie 文件：/var/lib/rabbitmq/.erlang.cookie。这里将 rmq-master-node 140机器的该文件远程复制到 rmq-backup-node 141机器上

使用scp命令，可以配置SSH远程免密码登录，就不用每次输入密码了

SSH远程免密码登录参考：[https://blog.csdn.net/weixin\_42465125/article/details/88370270][https_blog.csdn.net_weixin_42465125_article_details_88370270]

scp -r /var/lib/rabbitmq/.erlang.cookie root@192.168.174.141:/var/lib/rabbitmq/.erlang.cookie

可以使用cat命令，确认一下，两台机器的.erlang.cookie的值是不是一致的

组成集群。在rmq-backup-node 141机器上先停止RabbitMQ服务，然后加入集群，然后在启动，执行命令如下：

rabbitmqctl stop_app 
    rabbitmqctl join_cluster rabbit@rmq-master-node
    rabbitmqctl start_app

如果使用这个命令rabbitmqctl stop\_app 报下面这个错误，可以尝试使用：systemctl stop rabbitmq-server.service 停止

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 6][]

将备用节点加入集群，报错请往下看

![20190309202314806.png][]

注意到没有，这里的机器的主机名已经发生变化，在之前的操作都是可以的，但是在加入集群这里如果主机名没有修改成/etc/hosts里面IP对应的主机名，执行这一步是会报错的，所以修改一下主机名，然后重启一下

首先修改140主节点机器

vi /etc/hostname

![20190309202755556.png][]

141备用节点：

![20190309202827864.png][]

然后reboot两台机器

再在备用节点上面执行加入集群的操作，由于我们前面设置了开机启动的，所以在备用机上先stop掉RabbitMQ服务在执行加入集群的操作，下面是启动的情况，添加集群报的错，还有提示的

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 7][]

为了提高性能，不需要两个节点都是disc的节点，所以我们可以启动一个节点为RAM模式，在主节点执行下面的命令

rabbitmqctl change_cluster_node_type  ram

![20190309211207276.png][]

先stop掉在执行

![20190309211301721.png][]

查看集群状态

rabbitmqctl cluster_status

![20190309211337837.png][]

现在RabbitMQ的集群已经配置好，下面来配置高可用

###  两节点安装haproxy ###

yum -y install haproxy

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 8][]

配置两节点的haproxy配置文件，在末尾追加

扩展：Windows远程编辑Linux文件参考[https://blog.csdn.net/weixin\_42465125/article/details/88398976][https_blog.csdn.net_weixin_42465125_article_details_88398976]

直接使用vi编辑  
vi /etc/haproxy/haproxy.cfg 之后添加：【注意修改IP和认证的用户名和密码】更多配置另行学习配置

listen rabbitmq_cluster
        bind 0.0.0.0:56722
        mode tcp
        option tcplog
        balance roundrobin
        server rmq-master-node 192.168.174.140:56722 check inter 5000 rise 2 fall 2
        server rmq-backup-node 192.168.174.141:56722 check inter 5000 rise 2 fall 2
    
    listen admin_stats
        stats  enable
        bind  *:9090
        mode  http
        option  httplog
        log  global
        maxconn  10
        stats  refresh 30s
        stats  uri /haproxy  #访问的URL context-path
        stats  realm haproxy
        stats  auth admin:123456 # 访问的用户名和密码
        stats  hide-version

分别启动两台haproxy，有坑，往下看

### 巨坑错误：各种配置改了都起不来 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 9][]

解决方法一：

vim /etc/sysctl.conf

net.ipv4.ip\_nonlocal\_bind=1

然后sysctl -p成功启动

解决方法二：

setsebool -P haproxy\_connect\_any=1

参考地址：[https://www.cnblogs.com/aguncn/p/10493170.html][https_www.cnblogs.com_aguncn_p_10493170.html]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 10][]

成功启动

访问一下：[http://192.168.174.140:9090/haproxy][http_192.168.174.140_9090_haproxy]和[http://192.168.174.141:9090/haproxy][http_192.168.174.140_9090_haproxy] 输入用户名和密码

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 11][]

### 两节点安装keepalived服务 ###

yum -y install keepalived

配置主节点rmq-maste-node的Keepalived的配置

vi /etc/keepalived/keepalived.conf

global_defs {
      router_id rmq-master-node
    }
    
    vrrp_script chk_haproxy
    {
      script "/etc/keepalived/script/haproxy_check.sh" # 脚本位置
      interval 2
      timeout 2
      fall 3
    }
    
    vrrp_instance haproxy {
      state  MASTER # 主节点
      interface  ens33
      virtual_router_id  29
      priority  150  # 权重
      authentication
      {
        auth_type PASS
        auth_pass 29
      }
      virtual_ipaddress
      {
        192.168.174.150 # 虚拟IP，即VIP
      }
      track_script
      {
        chk_haproxy
      }
      notify_master "/etc/keepalived/script/haproxy_master.sh"
    }

修改备用节点rmq-backup-node的Keepalived的配置。和主节点很像，只是state和priority不同

global_defs {
      router_id rmq-backup-node
    }
    
    vrrp_script chk_haproxy
    {
      script "/etc/keepalived/script/haproxy_check.sh"
      interval 2
      timeout 2
      fall 3
    }
    
    vrrp_instance haproxy {
      state  BACKUP # 备节点
      interface  ens33
      virtual_router_id  29
      priority  100  # 权重
      authentication
      {
        auth_type PASS
        auth_pass 29
      }
      virtual_ipaddress
      {
        192.168.174.150 # 虚拟IP，即VIP
      }
      track_script
      {
        chk_haproxy
      }
      notify_master "/etc/keepalived/script/haproxy_master.sh"
    }

haproxy\_master.sh和haproxy\_check.sh两个脚本的内容

建立这个没有的目录，mkdir -p /etc/keepalived/scripts && cd /etc/keepalived/scripts

vi haproxy\_master.sh

#!/bin/bash
     LOGFILE="/var/log/keepalived-haproxy-state.log"
     echo "Being Master ..." >> $LOGFILE

vi haproxy\_check.sh

#!/bin/bash
    LOGFILE="/var/log/keepalived-haproxy-state.log"
    date >> $LOGFILE
    if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
    	echo "fail: check_haproxy status" >> $LOGFILE
    	exit 1
    else
    	echo "success: check_haproxy status" >> $LOGFILE
    	exit 0
    fi

添加执行权限

chmod +x \*.sh

分别两台启动keepalived，并设置开机启动

systemctl start keepalived.service
    systemctl enable keepalived.service

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 12][]

最后就是测试高可用好不好使了：

首先测试VIP访问是否可以：[http://192.168.174.150:15672][http_192.168.174.150_15672]

同样其他IP访问也是可以的：[http://192.168.174.140:15672][http_192.168.174.140_15672]  [http://192.168.174.141:15672][http_192.168.174.141_15672]

这里中途还遇到一个诡异的问题，就是之前已经添加了一个cuit/cuit的用户名和密码，结果后面不见了，不能登录了，然后又建了一遍用户

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 13][]

1）关闭主keepalived，VIP是否切换至备机，VIP访问RabbitMQ web界面（验证keepalived高可用）

2）关闭主HAproxy，VIP是否切换至备机，业务是否正常，恢复原状；（验证HAproxy高可用）

3）关闭后台服务器rmq-master-node，VIP访问RabbitMQ web界面。（验证HAproxy状态检查）

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 不积跬步无以至千里，不积小流无以成江海 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

[https_blog.csdn.net_sj349781478_article_details_78845852]: https://blog.csdn.net/sj349781478/article/details/78845852
[https_blog.csdn.net_sj349781478_article_details_78862315]: https://blog.csdn.net/sj349781478/article/details/78862315
[https_www.jianshu.com_p_6376936845ff]: https://www.jianshu.com/p/6376936845ff
[http_www.rabbitmq.com_install-rpm.html_package-cloud]: http://www.rabbitmq.com/install-rpm.html#package-cloud
[https_packagecloud.io_rabbitmq_rabbitmq-server_install_bash-rpm]: https://packagecloud.io/rabbitmq/rabbitmq-server/install#bash-rpm
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70]: /images/20220310/7551a45bd5b7435e9226ae093f1942b6.png
[20190309180002543.png]: /images/20220310/700578b69a724799aff7fc2fb2989712.png
[https_packagecloud.io_rabbitmq_erlang_install_bash-rpm]: https://packagecloud.io/rabbitmq/erlang/install#bash-rpm
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 1]: /images/20220310/a51426786a524fdeab967a08638e44f2.png
[20190309180323201.png]: /images/20220310/0128180bdd15489485c6ada0e1642796.png
[20190309180305495.png]: /images/20220310/af4457e40c454d8c8bf621e7fabd19f7.png
[20190309185243666.png]: /images/20220310/78dc3746d75449f79f44f3e7cd2b7cca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 2]: /images/20220310/42c89b3ce74f474da2d27b304967c7ed.png
[20190309183202380.png]: /images/20220310/cd455dd7d71f43aaa1445a458e312620.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 3]: /images/20220310/b42527912bff4fc18de268766c1d04ec.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 4]: /images/20220310/a0eb28b6e60b4da288c86fc35bdfc65d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 5]: /images/20220310/4264969a8ee74e6898bb070dabc9a37d.png
[20190309195116381.png]: /images/20220310/483351cf85794babac60759978cc5ba0.png
[https_blog.csdn.net_weixin_42465125_article_details_88370270]: https://blog.csdn.net/weixin_42465125/article/details/88370270
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 6]: /images/20220310/4e28c89caaa941f1a767f75754f0778c.png
[20190309202314806.png]: /images/20220310/bd3130d015c040c6857f045f379542ea.png
[20190309202755556.png]: /images/20220310/022172549ad345bca72afb74b46a1303.png
[20190309202827864.png]: /images/20220310/55c67b8a8a514d6b8ab6c5859f7d6e59.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 7]: /images/20220310/16b156ad47f547aba03e295385ae620f.png
[20190309211207276.png]: /images/20220310/ff6b6bf5353e4144baf792e63934941a.png
[20190309211301721.png]: /images/20220310/00a57b6449c74ea2b363a96bff36a172.png
[20190309211337837.png]: /images/20220310/6f562aee01474d56b1aa969060d00ced.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 8]: /images/20220310/2ec7343b21974861903dee61dae6dbea.png
[https_blog.csdn.net_weixin_42465125_article_details_88398976]: https://blog.csdn.net/weixin_42465125/article/details/88398976
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 9]: /images/20220310/5ff54c20fa1842af817494255ca31e4f.png
[https_www.cnblogs.com_aguncn_p_10493170.html]: https://www.cnblogs.com/aguncn/p/10493170.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 10]: /images/20220310/8d88be3222e74c1b8e41cc15ae1e8651.png
[http_192.168.174.140_9090_haproxy]: http://192.168.174.140:9090/haproxy
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 11]: /images/20220310/376b0b8599e84ae5afbe1821212912c5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 12]: /images/20220310/6f3c45ff0aa048218e90370b3f4e2336.png
[http_192.168.174.150_15672]: http://192.168.174.150:15672/#/
[http_192.168.174.140_15672]: http://192.168.174.140:15672/#/
[http_192.168.174.141_15672]: http://192.168.174.141:15672/#/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ2NTEyNQ_size_16_color_FFFFFF_t_70 13]: /images/20220310/257d793f583c4930bd80e8274d78c849.png