Docker 安装 Keepalived + Nginx 并实现高可用

﹏ヽ暗。殇╰゛Y 2022-10-11 00:40 368阅读 0赞

文章目录

Docker 安装 Keepalived + Nginx 并实现高可用  
准备工作  
下载 centos 镜像并配置环境  
下载安装 Nginx  
配置 yum 源  
下载安装 Keepalived  
编译安装  
配置 Keepalived  
检查配置  
测试  
容器创建快照  
创建主从容器  
进入 maser 容器修改配置  
进入 slave 容器修改配置  
测试 Keepalived

需求目标：基于 Centos 镜像安装 Keepalived + Nginx 并实现高可用  
准备工作

下载 centos 镜像并配置环境

$ docker pull centos \# 拉取最新 centos  
$ docker images \# 查看镜像  
$ docker run -itd --name centos centos \# 后台启动一个容器  
$ docker exec -it centos bash \# 进入容器  
$ yum -y upgrade \# 升级所有包   
$ yum install -y yum-utils vim make wget gcc openssl openssl-devel popt-devel \# 安装相关依赖  
1  
2  
3  
4  
5  
6  
下载安装 Nginx

配置 yum 源

官方参考文档 http://nginx.org/en/linux\_packages.html

$ cat /etc/yum.repos.d/nginx.repo

\[nginx-stable\]  
name=nginx stable repo  
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/  
gpgcheck=1  
enabled=1  
gpgkey=https://nginx.org/keys/nginx\_signing.key  
module\_hotfixes=true

\[nginx-mainline\]  
name=nginx mainline repo  
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/  
gpgcheck=1  
enabled=0  
gpgkey=https://nginx.org/keys/nginx\_signing.key  
module\_hotfixes=true  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
$ yum install -y nginx \# 安装 Nginx  
$ whereis nginx \# 查看安装路径,默认配置文件在 /etc/nginx  
$ nginx -t \# 插件配置  
$ nginx \# 启动  
$ curl localhost \# 检查启动是否成功  
1  
2  
3  
4  
5  
下载安装 Keepalived

Keepalived 官网 下载最新版本

$ mkdir -p /mnt/software && cd /mnt/software   
$ wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz  
$ docker cp keepalived-2.0.20.tar.gz centos:/mnt/ \# 下载慢！！也可以从本地 copy  
$ tar -zxvf keepalived-2.0.20.tar.gz   
1  
2  
3  
4  
编译安装

$ cd keepalived-2.0.20  
$ ./configure --prefix=/usr/local/keepalived  
1  
2  
编译完后可能会出现以下警告

\*\*\* WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.  
1  
按照提示按照

$ yum -y install libnl libnl-devel  
No match for argument: libnl  
No match for argument: libnl-devel  
Error: Unable to find a match: libnl libnl-devel  
1  
2  
3  
4  
what 没有匹配的软件源? 我们只需要用到 VRRP 功能，只要确保 Use VRRP Framework 、Use VRRP VMAC、Use VRRP authentication 为 Yes 即可，当然也可以单独找 RPM 来安装 libnl 。

https://centos.pkgs.org/7/centos-x86\_64/libnl-1.1.4-3.el7.x86\_64.rpm.html

$ wget http://mirror.centos.org/centos/7/os/x86\_64/Packages/libnl-1.1.4-3.el7.x86\_64.rpm  
$ wget http://mirror.centos.org/centos/7/os/x86\_64/Packages/libnl-devel-1.1.4-3.el7.x86\_64.rpm  
$ rpm -ivh libnl-1.1.4-3.el7.x86\_64.rpm  
$ rpm -ivh libnl-devel-1.1.4-3.el7.x86\_64.rpm  
$ ./configure --prefix=/usr/local/keepalived  
$ make && make install  
1  
2  
3  
4  
5  
6  
配置 Keepalived

$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/init.d/keepalived /etc/init.d/  
$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/sysconfig/keepalived /etc/sysconfig/  
$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/  
$ mkdir -p /etc/keepalived  
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/  
$ chmod +x /etc/init.d/keepalived  
$ ip address \# 本机 ip 172.17.0.2 网卡 eth0  
1: lo: <LOOPBACK,UP,LOWER\_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
    inet 127.0.0.1/8 scope host lo  
       valid\_lft forever preferred\_lft forever  
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000  
    link/ipip 0.0.0.0 brd 0.0.0.0  
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000  
    link/tunnel6 :: brd ::  
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc noqueue state UP group default  
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0  
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0  
       valid\_lft forever preferred\_lft forever  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
默认将 Keepalived 配置文件放在 /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global\_defs \{  
     \# 负载均衡标识，在局域网内应该是唯一的  
   router\_id LVS\_DEVEL  
   enable\_script\_security  
   \# 建议新建用户来执行脚本  
   script\_user root  
\}

vrrp\_script chk\_nginx \{  
    script "/etc/keepalived/check.sh"  
    \# 每2秒检测一次nginx的运行状态  
    interval 2   
    \# 失败一次，将自己的优先级调整为-5  
    weight -5  
    \# require 2 failures for KO  
    fall 2  
    \# require 2 successes for OK  
    rise 2  
\}

vrrp\_instance VI\_1 \{  
    \# 标示状态为 MASTER 备份机为 BACKUP  
    state MASTER  
    \# 设置实例绑定的网卡  
    interface eth0  
    \# 同一实例下 virtual\_router\_id 必须相同  
    virtual\_router\_id 51  
    \# MASTER 权重要高于 BACKUP 比如 BACKUP 为99  
    priority 100  
    \# MASTER 与 BACKUP 负载均衡器之间同步检查的时间间隔，单位是秒  
    advert\_int 1   
    \# 设置认证  
    authentication \{   
            \# 主从服务器验证方式  
        auth\_type PASS  
        auth\_pass 1111  
    \}  
    \# 虚拟IP，两个节点设置必须一致，可以设置多个  
    virtual\_ipaddress \{   
        172.17.0.200  
        172.17.0.210  
    \}  
    track\_script \{  
       chk\_nginx  
    \}  
\}  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
$ cd /etc/keepalived  
$ touch check.sh  
$ chmod +x check.sh  
1  
2  
3  
check.sh 脚本内容如下

\#!/bin/bash  
Nginx=\`ps -ef | grep nginx | grep -v grep | wc -l\`  
echo 'nginx status ' $Nginx  
if \[ $Nginx -eq 0 \]  
then  
    echo 'nginx shutdown...'  
    nginx  
    echo 'start nginx ...'  
    sleep 2  
    if \[ \`ps -ef | grep nginx | grep -v grep | wc -l\` -eq 0 \]  
    then  
                ps -ef | grep keepalived | grep -v grep | awk '\{print $2\}' | xargs kill -9  
                echo 'kill keepalived...'  
    else  
    echo 'nginx started...'  
    fi  
fi  
echo 'script end...'  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
脚本作用：当 nginx 进程不存在时，会自动重启 nginx 服务 ，休眠 2 秒再次检查 nginx 进程，如果不存在就停止 keepalived 服务

测试脚本：

$ ps ax|grep nginx  
   40 pts/1    S+     0:00 grep --color=auto nginx  
$ ./check.sh  
nginx status  0  
nginx shutdown...  
start nginx ...  
nginx started...  
script end...  
$ ps ax|grep nginx  
   48 ?        Ss     0:00 nginx: master process nginx  
   49 ?        S      0:00 nginx: worker process  
   57 pts/1    S+     0:00 grep --color=auto nginx  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
检查配置

$ keepalived -t  
1  
至此，Keepalived 以及 Nginx 都配置完成。

测试

容器创建快照

$ docker ps -a | grep centos  
ace32d2fa597 centos "/bin/bash" About an hour ago Up About an hour  centos  
$ docker commit ace32d2fa597 centos\_keepalived\_nginx:0.0.1 \# 创建该容器的快照  
$ docker images | grep centos\_keepalived\_nginx  
centos\_keepalived\_nginx   0.0.1               801e75c18232        51 seconds ago      569MB  
1  
2  
3  
4  
5  
创建主从容器

根据快照创建主从容器

$ docker run --privileged -itd -p 6698:80 --name keepalived\_master --hostname keepalived\_master centos\_keepalived\_nginx:0.0.1 /usr/sbin/init  
$ docker run --privileged -itd -p 6699:80 --name  keepalived\_slave --hostname keepalived\_slave centos\_keepalived\_nginx:0.0.1 /usr/sbin/init  
1  
2  
进入 maser 容器修改配置

$ docker exec -it keepalived\_master bash \# 进入 master 容器  
$ vi /usr/share/nginx/html/index.html \# 任意修改一点内容与 slave 做区分  
$ nginx  
$ curl localhoast  
1  
2  
3  
4  
进入 slave 容器修改配置

$ docker exec -it keepalived\_master bash \# 进入 master 容器  
$ vi /usr/share/nginx/html/index.html \# 任意修改一点内容与 slave 做区分  
$ nginx  
$ curl localhoast  
1  
2  
3  
4  
测试 Keepalived

启动 master 容器中的 keepalived

$ docker exec -it keepalived\_master bash  
$ keepalived  
$ curl 172.17.0.200 \# curl 172.17.0.210  
$ ip address  
1: lo: <LOOPBACK,UP,LOWER\_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
    inet 127.0.0.1/8 scope host lo  
       valid\_lft forever preferred\_lft forever  
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000  
    link/ipip 0.0.0.0 brd 0.0.0.0  
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000  
    link/tunnel6 :: brd ::  
36: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc noqueue state UP group default  
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0  
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0  
       valid\_lft forever preferred\_lft forever  
    inet 172.17.0.200/32 scope global eth0  
       valid\_lft forever preferred\_lft forever  
    inet 172.17.0.210/32 scope global eth0  
       valid\_lft forever preferred\_lft forever  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
启动 slave 容器中的 keepalived

$ docker exec -it keepalived\_slave bash  
$ vi /etc/keepalived/keepalived.conf \# state 改为 BACKUP priority 改为 小于 100 的任意值 比如 99  
$ keepalived  
$ curl 172.17.0.200 \# curl 172.17.0.210  
1  
2  
3  
4  
总结：

你会发现访问 172.17.0.200 以及 172.17.0.210 都是访问到 maser 容器上的内容

如果关闭 master 上的 keepalived，就会自动切换访问到 slave 容器上的内容

当 nginx 进程不存在时尝试重新启动 nginx，如果 nginx 启动失败就关闭 keepalived 服务并实现自动切换

遗留问题：

Docker 中设置的虚拟 IP，如何让外网访问，如何做映射呢？Docker 研究得不深入，请多多指教。  
check.sh 脚本检查 Nginx 服务时有时间返回 0 有时候返回 2 (Nginx 服务都是关闭的状态)，但使用 touch 新建一个文件又没有问题！！！ 这是为何呢？