Nginx基于keepalived实现高可用

妖狐艹你老母 2022-04-15 01:09 205阅读 0赞

一，安装包

\* nginx-1.14.0.tar.gz

\* keepalived-2.0.6.tar.gz

-- 基于Lua脚本实现keepalived服务关联处理，之前有安装，不再安装

\* lua-5.3.5.tar.gz

二，Nginx安装

1，tar包解压

[root@localhost develop]# tar -zxvf /usr/myapp/nginx-1.14.0.tar.gz -C /usr/develop/nginx/

2，源码编译，在文件根路径下执行

[root@localhost nginx-1.14.0]# ./configure --prefix=/usr/develop/nginx/

3，编译过程中可能存在的部分插件安装

[root@localhost nginx-1.14.0]#  yum install pcre-devel
    [root@localhost nginx-1.14.0]#  yum install zlib-devel

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70][]

4，安装

[root@localhost nginx-1.14.0]# make && make install

![20181119213910654.png][]

![20181119214057585.png][]

5，启动 | 终止 | 重启 | 带配置启动

a, 启动
    	[root@localhost nginx]# ./sbin/nginx
    b, 终止
    	[root@localhost nginx]# ./sbin/nginx -s stop
    c, 重启
    	[root@localhost nginx]# ./sbin/nginx -s reload
    d, 带配置路径启动
    	[root@localhost nginx]# ./nginx -p conf文件所在路径

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 1][]

6，Nginx安装成功

三，Keepalived安装

1，tar包解压

[root@localhost develop]# mkdir keepalivd
    [root@localhost develop]# tar -zxvf /usr/myapp/keepalived-2.0.6.tar.gz -C /usr/develop/keepalivd/

2，编译

[root@localhost keepalivd]# cd keepalived-2.0.6/
    [root@localhost keepalived-2.0.6]# ./configure --prefix=/usr/develop/keepalivd/ --sysconf=/usr/develop/keepalivd/

3，编译过程中可能存在的部分插件安装

// !!! OpenSSL is not properly installed on your system. !!!
    // !!! Can not include OpenSSL headers files.            !!!
    [root@localhost keepalived-2.0.6]# yum install openssl-devel
    // *** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
    [root@localhost keepalived-2.0.6]# yum install libnl-devel
    // checking for libnfnetlink/libnfnetlink.h... no
    // configure: error: libnfnetlink headers missing
    [root@localhost keepalived-2.0.6]# yum install libnfnetlink-devel

4，安装

[root@localhost keepalived-2.0.6]# make && make install

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 2][]

5，创建软连接

// 注意 : 此处路径必须为绝对路径
    [root@localhost sbin]# ln -s /usr/develop/keepalivd/sbin/keepalived /usr/sbin/

6，复制启动程序到etc下

[root@localhost keepalivd]# cp ./keepalived-2.0.6/keepalived/etc/init.d/keepalived /etc/init.d

7，添加keepalived服务到chkconfig

[root@localhost keepalivd]# chkconfig --add keepalived

8，在chkconfig中开启keepalived服务

[root@localhost keepalivd]# chkconfig keepalived on

9，配置启动路径为解压时重定向的路径

[root@localhost keepalivd]# vim /etc/init.d/keepalived

\* 对下一条配置信息进行修改

![20181119215839682.png][]

\* 修改路径为编译时重定向的sysconfig路径

[root@localhost keepalivd]# cd sysconfig/
    [root@localhost sysconfig]# pwd
    /usr/develop/keepalivd/sysconfig
    [root@localhost sysconfig]#

\* 修改后路径

![20181119220008721.png][]

10，修改默认配置加载系统路径及日志路径

\* 位置

[root@localhost sysconfig]# pwd
    /usr/develop/keepalivd/sysconfig
    [root@localhost sysconfig]# ll
    total 4
    -rw-r--r--. 1 root root 667 Nov 19 13:52 keepalived
    [root@localhost sysconfig]# vim keepalived

\* 要修改的内容

![20181119220249515.png][]

\* 修改为

// -f表示对配置路径重定向
    // -S表示对日志路径重定向, 0表示配置为local0的路径
    KEEPALIVED_OPTIONS="-f /usr/develop/keepalivd/keepalived/keepalived.conf -D -S 0"

\* 日志local0重定向处理

// 进入到/etc/rsyslog.conf
    [root@localhost sysconfig]# vim /etc/rsyslog.conf

\* 配置local0路径

![20181119220737598.png][]

\* 配置rsyslog配置文件

[root@localhost conf]# vim /etc/sysconfig/rsyslog

![20181207191632170.png][]

\* 重启rsyslog服务，引用配置信息

[root@localhost sysconfig]# service rsyslog restart

11，配置核心配置文件 ./keepalived/keepalived.conf

[root@localhost keepalived]# cat keepalived/keepalived.conf    
    ! Configuration File for keepalived
    
    global_defs {
       # 运行keepalived服务器的标识, 网络内唯一
       router_id LVS_DEVEL
    }
    
    # VRRP实例定义部分
    vrrp_instance VI_1 {
        # 设置LVS状态, 为MASTER或者BACKER,必须大写
        state MASTER
        # 网卡表示, 通过ifconfig查看
        interface eth0
        # 设置虚拟路由标示，这个标示是一个数字，同一个vrrp实例使用唯一标示
        virtual_router_id 51
        # 定义优先级，数字越大优先级越高，在一个vrrp——instance下，master的优先级必须大于backup
        priority 100
        # 设定master与backup负载均衡器之间同步检查的时间间隔，单位是秒
        advert_int 1
        # 设置验证类型和密码
        authentication {
            auth_type PASS
            # 验证密码，同一个vrrp_instance下MASTER和BACKUP密码必须相同
            auth_pass 1111
        }
        # 设置虚拟ip地址，可以设置多个，每行一个
        virtual_ipaddress {
            192.168.91.127
        }
    }
    
    # 设置虚拟服务器，需要指定虚拟ip和服务端口
    # 服务器IP与虚拟IP一致
    virtual_server 192.168.91.127 80 {
        # 健康检查时间间隔
        delay_loop 6
        # 负载均衡调度算法
        lb_algo rr
        # 负载均衡转发规则
        lb_kind NAT
        # 设置会话保持时间
        persistence_timeout 50
        # 指定转化协议, TCP/UDP
        protocol TCP
    
        # 服务器节点, 配置IP和端口为真实IP和端口
        real_server 192.168.91.128 80 {
            # 设置当前服务权重
            weight 1
            TCP_CHECK {
                # 超时时间
                connect_timeout 3
                # 重试间隔
                retry 3
                # 检测端口
                connect_port 80
            }
        }
    }

12，启动keepalived

[root@localhost sysconfig]# service keepalived start

\* 主路径访问

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 3][]

\* 虚拟路径访问

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 4][]

13，keepalived安装完成

四，Tomcat直接解压启动， 并通过Nginx实现反向代理，此处不做演示，效果如下

![2018111922302993.png][]

五，Nginx + Keepalived实现高可用

1，Nginx不进行特别配置，其中一台有做反向代理的部分实验，这一块后续补充

2，Keepalived\_MASTER配置

3，Keepalived\_BACKUP配置

[root@localhost keepalived]# cat keepalived.conf 
    ! Configuration File for keepalived
    
    global_defs {
       router_id LVS_DEVEL
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.91.127
        }
    }
    
    virtual_server 192.168.91.127 80 {
        delay_loop 6
        lb_algo rr
        lb_kind NAT
        persistence_timeout 50
        protocol TCP
    
        real_server 192.168.91.129 80 {
            weight 1
            TCP_CHECK {
                connect_timeout 3
                retry 3
                connect_port 80
            }
        }
    }

4，分别启动Ningx和Keepalived服务

\* MASTER节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 5][]

\* BACKUP节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 6][]

\* 虚拟节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 7][]

\* 查看VIP

-- 主节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 8][]

-- 从节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 9][]

\* 关掉主节点服务，模拟宕机处理，查看VIP漂移

-- 主节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 10][]

-- 从节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 11][]

-- 虚拟节点访问

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 12][]

\* 可以看到从节点拿到了VIP，IP漂移完成

6，Keepalived与所监控服务同步宕机处理

\* 上述配置存在一个问题，即Nginx节点宕机后，Keepalived节点可以监控到服务宕机，但是并没有做同步宕机处理。在这种情况下，如果主节点的被监控服务宕机，依旧会造成服务器崩溃；

\* 如下图，杀掉主节点的Nginx服务，主节点并没有释放虚拟IP的VIP权限，页面不能正常访问

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 13][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 14][]

\* 基于这种现象，可以基于Lua脚本实现同步宕机处理，然后引入到配置信息

-- Lua脚本实现，对nginx进程进行监控，如果nginx服务宕机，则尝试重启，重启失败后，关闭keepalived服务

#!bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        /usr/develop/nginx/sbin/nginx
        sleep 2
        counter=$(ps -C nginx --no-heading|wc -l)
        if [ "${counter}" = "0" ]; then
            service keepalived stop
        fi
    fi

// 授权
    chmod +x nginx-alive-check.sh

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 15][]

\* Keepalived.conf核心配置，添加可执行脚本的权限及vrrp\_script脚本块，并引入到vrrp实例中进行执行

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 17][]

\* 两台机器全部配置完成后，查看运行结果，从截图中可以看出，杀死主节点的进程后，keepalived对主节点的进程进行了二次启动，继续保留VIP

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 18][]

\* 修改主节点的Shell脚本，去除重新启动nginx操作；可以看到杀掉nginx进程后，keepalived进程同步关闭，且VIP漂移到从节点

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 19][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 20][]

\* 页面访问也从主节点的Tomcat页面变为从节点的nginx页面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 21][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70]: /images/20220415/15d4f5fcf47c49c8971d5d8a4b52212c.png
[20181119213910654.png]: /images/20220415/0f04379e97cd4df68fa04fb4fc7f3ef6.png
[20181119214057585.png]: /images/20220415/04421e51492044c799ac2677f8a49b6d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 1]: /images/20220415/6be5e95bb06c4aa09a794e80872ea6b3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 2]: /images/20220415/fb4d5ad1b6ae467da1aaf0a5c9f20503.png
[20181119215839682.png]: /images/20220415/2a4bc90ab1ce49958ca63130589b0989.png
[20181119220008721.png]: /images/20220415/fedd18d8d0fb420d91e7acdc6cc17fa0.png
[20181119220249515.png]: /images/20220415/99c6a4a99ea1476596cf5461c0e80d40.png
[20181119220737598.png]: /images/20220415/97dd8c6331cb4905bef525f1460a4a3c.png
[20181207191632170.png]: /images/20220415/8debafb32db24fa080611365f1afd8ff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 3]: /images/20220415/a639ef14319547019bf78b95ded29ce4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 4]: /images/20220415/8e5579af06bf4500a24829b157002377.png
[2018111922302993.png]: /images/20220415/1aae49713b434dbdaf2ad9c00f0067ca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 5]: /images/20220415/95ed21702b0c43b1a1f24c83101b296d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 6]: /images/20220415/f3c59f18971144ad88d76077d716468e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 7]: /images/20220415/8cd63ff39fd549febc00d08dcd7eb57f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 8]: /images/20220415/0b731519ef724a8ab28a77ede6967212.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 9]: /images/20220415/f70ce89e529048a7ad41419ffdc06e1f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 10]: /images/20220415/1a9aad85398e4994bf53f81a78a4b883.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 11]: /images/20220415/1952e40b439d4304bb71d8c60addc659.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 12]: /images/20220415/ec146fc9669b4e48bd4e85ea935cec23.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 13]: /images/20220415/3dec8e81ba114e8888fe479c95f52bbc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 14]: /images/20220415/42381fe6417c45d8bedd9a8a72a54133.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 15]: /images/20220415/68a344b5d3984c4d8de68804965c3700.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 16]: /images/20220415/1e9e6f82475341b384aa998be2b985c4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 17]: /images/20220415/3164c24e1ef34f279dd0c33646ee25be.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 18]: /images/20220415/e71bd36d5c8d41d49195bc71ba681d34.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 19]: /images/20220415/edfe02c18bce4ba9b42356966013cba3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 20]: /images/20220415/3ef562acb8ba483590c0b7d479df417b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE5NzYzODg_size_16_color_FFFFFF_t_70 21]: /images/20220415/5cf180f17f304f0ab9a74d20e01e687c.png