NGINX高可用之keepalived+nginx主从模式+主主模式配置实践

￡神魔★判官ぃ 2023-02-21 06:55 23阅读 0赞

nginx进程基于Master+Slave(worker)多进程模型，自身具有非常稳定的子进程管理功能。在Master进程分配模式下，Master进程永远不进行业务处理，只是进行任务分发，从而达到Master进程的存活高可靠性，Slave(worker)进程所有的业务信号都 由主进程发出，Slave(worker)进程所有的超时任务都会被Master中止，属于非阻塞式任务模型。

Keepalived是Linux下面实现VRRP备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。二者结合，可以构架出比较稳定的软件LB(LoadBalance)方案。

Keepalived是一个基于VRRP协议来实现的服务高可用方案，可以利用其来避免IP单点故障，类似的工具还有heartbeat、corosync、pacemaker。但是它一般不会单独出现，而是与其它负载均衡技术（如lvs、haproxy、nginx）一起工作来达到集群的高可用。

**VRRP协议**

VRRP全称 `Virtual Router Redundancy Protocol，即 虚拟路由冗余协议`。可以认为它是实现路由器高可用的容错协议，即将N台提供相同功能的路由器组成一个路由器组(Router Group)，这个组里面有一个master和多个backup，但在外界看来就像一台一样，构成虚拟路由器，拥有一个虚拟IP（vip，也就是路由器所在局域网内其他机器的默认路由），占有这个IP的master实际负责ARP相应和转发IP数据包，组中的其它路由器作为备份的角色处于待命状态。master会发组播消息，当backup在超时时间内收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master，保证路由器的高可用。

在VRRP协议实现里，虚拟路由器使用 00-00-5E-00-01-XX 作为虚拟MAC地址，XX就是唯一的 VRID （Virtual Router IDentifier），这个地址同一时间只有一个物理路由器占用。在虚拟路由器里面的物理路由器组里面通过多播IP地址 224.0.0.18 来定时发送通告消息。每个Router都有一个 1-255 之间的优先级别，级别最高的（highest priority）将成为主控（master）路由器。通过降低master的优先权可以让处于backup状态的路由器抢占（pro-empt）主路由器的状态，两个backup优先级相同的IP地址较大者为master，接管虚拟IP。

## **【1】基础环境准备** ##

两台服务器：192.168.88.129(主) 192.168.88.130(从)，每台服务器上分别安装nginx 、keepalived。

**首先要关闭防火墙、关闭selinux。当然，你可以选择不关闭，但是可能会遇到各种奇怪问题。**

> 关闭selinux可以参考博文 https://blog.csdn.net/J080624/article/details/79571986

安装nginx教程参考博文：[CentOS 7下使用yum安装Nginx][CentOS 7_yum_Nginx] ，[源码方式安装nginx][nginx]。

安装keepalived可使用yum直接安装：`yum -y install keepalived`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70]  
**默认keepalived.conf配置文件**

! Configuration File for keepalived
    
    global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 192.168.200.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.200.16
            192.168.200.17
            192.168.200.18
        }
    }
    
    virtual_server 192.168.200.100 443 {
        delay_loop 6
        lb_algo rr
        lb_kind NAT
        persistence_timeout 50
        protocol TCP
    
        real_server 192.168.201.100 443 {
            weight 1
            SSL_GET {
                url {
                  path /
                  digest ff20ad2481f97b1754ef3e12ecd3a9cc
                }
                url {
                  path /mrtg/
                  digest 9b3a0c85a887a256d6939da88aabd8cd
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }
    
    virtual_server 10.10.10.2 1358 {
        delay_loop 6
        lb_algo rr 
        lb_kind NAT
        persistence_timeout 50
        protocol TCP
    
        sorry_server 192.168.200.200 1358
    
        real_server 192.168.200.2 1358 {
            weight 1
            HTTP_GET {
                url { 
                  path /testurl/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl2/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl3/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    
        real_server 192.168.200.3 1358 {
            weight 1
            HTTP_GET {
                url { 
                  path /testurl/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334c
                }
                url { 
                  path /testurl2/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334c
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }
    
    virtual_server 10.10.10.3 1358 {
        delay_loop 3
        lb_algo rr 
        lb_kind NAT
        persistence_timeout 50
        protocol TCP
    
        real_server 192.168.200.4 1358 {
            weight 1
            HTTP_GET {
                url { 
                  path /testurl/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl2/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl3/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    
        real_server 192.168.200.5 1358 {
            weight 1
            HTTP_GET {
                url { 
                  path /testurl/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl2/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                url { 
                  path /testurl3/test.jsp
                  digest 640205b7b0fc66c1ea91c463fac6334d
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }

接下来就是要对该配置文件进行修改，该配置文件在/etc/keepalived/路径下。在默认的keepalive.conf里面还有 virtual\_server,real\_server 这样的配置，它是为lvs准备的。

--------------------

首先开始配置主从模式，实例图如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 1]

## **【2】修改keepalived.conf配置文件** ##

**① 修改主机keepalived.conf配置**

global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 192.168.88.129 #这里修改为本机IP
       smtp_connect_timeout 30
       router_id LVS_DEVEL
    }
    #添加检测脚本
    vrrp_script chk_http_port {
            script "/usr/local/nginx/nginx_check.sh"
            interval 2
            weight 2
    }
    vrrp_instance VI_1 {
        state MASTER   #主机这里是MASTER 从机是BACKUP
        interface ens33  #网卡
        virtual_router_id 51  # 主、从机的virtual_router_id必须相同
        priority 100   # 主备机取不同的优先级，主机优先级大
        advert_int 1  #心跳检测间隔时间
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.50   # VRRP 虚拟IP ；可换行输入多个进行绑定
        }
    }

**在默认的keepalive.conf里面还有 virtual\_server,real\_server 这样的配置，我们这用不到，它是为lvs准备的。**

/usr/local/nginx/nginx\_check.sh 脚本内容如下：

#!/bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        /usr/local/nginx/sbin/nginx
        sleep 2
        counter=$(ps -C nginx --no-heading|wc -l)
        if [ "${counter}" = "0" ]; then
            systemctl stop keepalived
        fi
    fi

如果发现nginx进程不存在，则尝试启动；sleep2秒后再次检测，如果还是不存在则认为启动失败，就停止keepalived服务。

--------------------

**② 修改从机keepalived.conf配置文件**

! Configuration File for keepalived
    global_defs {
       notification_email {  #指定keepalived在发生事件时(比如切换)发送通知邮件的邮箱
         acassen@firewall.loc  #设置报警邮件地址，可以设置多个，每行一个。 需开启本机的sendmail服务
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc   #keepalived在发生诸如切换操作时需要发送email通知地址
       smtp_server 192.168.88.129   #指定发送email的smtp服务器
       smtp_connect_timeout 30       #设置连接smtp server的超时时间
       router_id LVS_DEVEL            #运行keepalived的机器的一个标识，通常可设为hostname。故障发生时，发邮件时显示在邮件主题中的信息。
    }
    vrrp_script chk_http_port {
            script "/usr/local/nginx/nginx_check.sh"
            interval 2   #检测脚本执行间隔时间
            weight 2    #设置当前服务器权重增量
    }
    vrrp_instance VI_1 {
        state BACKUP
        interface ens33
        virtual_router_id 51
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.50
        }
    }

同样需要在/usr/local/nginx/路径下放上nginx\_check.sh脚本文件。

--------------------

**③ 启动nginx、keepalived**

/usr/local/nginx/sbin/nginx   #启动nginx
    
    systemctl start keepalived  #启动keepalived

## **【3】测试** ##

**① 浏览器访问http://192.168.88.50/**

192.168.88.129主nginx访问日志打印如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 2]  
**查看主服务器keepalived日志**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 3]

> Keepalived默认所有的日志都是写入到/var/log/message ，你可以使用命令 tail -f /var/log/messages|grep Keepalived 进行查看

尝试访问主服务器部署的页面http://192.168.88.50/edu/index.html：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 4]  
主机192.168.88.129使用ip address命令检测可以发现虚拟ip已经飘到了主机上面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 5]

**② 停掉主机的keepalived**

查看主机日志信息：

Jun 30 15:42:34 bogon Keepalived[5789]: Stopping
    Jun 30 15:42:34 bogon Keepalived_healthcheckers[5791]: Stopped
    Jun 30 15:42:34 bogon Keepalived_vrrp[5792]: VRRP_Instance(VI_1) sent 0 priority
    Jun 30 15:42:34 bogon Keepalived_vrrp[5792]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jun 30 15:42:35 bogon Keepalived_vrrp[5792]: Stopped
    Jun 30 15:42:35 bogon Keepalived[5789]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

查看从机日志信息：

#当主服务器停掉后，从机将会切换为MASTER状态
    Jun 30 13:26:14 bogon avahi-daemon[731]: Withdrawing address record for 192.168.88.50 on ens33.
    Jun 30 13:28:55 bogon Keepalived_vrrp[2966]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:28:56 bogon avahi-daemon[731]: Registering new address record for 192.168.88.50 on ens33.IPv4.
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.50
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 13:29:01 bogon Keepalived_vrrp[2966]: Sending gratuitous ARP on ens33 for 192.168.88.50

检测从机ip地址:

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 6]

可以发现从机的keepalived已经切换为了MASTER状态，且从机的ip address检测时发现VIP绑定到了ens33上面。此时访问http://192.168.88.50/就会跑到从服务器的nginx上面！  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 7]

**再次启动主服务器的keepalived，查看主机日志：**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 8]  
主服务器的keepalived进入MASTER STATE，从服务器的keepalived进入BACKUP STATE：

#从服务器日志
    Jun 30 15:01:24 bogon Keepalived_vrrp[1900]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90
    Jun 30 15:01:24 bogon Keepalived_vrrp[1900]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Jun 30 15:01:24 bogon Keepalived_vrrp[1900]: VRRP_Instance(VI_1) removing protocol VIPs.

**③ 停掉主服务器的nginx，不停keepalived**

则会执行脚本nginx\_check.sh进行nginx启动！  
![在这里插入图片描述][20200630154007633.png]  
**总结**

*  master没挂，则master占有vip且nginx运行在master上
 *  master挂了，则slave抢占vip且在slave上运行nginx服务
 *  如果master上的nginx服务挂了，则nginx会自动重启，重启失败后会自动关闭keepalived，这样vip资源也会转移到slave上。
 *  master和slave两边都开启nginx服务，无论master还是slave，当其中的一个keepalived服务停止后，vip都会漂移到keepalived服务还在的节点上；
 *  如果要想使nginx服务挂了，vip也漂移到另一个节点，则必须用脚本或者在配置文件里面用shell命令来控制。（nginx服务宕停后会自动启动，启动失败后会强制关闭keepalived，从而致使vip资源漂移到另一台机器上）

**一种常见的主从热备应用实例如下图所示：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70_pic_center]

主从的缺点在于如果主机一直稳定，那么从机就会一直处于空闲状态，造成了资源的浪费。

--------------------

## **【4】问题总结** ##

**① 不能完全停掉keepalived进程**

使用yum 安装的keepalived，当使用命令`systemctl stop keepalived` 停掉keepalived服务时，使用ps命令检测发现还存，如下图示：  
![在这里插入图片描述][20200630124234932.png]  
当使用命令`systemctl status keepalived`检测状态，会发现有一条警告信息，如下所示：

Jun 30 11:52:20 bogon systemd[1]: Stopped LVS and VRRP High Availability Monitor.
    Jun 30 11:52:20 bogon systemd[1]: Stopping LVS and VRRP High Availability Monitor...
    Jun 30 11:52:20 bogon systemd[1]: Starting LVS and VRRP High Availability Monitor...
    Jun 30 11:52:20 bogon systemd[1]: Can't open PID file /var/run/keepalived.pid (yet?) after start: No such file or directory
    Jun 30 11:52:20 bogon systemd[1]: Started LVS and VRRP High Availability Monitor.
    Jun 30 12:30:57 bogon systemd[1]: Stopped LVS and VRRP High Availability Monitor.

查看其服务脚本信息`vim /usr/lib/systemd/system/keepalived.service`：

[Unit]
    Description=LVS and VRRP High Availability Monitor
    After=syslog.target network-online.target
    
    [Service]
    Type=forking
    PIDFile=/var/run/keepalived.pid
    KillMode=process
    EnvironmentFile=-/etc/sysconfig/keepalived
    ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS
    ExecReload=/bin/kill -HUP $MAINPID
    
    [Install]
    WantedBy=multi-user.target

`KillMode=process`的大致意思是当停止keepalived的时候只会停掉主进程，而主进程产生的子进程是不会被干掉的。而killmode的默认值是control-group，意思时所有进程都会被干掉，这里选择把这项注释掉。

重载配置

systemctl daemon-reload

杀掉keepalived所有进程，然后再次启动keepalived即可：

pkill -9 keepalived

--------------------

**② Unsafe permissions found for script ‘/usr/local/nginx/nginx\_check.sh’.**

说明你赋予的权限太高了，可以使用如下命令尝试：

chmod 755 /usr/local/nginx/nginx_check.sh

--------------------

## **【5】keepalived与heartbeat/corosync等比较** ##

**Heartbeat、Corosync、Keepalived这三个集群组件我们到底选哪个好呢？**

首先要说明的是，Heartbeat、Corosync是属于同一类型，Keepalived与Heartbeat、Corosync，根本不是同一类型的。

Keepalived使用的vrrp协议方式，虚拟路由冗余协议 (Virtual Router Redundancy Protocol，简称VRRP)；Heartbeat或Corosync是基于主机或网络服务的高可用方式。

**简单的说就是，Keepalived的目的是模拟路由器的高可用，Heartbeat或Corosync的目的是实现Service的高可用。**

`所以一般Keepalived是实现前端高可用`，常用的前端高可用的组合有`LVS+Keepalived、Nginx+Keepalived、HAproxy+Keepalived`。

而Heartbeat或Corosync是实现服务的高可用。常见的组合有Heartbeat v3(Corosync)+Pacemaker+NFS+Httpd 实现Web服务器的高可用、Heartbeat v3(Corosync)+Pacemaker+NFS+MySQL 实现MySQL服务器的高可用。

总结一下，Keepalived中实现轻量级的高可用，一般用于前端高可用，且不需要共享存储，一般常用于两个节点的高可用。而Heartbeat(或Corosync)一般用于服务的高可用，且需要共享存储，一般用于多节点的高可用。这个问题我们说明白了。

**那heartbaet与corosync又应该选择哪个好？**

一般用corosync，因为corosync的运行机制更优于heartbeat，就连从heartbeat分离出来的pacemaker都说在以后的开发当中更倾向于corosync，所以现在corosync+pacemaker是最佳组合。

参考博文：https://blog.csdn.net/weixin\_34389926/article/details/85837841

--------------------

## **【6】主主模式配置实践** ##

主主模式相对于主从模式而言区别在于，每个机器都互为主从。示意图如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 9]

**① 修改192.168.88.129的keepalived.conf配置**

添加如下配置：

vrrp_instance VI_2 {
        state BACKUP  #这里修改为BACKUP
        interface ens33
        virtual_router_id 52 #这里修改为52
        priority 90    #修改优先级
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.60  #绑定另外一个VIP
        }
    
    track_script {
       chk_http_port
    }
    }

完整配置如下：

global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
    }
    vrrp_script chk_http_port {
            script "/usr/local/nginx/nginx_check.sh"
            interval 2
            weight 2
    }
    vrrp_instance VI_1 {
        state MASTER
        interface ens33
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
                    192.168.88.50
        }
    
    track_script {
       chk_http_port
    }
    }
    
    vrrp_instance VI_2 {
        state BACKUP
        interface ens33
        virtual_router_id 52
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.60
        }
    
    track_script {
       chk_http_port
    }
    }

--------------------

**② 修改192.168.88.130的keepalived.conf配置**

添加配置如下：

vrrp_instance VI_2 {
        state MASTER
        interface ens33
        virtual_router_id 52
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.60
        }
    
    track_script {
       chk_http_port
    }
    }

完整配置如下：

global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
    }
    vrrp_script chk_http_port {
            script "/usr/local/nginx/nginx_check.sh"
            interval 2
            weight 2
    }
    vrrp_instance VI_1 {
        state BACKUP
        interface ens33
        virtual_router_id 51
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
                    192.168.88.50
        }
    
    track_script {
       chk_http_port
    }
    }
    vrrp_instance VI_2 {
        state MASTER
        interface ens33
        virtual_router_id 52
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.88.60
        }
    
    track_script {
       chk_http_port
    }
    }

分别重启两台服务器上的keepalived服务，查看对应的日志信息。

192.168.88.129上keepalived日志信息如下：

Jun 30 17:00:13 bogon Keepalived[7503]: Stopping
    Jun 30 17:00:13 bogon Keepalived_healthcheckers[7504]: Stopped
    Jun 30 17:00:13 bogon Keepalived_vrrp[7505]: VRRP_Instance(VI_1) sent 0 priority
    Jun 30 17:00:13 bogon Keepalived_vrrp[7505]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jun 30 17:00:14 bogon Keepalived_vrrp[7505]: Stopped
    Jun 30 17:00:14 bogon Keepalived[7503]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
    Jun 30 17:00:14 bogon Keepalived[16827]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
    Jun 30 17:00:14 bogon Keepalived[16827]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:14 bogon Keepalived[16828]: Starting Healthcheck child process, pid=16830
    Jun 30 17:00:14 bogon Keepalived[16828]: Starting VRRP child process, pid=16831
    Jun 30 17:00:14 bogon Keepalived_healthcheckers[16830]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:14 bogon Keepalived_vrrp[16831]: Registering Kernel netlink reflector
    Jun 30 17:00:14 bogon Keepalived_vrrp[16831]: Registering Kernel netlink command channel
    Jun 30 17:00:14 bogon Keepalived_vrrp[16831]: Registering gratuitous ARP shared channel
    Jun 30 17:00:14 bogon Keepalived_vrrp[16831]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:14 bogon Keepalived_vrrp[16831]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) removing protocol VIPs.
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: Using LinkWatch kernel netlink reflector...
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Entering BACKUP STATE
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
    Jun 30 17:00:15 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.50
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:16 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:17 bogon Keepalived_vrrp[16831]: /usr/local/nginx/nginx_check.sh exited due to signal 15
    Jun 30 17:00:17 bogon Keepalived_vrrp[16831]: VRRP_Script(chk_http_port) succeeded
    Jun 30 17:00:18 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) Changing effective priority from 100 to 102
    Jun 30 17:00:18 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Changing effective priority from 90 to 92
    Jun 30 17:00:18 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Transition to MASTER STATE
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Entering MASTER STATE
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) setting protocol VIPs.
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.60
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:19 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.50
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:21 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.60
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:24 bogon Keepalived_vrrp[16831]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:28 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Received advert with higher priority 100, ours 92
    Jun 30 17:00:28 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) Entering BACKUP STATE
    Jun 30 17:00:28 bogon Keepalived_vrrp[16831]: VRRP_Instance(VI_2) removing protocol VIPs.

可以看到 对VRRP\_Instance(VI\_1)而言，192.168.88.129为MASTER；对VRRP\_Instance(VI\_2)而言，192.168.88.129为BACKUP 。

192.168.88.130上keepalived日志信息如下：

Jun 30 17:00:13 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.50
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:14 bogon Keepalived_vrrp[5752]: Sending gratuitous ARP on ens33 for 192.168.88.50
    Jun 30 17:00:15 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 92
    Jun 30 17:00:15 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Jun 30 17:00:15 bogon Keepalived_vrrp[5752]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jun 30 17:00:26 bogon Keepalived[5749]: Stopping
    Jun 30 17:00:26 bogon Keepalived_healthcheckers[5751]: Stopped
    Jun 30 17:00:27 bogon Keepalived_vrrp[5752]: Stopped
    Jun 30 17:00:27 bogon Keepalived[5749]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
    Jun 30 17:00:27 bogon Keepalived[19665]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
    Jun 30 17:00:27 bogon Keepalived[19665]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:27 bogon Keepalived[19666]: Starting Healthcheck child process, pid=19668
    Jun 30 17:00:27 bogon Keepalived[19666]: Starting VRRP child process, pid=19669
    Jun 30 17:00:27 bogon Keepalived_healthcheckers[19668]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:27 bogon Keepalived_vrrp[19669]: Registering Kernel netlink reflector
    Jun 30 17:00:27 bogon Keepalived_vrrp[19669]: Registering Kernel netlink command channel
    Jun 30 17:00:27 bogon Keepalived_vrrp[19669]: Registering gratuitous ARP shared channel
    Jun 30 17:00:27 bogon Keepalived_vrrp[19669]: Opening file '/etc/keepalived/keepalived.conf'.
    Jun 30 17:00:27 bogon Keepalived_vrrp[19669]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) removing protocol VIPs.
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: Using LinkWatch kernel netlink reflector...
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP_Script(chk_http_port) succeeded
    Jun 30 17:00:28 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) Transition to MASTER STATE
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_1) Changing effective priority from 90 to 92
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) Changing effective priority from 100 to 102
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) Entering MASTER STATE
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) setting protocol VIPs.
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.60
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:29 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: VRRP_Instance(VI_2) Sending/queueing gratuitous ARPs on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60
    Jun 30 17:00:34 bogon Keepalived_vrrp[19669]: Sending gratuitous ARP on ens33 for 192.168.88.60

可以看到 对VRRP\_Instance(VI\_1)而言，192.168.88.130为BACKUP；对VRRP\_Instance(VI\_2)而言，192.168.88.130为MASTER。

浏览器访问http://192.168.88.50/，192.168.88.129上的nginx会处理该请求；浏览器访问http://192.168.88.60/，192.168.88.130上的nginx会处理该请求 !

当任何一台服务器上面的keepalived服务停掉后，另外一台服务器上面的keepalived都会进入MASTER状态处理请求。如这里停掉192.168.88.129上的keepalived服务，则192.168.88.130上的keepalived服务的VRRP\_Instance(VI\_1)进入MASTER状态，并对http://192.168.88.50/ http://192.168.88.60/进行处理！  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 10]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 11]  
**查看192.168.88.130此时ip 地址如下：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 12]

[CentOS 7_yum_Nginx]: https://blog.csdn.net/J080624/article/details/78087813
[nginx]: https://blog.csdn.net/J080624/article/details/79240685
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/202006291719347.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200630175310781.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200630103605868.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200630140348350.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200630103936235.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200630104138386.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200630134555718.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200630151141413.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200630150314117.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[20200630154007633.png]: https://img-blog.csdnimg.cn/20200630154007633.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20200630192941990.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70#pic_center
[20200630124234932.png]: https://img-blog.csdnimg.cn/20200630124234932.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200630185858272.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200630171509638.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200630171432935.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200630171833417.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0owODA2MjQ=,size_16,color_FFFFFF,t_70