高可用系统组件keepalived部署

忘是亡心i 2022-06-12 10:43 370阅读 0赞

![Image 1][]

# 1 keepalived简介 #

## 1.1什么是Keepalived？ ##

什么是Keepalived呢，keepalived观其名可知，保持存活，在网络里面就是保持在线了，也就是所谓的高可用或热备，用来防止单点故障(单点故障是指一旦某一点出现故障就会导致整个系统架构的不可用)的发生，那说到keepalived时不得不说的一个协议就是VRRP协议，可以说这个协议就是keepalived实现的基础，那么首先我们来看看VRRP协议.

## 1.2 VRRP协议简介 ##

学过网络的朋友都知道，网络在设计的时候必须考虑到冗余容灾，包括线路冗余，设备冗余等，防止网络存在单点故障，那在路由器或三层交换机处实现冗余就显得尤为重要，在网络里面有个协议就是来做这事的，这个协议就是VRRP协议，Keepalived就是巧用VRRP协议来实现高可用性(HA)的.

LAN客户端判定哪个路由器应该为其到达目标主机的下一跳网关的方式有动态及静态决策两种方式，其中，常见的动态路由发现方式有如下几种：

1、Proxy ARP ——客户端使用ARP协议获取其想要到达的目标，而后，由某路由以其MAC地址响应此ARP请求；

2、RoutingProtocol —— 客户端监听动态路由更新(如通过RIP或OSPF协议)并以之重建自己的路由表；

3、ICMP IRDP(Router Discovery Protocol) 客户端 —— 客户端主机运行一个ICMP路由发现客户端程序；

动态路由发现协议的不足之处在于它会导致在客户端引起一定的配置和处理方面的开销，并且，如果路由器故障，切换至其它路由器的过程会比较慢。解决此类问题的一个方案是为客户端静态配置默认路由设备，这大大简化了客户端的处理过程，但也会带来单点故障类的问题。默认网关故障时，LAN客户端仅能实现本地通信。

VRRP可以通过在一组路由器(一个VRRP组)之间共享一个虚拟IP(VIP)解决静态配置的问题，此时仅需要客户端以VIP作为其默认网关即可。

VRRP全称 VirtualRouter Redundancy Protocol，即 [虚拟路由冗余协议][Link 1]。可以认为它是实现路由器高可用的容错协议，即将N台提供相同功能的路由器组成一个路由器组(Router Group)，这个组里面有一个master和多个backup，但在外界看来就像一台一样，构成虚拟路由器，拥有一个虚拟IP（vip，也就是路由器所在局域网内其他机器的默认路由），占有这个IP的master实际负责ARP相应和转发IP数据包，组中的其它路由器作为备份的角色处于待命状态。master会发组播消息，当backup在超时时间内收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master，保证路由器的高可用。

在VRRP协议实现里，虚拟路由器使用00-00-5E-00-01-XX 作为虚拟MAC地址，XX就是唯一的 VRID （Virtual Router IDentifier），这个地址同一时间只有一个物理路由器占用。在虚拟路由器里面的物理路由器组里面通过多播IP地址 224.0.0.18 来定时发送通告消息。每个Router都有一个 1-255 之间的优先级别，级别最高的（highest priority）将成为主控（master）路由器。通过降低master的优先权可以让处于backup状态的路由器抢占（pro-empt）主路由器的状态，两个backup优先级相同的IP地址较大者为master，接管虚拟IP。

## 1.3 keepalived与同类型组件的比较 ##

Heartbeat、Corosync、Keepalived这三个集群组件我们到底选哪个好，首先我想说明的是，Heartbeat、Corosync是属于同一类型，Keepalived与Heartbeat、Corosync，根本不是同一类型的。Keepalived使用的vrrp协议方式，虚拟路由冗余协议 (Virtual Router Redundancy Protocol，简称VRRP)；Heartbeat或Corosync是基于主机或网络服务的高可用方式；简单的说就是，Keepalived的目的是模拟路由器的高可用，Heartbeat或Corosync的目的是实现Service的高可用。

所以一般Keepalived是实现前端高可用，常用的前端高可用的组合有，就是我们常见的LVS+Keepalived、Nginx+Keepalived、HAproxy+Keepalived。而Heartbeat或Corosync是实现服务的高可用，常见的组合有Heartbeat v3(Corosync)+Pacemaker+NFS+Httpd 实现Web服务器的高可用、Heartbeatv3(Corosync)+Pacemaker+NFS+MySQL 实现MySQL服务器的高可用。总结一下，Keepalived中实现轻量级的高可用，一般用于前端高可用，且不需要共享存储，一般常用于两个节点的高可用。而Heartbeat(或Corosync)一般用于服务的高可用，且需要共享存储，一般用于多节点的高可用。这个问题我们说明白了。

又有博友会问了，那heartbaet与corosync我们又应该选择哪个好啊，我想说我们一般用corosync，因为corosync的运行机制更优于heartbeat，就连从heartbeat分离出来的pacemaker都说在以后的开发当中更倾向于corosync，所以现在corosync+pacemaker是最佳组合。

# 2 安装keepalived #

第一步，安装基础库

yum -y install openssl-devel libnl3-develipset-devel iptables-devel libnfnetlink-devel net-snmp-devel

第二步，源码安装keepalived

1. tar zxf keepalived-1.3.5.tar.gz

2.cd keepalived-1.3.5

3.'./configure--prefix=/usr/local/keepalived/'

4.'make'

5.'make install'. This will install keepalived on your system,

binaries and configuration file :

\* keepalived : The keepalived daemon program.

\* genhash :The MD5 url digest generator. You need it to

configure HTTP GET check andSSL GET check in

order to compute MD5SUM digestetalon.

\* /etc/keepalived/keepalived.conf

拷贝需要的文件：

$ cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/keepalived

$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

$ mkdir -p /etc/keepalived/

$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

/etc/keepalived/keepalived.conf是默认的配置文件

# 3 keepalived + Nginx部署 #

## 3.1 部署拓补图 ##

基于如下的拓扑图：

![20170721164917450][]

这里Nginx的角色是负载均衡器或者反向代理服务器。

## 3.2 Nginx监控脚本 ##

该脚本检测ngnix的运行状态，并在nginx进程不存在时尝试重新启动ngnix，如果启动失败则停止keepalived，准备让其它机器接管。

/etc/keepalived/check\_nginx.sh：

#!/bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        /usr/local/nginx/sbin/nginx
        sleep 2
        counter=$(ps -C nginx --no-heading|wc -l)
        if [ "${counter}" = "0" ]; then
            #/etc/init.d/keepalived stop
    		exit 1
        fi
    fi
    exit 0

脚本中/etc/init.d/keepalived stop根据需要看是否要注释掉：

1. 若注释掉，通过脚本执行的返回码，改变优先级，keepalived继续发送通告消息，backup比较优先级再决定。但是这种方式有时候会导致backup重新选举出master失败。比如：若master监控的服务挂了，优先级降低，若此时master降低的优先级还是比backup的优先级高，那么backup就不能重新选举出新的master。

2. 若不注释，脚本里面检测到异常，直接关闭keepalived进程，backup机器接收不到通告消息会重新选举出master接管VIP。这种方式可以保证backup能够重新选举出新的master。

**keepalived根据监控脚本的返回码调整优先级：**

☉如果脚本返回码为0，并且weight配置的值大于0，则优先级相应的增加；

☉如果脚本返回码为非0，并且weight配置的值小于0，则优先级相应的减少；

☉其他情况，原本配置的优先级不变，即配置文件中priority对应的值。

提示：

优先级不会不断的提高或者降低；

可以编写多个检测脚本并为每个检测脚本设置不同的weight（在配置中列出就行）；

不管提高优先级还是降低优先级，最终优先级的范围是在\[1,254\]，不会出现优先级小于等于0或者优先级大于等于255的情况；

在MASTER节点的vrrp\_instance 中配置 nopreempt ，当它异常恢复后，即使它 prio 更高也不会抢占，这样可以避免正常情况下做无谓的切换。

以上可以做到利用脚本检测业务进程的状态，并动态调整优先级从而实现主备切换。

## 3.3 配置keepalived.conf ##

###       keepalived.conf样例 ###

! Configuration File for keepalived
    global_defs {
       notification_email {
         test1@163.com
       }
       notification_email_from test2@163.com
       smtp_server smtp.163.com
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    vrrp_script chk_nginx {
        script "/etc/keepalived/check_nginx.sh"
        interval 4
        weight -5
        fall 3  
        rise 2
    }
    vrrp_instance VI_1 {
        state MASTER
        interface eth1
        virtual_router_id 51
        priority 100
    advert_int 1
    nopreempt
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
        	172.30.1.10
        }
        track_script {
           chk_nginx
        }
    }

以上是keepalived MASTER节点配置文件/etc/keepalived/keepalived.conf的配置信息。在BACKUP节点，只需把vrrp\_instance->state改为BACKUP，vrrp\_instance->priority改为99即可。

在默认的keepalive.conf里面还有virtual\_server,real\_server 这样的配置，我们这用不到，它是为lvs准备的。notify 可以定义在切换成MASTER或BACKUP时执行的脚本，如有需求请自行google。

###       配置选项说明 ###

**global\_defs**

☉notification\_email： keepalived在发生诸如切换操作时需要发送email通知地址，后面的 smtp\_server 相比也都知道是邮件服务器地址。也可以通过其它方式报警，毕竟邮件不是实时通知的。

☉router\_id： 机器标识，通常可设为hostname。故障发生时，邮件通知会用到

**vrrp\_instance**

☉state ： 指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定。如果这里设置为MASTER，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为MASTER

☉interface： 实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的，可以用ifconfig命令查看网卡。

☉mcast\_src\_ip： 发送多播数据包时的源IP地址，这里注意了，这里实际上就是在那个地址上发送VRRP通告，这个非常重要，一定要选择稳定的网卡端口来发送，这里相当于heartbeat的心跳端口，如果没有设置那么就用默认的绑定的网卡的IP，也就是interface指定的IP地址

☉virtual\_router\_id： 这里设置VRID，这里非常重要，相同的VRID为一个组，他将决定多播的MAC地址

☉priority： 设置本节点的优先级，优先级高的为master

☉advert\_int： 检查间隔，默认为1秒。这就是VRRP的定时器，MASTER每隔这样一个时间间隔，就会发送一个advertisement报文以通知组内其他路由器自己工作正常

☉authentication： 定义认证方式和密码，主从必须一样，样例用的是密码方式。

☉virtual\_ipaddress： 这里设置的就是VIP，也就是虚拟IP地址，他随着state的变化而增加删除，当state为master的时候就添加，当state为backup的时候删除，这里主要是有优先级来决定的，和state设置的值没有多大关系。这里可以设置多个虚拟IP地址，类似于一个域名可以解析对应多个IP地址。

☉track\_script： 引用VRRP脚本，即在vrrp\_script 部分指定的名字。每隔vrrp\_script->interval时间运行脚本，如果监控服务有异常则改变优先级，并最终引发主备切换。

**vrrp\_script**

告诉 keepalived 在什么情况下切换，所以尤为重要。可以有多个 vrrp\_script

☉script ：自己写的检测脚本。也可以是一行命令如killall -0 nginx

☉interval4 ： 每4s检测一次，这里要大于监控脚本执行的时间，监控脚本会执行超时，☉keepalived会发送SIGTERM信号结束监控脚本的执行。

☉weight-5 ： 检测失败（脚本返回非0）则优先级 -5

☉fall 2： 检测连续 2 次失败才算确定是真失败。会用weight减少优先级（1-255之间）

☉rise 1： 检测 1 次成功就算成功。但不修改优先级

## 3.4 Nginx配置 ##

Nginx的安装，这里不是本文的重点，请读者自行查阅安装。

keepalived和Nginx是完全独立的，解耦的。 Nginx没有什么可配置的，因为它与keepalived并没有联系。但记住，2台nginx服务器上的配置应该是完全一样的（rsync同步），这样才能做到对用户透明，nginx.conf 里面的server\_name 尽量使用域名来代替，然后dns解析这个域名到虚拟IP 172.30.1.10。

# 4 测试 #

## 4.1启动keepalived  ##

在172.30.1.20和172.30.1.21上将Nginx启动。

启动Nginx：/usr/local/nginx/sbin/nginx

在172.30.1.20和172.30.1.21上将keepalived启动

启动keepalived: servicekeepalived start

启动keepalived的方式还可以是”/etc/init.d/keepalivedstart”, 或者直接执行可执行性文件。

启动keepalived：

service keepalived start

或者

/etc/init.d/keepalived start

或者

/usr/local/keepalived/sbin/keepalived -f /etc/keepalived/keepalived.conf-D

查看进程，正常会有三个进程

\[root@localhost ~\]\# ps -ef | grepkeepalived

root     3870     1  0 14:46 ?        00:00:00 keepalived -D

root     3872  3870  0 14:46 ?        00:00:00 keepalived -D

root     3873  3870  0 14:46 ?        00:00:00 keepalived -D

root     3887 18774  0 14:46 pts/1    00:00:00 grep keepalived

\[root@localhost ~\]\#

用ip命令查看VIP，ifconfig命令不能查看虚拟IP。

\[root@localhost ~\]\# ip a | grep eth1

2: eth1:<BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen1000

inet 172.30.1.20/24 brd 172.30.1.255scope global eth1

inet 172.30.1.10/32scope global eth1

\[root@localhost ~\]\#

可以看到现在是172.30.1.20接管着VIP。

可以用命令tail-f  /var/log/messages查看keepalived的日志

**在172.30.1.20的日志如下：**

Jul 21 15:11:51 localhost Keepalived\[5824\]:Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Jul 21 15:11:51 localhost Keepalived\[5824\]:Unable to resolve default script username 'keepalived\_script' - ignoring

Jul 21 15:11:51 localhost Keepalived\[5824\]:Opening file '/etc/keepalived/keepalived.conf'.

Jul 21 15:11:51 localhost Keepalived\[5825\]:Starting Healthcheck child process, pid=5827

Jul 21 15:11:51 localhost Keepalived\[5825\]:Starting VRRP child process, pid=5828

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: Registering Kernel netlink reflector

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: Registering Kernel netlink command channel

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: Registering gratuitous ARP shared channel

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: Opening file '/etc/keepalived/keepalived.conf'.

Jul 21 15:11:51 localhostKeepalived\_healthcheckers\[5827\]: Opening file'/etc/keepalived/keepalived.conf'.

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: (VI\_1): Cannot start in MASTER state if not addressowner

Jul 21 15:11:51 localhost kernel:ip\_tables: set match: invalid size 32 != 8

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) removing protocol VIPs.

Jul 21 15:11:51 localhost Keepalived\_vrrp\[5828\]:VRRP\_Instance(VI\_1) removing protocol iptable drop rule

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: SECURITY VIOLATION - scripts are being executed butscript\_security not enabled.

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: iptc\_commit returned 0: Invalid argument

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: Using LinkWatch kernel netlink reflector...

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) Entering BACKUP STATE

Jul 21 15:11:51 localhost Keepalived\_vrrp\[5828\]:VRRP sockpool: \[ifindex(2), proto(112), unicast(0), fd(10,11)\]

Jul 21 15:11:51 localhostKeepalived\_vrrp\[5828\]: VRRP\_Script(chk\_nginx) succeeded

Jul 21 15:11:55 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) Transition to MASTER STATE

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1)  EnteringMASTER STATE

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) setting protocol iptable drop rule

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) setting protocol VIPs.

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) Sending/queueing gratuitous ARPs oneth1 for 172.30.1.10

Jul 21 15:11:56 localhostKeepalived\_vrrp\[5828\]: Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:12:01 localhostKeepalived\_vrrp\[5828\]: Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:12:01 localhost Keepalived\_vrrp\[5828\]:VRRP\_Instance(VI\_1) Sending/queueing gratuitous ARPs on eth1 for 172.30.1.10

Jul 21 15:12:01 localhostKeepalived\_vrrp\[5828\]: Sending gratuitous ARP on eth1 for 172.30.1.10

**在172.30.1.21的日志如下：**

Jul 21 15:12:19 localhost Keepalived\[9239\]:Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Jul 21 15:12:19 localhost Keepalived\[9239\]:Unable to resolve default script username 'keepalived\_script' - ignoring

Jul 21 15:12:19 localhost Keepalived\[9239\]:Opening file '/etc/keepalived/keepalived.conf'.

Jul 21 15:12:19 localhost Keepalived\[9240\]:Starting Healthcheck child process, pid=9242

Jul 21 15:12:19 localhost Keepalived\[9240\]:Starting VRRP child process, pid=9243

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: Registering Kernel netlink reflector

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: Registering Kernel netlink command channel

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: Registering gratuitous ARP shared channel

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: Opening file '/etc/keepalived/keepalived.conf'.

Jul 21 15:12:19 localhostKeepalived\_healthcheckers\[9242\]: Opening file'/etc/keepalived/keepalived.conf'.

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) removing protocol VIPs.

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) removing protocol iptable drop rule

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: SECURITY VIOLATION - scripts are being executed butscript\_security not enabled.

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: iptc\_commit returned 0: Invalid argument

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: Using LinkWatch kernel netlink reflector...

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1)  EnteringBACKUP STATE

Jul 21 15:12:19 localhostKeepalived\_vrrp\[9243\]: VRRP sockpool: \[ifindex(2), proto(112), unicast(0),fd(10,11)\]

Jul 21 15:12:20 localhost kernel:ip\_tables: set match: invalid size 32 != 8

Jul 21 15:12:20 localhost Keepalived\_vrrp\[9243\]:VRRP\_Script(chk\_nginx) succeeded

## 4.2 测试VIP漂移 ##

**1. 先看下[http://172.30.1.20/][http_172.30.1.20]，[http://172.30.1.21][http_172.30.1.21]，http://172.30.1.10/是否都能正常访问。**

![20170721165443030][]

172.30.1.20上的Nginx能正常访问

![20170721165834487][]

172.30.1.21上的Nginx能正常访问。

![20170721165939702][]

172.30.1.10（VIP）上的Nginx能正常访问。

**在现有VIP主机上停掉Nginx，测试VIP漂移到BACKUP节点，使得BACKUP成为MASTER。**

为测试方便，先在VIP主机172.30.1.20把/etc/keepalived/check\_nginx.sh，启动Nginx那行注释掉。

#!/bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        #/usr/local/nginx/sbin/nginx
        sleep 2
        counter=$(ps -C nginx --no-heading|wc -l)
        if [ "${counter}" = "0" ]; then
            exit 1
        fi
    fi
    exit 0

然后停掉Nginx: killall nginx

看下172.30.1.20 keepalived的日志：

Jul 21 15:25:09 localhostKeepalived\_vrrp\[5828\]: /etc/keepalived/check\_nginx.sh exited with status 1

Jul 21 15:25:13 localhostKeepalived\_vrrp\[5828\]: /etc/keepalived/check\_nginx.sh exited with status 1

Jul 21 15:25:17 localhostKeepalived\_vrrp\[5828\]: /etc/keepalived/check\_nginx.shexited with status 1

Jul 21 15:25:17 localhostKeepalived\_vrrp\[5828\]: VRRP\_Script(chk\_nginx) failed

Jul 21 15:25:17 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) Changingeffective priority from 100 to 95

Jul 21 15:25:18 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) Receivedadvert with higher priority 99, ours 95

Jul 21 15:25:18 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) EnteringBACKUP STATE

Jul 21 15:25:18 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) removingprotocol VIPs.

Jul 21 15:25:18 localhostKeepalived\_vrrp\[5828\]: VRRP\_Instance(VI\_1) removing protocol iptable drop rule

Jul 21 15:25:21 localhostKeepalived\_vrrp\[5828\]: /etc/keepalived/check\_nginx.sh exited with status 1

先是检测三次（因为vrrp\_scriptchk\_nginx的fall配置的是3）/etc/keepalived/check\_nginx.sh返回码非0，然后VRRP\_Script(chk\_nginx)failed，接着降低优先级from 100 to 95（因为vrrp\_script chk\_nginx的weight -5），接着进入BACKUP STATE，最后移除VIP。

看下172.30.1.21 keepalived的日志：

Jul 21 15:25:42 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) forcing anew MASTER election

Jul 21 15:25:43 localhost Keepalived\_vrrp\[9243\]:VRRP\_Instance(VI\_1) Transition to MASTER STATE

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) EnteringMASTER STATE

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) settingprotocol iptable drop rule

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) settingprotocol VIPs.

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: Sending gratuitous ARP on eth1for 172.30.1.10

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) Sending/queueing gratuitous ARPs oneth1 for 172.30.1.10

Jul 21 15:25:44 localhostKeepalived\_vrrp\[9243\]: Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:25:49 localhostKeepalived\_vrrp\[9243\]: Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:25:49 localhostKeepalived\_vrrp\[9243\]: VRRP\_Instance(VI\_1) Sending/queueing gratuitous ARPs oneth1 for 172.30.1.10

Jul 21 15:25:49 localhost Keepalived\_vrrp\[9243\]:Sending gratuitous ARP on eth1 for 172.30.1.10

Jul 21 15:25:49 localhostKeepalived\_vrrp\[9243\]: Sending gratuitous ARP on eth1 for 172.30.1.10

可以看到172.30.1.21的接管VIP的流程: forcing a new MASTER election à Transitionto MASTER STATE à Entering MASTER STATE à settingprotocol iptable drop rule à setting protocol VIPs à Sendinggratuitous ARP on eth1 for 172.30.1.10

最后一步Sendinggratuitous ARP on eth1 for 172.30.1.10是广播免费ARP报文，通知对方更新ARP缓存。

看下172.30.1.21的IP信息

\[root@localhost ~\]\# ip a | grep eth1

2: eth1: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdiscpfifo\_fast state UP qlen 1000

inet 172.30.1.21/24 brd 172.30.1.255 scope global eth1

inet 172.30.1.10/32 scope global eth1

\[root@localhost ~\]\#

可以看到VIP 172.30.1.10已经从172.30.1.20漂移到172.30.1.21上了。

继续测试下[http://172.30.1.20/][http_172.30.1.20]，[http://172.30.1.21][http_172.30.1.21]，http://172.30.1.10/是否都能正常访问。（记得清除浏览器缓存，或者用curl命令测试）。

\[root@localhost ~\]\# curl http://172.30.1.20/

curl: (7) couldn't connect to host

\[root@localhost ~\]\# curl http://172.30.1.21

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body \{

width: 35em;

margin: 0 auto;

font-family: Tahoma,Verdana, Arial, sans-serif;

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfullyinstalled and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<ahref="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<ahref="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

\[root@localhost ~\]\# curl http://172.30.1.10

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body \{

width: 35em;

margin: 0 auto;

font-family: Tahoma,Verdana, Arial, sans-serif;

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfullyinstalled and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<ahref="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<ahref="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

\[root@localhost ~\]\#

可以看到172.30.1.20的Nginx已经不能访问，但是172.30.1.21的Nginx正常工作，所以VIP 172.30.1.10也能正常访问Nginx.

**测试过程遇到的疑问：**

若nginx挂掉，在监控脚本里执行/etc/init.d/keepalived stop 和在命令行直接执行/etc/init.d/keepalived stop，效果不一样。在监控脚本keepalived进程不会退出，在命令行直接keepalived进程退出了。在监控脚本keepalived进程不会退出，但优先级会降低，也会让出VIP。

tail -f /var/log/messages

Jul 20 18:21:50 localhostKeepalived\_vrrp\[29328\]: /etc/keepalived/check\_nginx.sh exited due to signal 15

Jul 20 18:21:52 localhostKeepalived\_vrrp\[29328\]: /etc/keepalived/check\_nginx.sh exited due to signal 15

Jul 20 18:21:54 localhostKeepalived\_vrrp\[29328\]: /etc/keepalived/check\_nginx.sh exited due to signal 15

查看keepalived版本的ChangeLog，有这么一段：

\* Allow tracking and misc\_check scriptstime to terminate after

timeout. If a script exceeds the timeout, itis sent a SIGTERM,

and then if it still doesn't terminate, it issent a SIGKILL.

The problem was that the script was onlyallowed 2 microseconds

to terminate, whereas it should have been 2seconds.

**原因：**在check\_nginx.sh里，若检测到nginx退出了，在“/etc/init.d/keepalivedstop”之前会“sleep 2”睡眠2秒，而在keepalived.conf中，vrrp\_script chk\_nginx配置模块中的interval也是2秒，所以keepalived判断check\_nginx.sh脚本执行超时，就发送SIGTERM把脚本咔擦掉了，所以“/etc/init.d/keepalived stop”就没有执行到。

# 5 keepalived配置双VIP #

可以将keepalived.conf-->vrrp\_instance-->virtual\_ipaddress模块配置两个或者多个VIP。例如，现在为上文Nginx新增一个VIP 172.30.1.11，使得Nginx拥有两个VIP。

1.      修改keepalived.conf:

……
    virtual_ipaddress {
        172.30.1.10
        172.30.1.11
    }
    ……

2.      重启keepalived

service keepalived restart

3． 测试新VIP是否能正常访问Nginx

在浏览器直接输入172.30.1.11

![SouthEast][]

可以看到在新的VIP上能正常访问Nginx服务。当然旧的VIP 172.30.1.10仍然也能正常访问Nginx服务。

若backup接管VIP，这两个或多个VIP都会一起被接管。

# 参考 #

官网

[http://www.keepalived.org/][http_www.keepalived.org]

[https://github.com/acassen/keepalived][https_github.com_acassen_keepalived]

博客

[http://www.voidcn.com/blog/xuzhouweihao/article/p-1728940.html][http_www.voidcn.com_blog_xuzhouweihao_article_p-1728940.html]

[http://seanlook.com/2015/05/18/nginx-keepalived-ha/][http_seanlook.com_2015_05_18_nginx-keepalived-ha]

![Image 1][]

[Image 1]: 
[Link 1]: http://en.wikipedia.org/wiki/VRRP
[20170721164917450]: /images/20220612/44cf8169cbee401499c05e125d7e8e44.png
[http_172.30.1.20]: http://172.30.1.20/
[http_172.30.1.21]: http://172.30.1.21/
[20170721165443030]: /images/20220612/0691c0c6526d4ae5a826cbb4fbc31b0c.png
[20170721165834487]: /images/20220612/dc3f7edf6ba743f490294e1ceddd836d.png
[20170721165939702]: /images/20220612/aed46927fb54443ca621a52c5a3e3a73.png
[SouthEast]: /images/20220612/1e912dab84de4961898f8739fb50b16f.png
[http_www.keepalived.org]: http://www.keepalived.org/
[https_github.com_acassen_keepalived]: https://github.com/acassen/keepalived
[http_www.voidcn.com_blog_xuzhouweihao_article_p-1728940.html]: http://www.voidcn.com/blog/xuzhouweihao/article/p-1728940.html
[http_seanlook.com_2015_05_18_nginx-keepalived-ha]: http://seanlook.com/2015/05/18/nginx-keepalived-ha/