Keepalived实现LVS（DR模式）+自动后端健康检测

傷城~ 2022-05-18 01:09 714阅读 0赞

上一篇博客《[Keepalived实现NAT+LVS+后端健康检测][Keepalived_NAT_LVS]》简单介绍了NAT模式的LVS搭建步骤，下面介绍一下使用更普遍、网络结构更简单的DR模式搭建方法。

不同与NAT模式，DR模式不需要跨两个网段，负载均衡服务器和真实Web服务器还有客户端都可以放在同一个网段里，对网络的复杂性要求降低，而且有一个好处是负载量会提高，因为数据流量不需要通过负载均衡器进行路由，使负载均衡器不会产生瓶颈。但是也有缺点，一是真实Web服务器需要修改自己的lo网卡，这样需要对真实web服务器进行一些网络配置，另一个是无法实现NAT中的端口映射功能，比如负载均衡器开放的是80端口，那么后端的真实Web服务器也必须开放80端口，而在NAT模式下，对后端服务器的端口没有限制。

首先介绍一下网络分配

<table> 
 <tbody> 
  <tr> 
   <td>服务器</td> 
   <td>真实地址</td> 
   <td>虚拟地址</td> 
   <td>网关地址</td> 
  </tr> 
  <tr> 
   <td>Keepalived主服务器</td> 
   <td>10.100.100.89</td> 
   <td>10.100.100.200</td> 
   <td>10.100.100.1</td> 
  </tr> 
  <tr> 
   <td>Keepalived备服务器</td> 
   <td>10.100.100.88</td> 
   <td>10.100.100.200</td> 
   <td>10.100.100.1</td> 
  </tr> 
  <tr> 
   <td>真实HTTP服务1</td> 
   <td>10.100.100.101</td> 
   <td>10.100.100.200</td> 
   <td>10.100.100.1</td> 
  </tr> 
  <tr> 
   <td>真实HTTP服务2</td> 
   <td>10.100.100.102</td> 
   <td>10.100.100.200</td> 
   <td>10.100.100.1</td> 
  </tr> 
 </tbody> 
</table>

在这个网络中，所有网络流量都通过网关10.100.100.1进行进出，客户端访问地址为虚拟地址10.100.100.200，客户端的请求到达keepalived后，会修改数据包的MAC地址然后发送给两台真实HTTP服务器，真实HTTP由于也配有虚拟地址，所以会正常接收这个数据包，然后返回流量通过网关直接返回给用户而非流经负载均衡器再走网关，所以负载均衡器的压力相对NAT模式大大减小。负载均衡器和HTTP服务器均只开放8080端口。因为DR模式下开放的端口必须一致。

在真实Web服务器上因为要设置lo网卡上的虚拟ip，并关闭arp响应（如果不关闭，那么所有使用vip的服务器就会造成IP冲突或者说是ARP欺骗），方法很简单，直接写一个脚本即可，并且还可以开机自启

vi  /etc/init.d/realserver

SNS_VIP=10.100.100.200
    #/etc/rc.d/init.d/functions
    case "$1" in
    start)
           ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
           /sbin/route add -host $SNS_VIP dev lo:0
           echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
           echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
           echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
           echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
           sysctl -p >/dev/null 2>&1
           echo "RealServer Start OK"
           ;;
    stop)
           ifconfig lo:0 down
           route del $SNS_VIP >/dev/null 2>&1
           echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
           echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
           echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
           echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
           echo "RealServer Stoped"
           ;;
    *)
           echo "Usage: $0 {start|stop}"
           exit 1
    esac
    exit 0

增加可执行权限

chmod 755 /etc/init.d/realserver

然后启动很方便，因为已经注册为了linux服务

service realserver start

或者

/etc/init.d/realserver start

下面是keepalived主机的配置文件

! Configuration File for keepalived
    
    global_defs {
       router_id LVS_MASTER
       vrrp_skip_check_adv_addr
       #vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface ens192
        virtual_router_id 91
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            10.100.100.200
        }
    }
    
    virtual_server 10.100.100.200 8080 {
        delay_loop 15
        lb_algo rr
        lb_kind DR
        persistence_timeout 50
    	nat_mask 255.255.255.0
        protocol TCP
    
        real_server 10.100.100.101 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest d7f986677d9f563bd1794b09d82206a3
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    	
    	real_server 10.100.100.102 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest 3dc49f3f365e728446c8f8df2475464d
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    }

启动命令

service keepalived start

keepalived主的启动日志如下 /var/log/message

Jul 27 22:26:27 lvsnat1 systemd: Starting LVS and VRRP High Availability Monitor...
    Jul 27 22:26:27 lvsnat1 Keepalived[7331]: Starting Keepalived v2.0.6 (07/23,2018)
    Jul 27 22:26:27 lvsnat1 Keepalived[7331]: Running on Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 (built for Linux 3.10.0)
    Jul 27 22:26:27 lvsnat1 Keepalived[7331]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 27 22:26:27 lvsnat1 Keepalived[7332]: Starting Healthcheck child process, pid=7333
    Jul 27 22:26:27 lvsnat1 systemd: Started LVS and VRRP High Availability Monitor.
    Jul 27 22:26:27 lvsnat1 Keepalived[7332]: Starting VRRP child process, pid=7334
    Jul 27 22:26:27 lvsnat1 Keepalived_healthcheckers[7333]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 27 22:26:27 lvsnat1 Keepalived_healthcheckers[7333]: Unknown keyword 'nat_mask'
    Jul 27 22:26:27 lvsnat1 Keepalived_healthcheckers[7333]: Gained quorum 1+0=1 <= 2 for VS [10.100.100.200]:tcp:8080
    Jul 27 22:26:27 lvsnat1 Keepalived_healthcheckers[7333]: Activating healthchecker for service [10.100.100.101]:tcp:8080 for VS [10.100.100.200]:tcp:8080
    Jul 27 22:26:27 lvsnat1 Keepalived_healthcheckers[7333]: Activating healthchecker for service [10.100.100.102]:tcp:8080 for VS [10.100.100.200]:tcp:8080
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Registering Kernel netlink reflector
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Registering Kernel netlink command channel
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Assigned address 10.100.100.89 for interface ens192
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Assigned address fe80::7602:aaf3:d259:a375 for interface ens192
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: Registering gratuitous ARP shared channel
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: (VI_1) removing VIPs.
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: (VI_1) Entering BACKUP STATE (init)
    Jul 27 22:26:27 lvsnat1 Keepalived_vrrp[7334]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(8,9)]
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: (VI_1) Receive advertisement timeout
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: (VI_1) Entering MASTER STATE
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: (VI_1) setting VIPs.
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.100.100.200
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:31 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 27 22:26:36 lvsnat1 Keepalived_vrrp[7334]: Sending gratuitous ARP on ens192 for 10.100.100.200

由于添加了页面MD5值来判断服务的可用性，于是我们修改页面，模拟http服务页面错误，观察keepalived的日志

Jul 27 22:28:34 lvsnat1 Keepalived_healthcheckers[7333]: MD5 digest error to server [10.100.100.102]:tcp:8080.
    Jul 27 22:28:37 lvsnat1 Keepalived_healthcheckers[7333]: MD5 digest error to server [10.100.100.102]:tcp:8080.
    Jul 27 22:28:40 lvsnat1 Keepalived_healthcheckers[7333]: MD5 digest error to server [10.100.100.102]:tcp:8080.
    Jul 27 22:28:43 lvsnat1 Keepalived_healthcheckers[7333]: MD5 digest error to server [10.100.100.102]:tcp:8080.
    Jul 27 22:28:43 lvsnat1 Keepalived_healthcheckers[7333]: Check on service [10.100.100.102]:tcp:8080 failed.
    Jul 27 22:28:43 lvsnat1 Keepalived_healthcheckers[7333]: Removing service [10.100.100.102]:tcp:8080 to VS [10.100.100.200]:tcp:8080

可以看到在多次检查MD5值错误之后，keepalived将真实服务器移出列表，再次之后就访问不到这个服务器上的HTTP服务了。但是在这之前是可以访问到错误页面的。

然后我们再把真实服务器的页面内容改回来，模拟服务恢复正常，观察keepalived的日志

ul 27 23:55:16 lvsnat1 Keepalived_healthcheckers[7333]: MD5 digest success to [10.100.100.102]:tcp:8080 url(1).
    Jul 27 23:55:16 lvsnat1 Keepalived_healthcheckers[7333]: Remote Web server [10.100.100.102]:tcp:8080 succeed on service.
    Jul 27 23:55:16 lvsnat1 Keepalived_healthcheckers[7333]: Adding service [10.100.100.102]:tcp:8080 to VS [10.100.100.200]:tcp:8080

然后又可以正常的负载均衡了。

下面再看一下备机的配置文件写法

! Configuration File for keepalived
     
    global_defs {
       router_id LVS_BACKUP            #BACKUP上修改为LVS_BACKUP
    }
     
    vrrp_instance VI_1 {
        state BACKUP                   #BACKUP上修改为BACKUP
        interface ens192               #此处是要绑定虚拟IP的网卡名称
        virtual_router_id 91           #此处要与master一致
        priority 80                    #BACKUP上修改为80
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            10.100.100.200             #此处要与master一致
        }
    }
    
     
    virtual_server 10.100.100.200 8080 {
        delay_loop 15
        lb_algo rr
        lb_kind DR
        persistence_timeout 50
        protocol TCP
     
        real_server 10.100.100.101 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest d7f986677d9f563bd1794b09d82206a3
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    	
    	real_server 10.100.100.102 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest 3dc49f3f365e728446c8f8df2475464d
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }

启动日志

Jul 28 00:32:15 lvsdr1 systemd: Starting LVS and VRRP High Availability Monitor...
    Jul 28 00:32:15 lvsdr1 Keepalived[1811]: Starting Keepalived v2.0.6 (07/23,2018)
    Jul 28 00:32:15 lvsdr1 Keepalived[1811]: Running on Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 (built for Linux 3.10.0)
    Jul 28 00:32:15 lvsdr1 Keepalived[1811]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 28 00:32:15 lvsdr1 Keepalived[1811]: WARNING - 1 missing '}'(s) in the config file(s)
    Jul 28 00:32:15 lvsdr1 Keepalived[1812]: Starting Healthcheck child process, pid=1813
    Jul 28 00:32:15 lvsdr1 systemd: Started LVS and VRRP High Availability Monitor.
    Jul 28 00:32:15 lvsdr1 Keepalived[1812]: Starting VRRP child process, pid=1814
    Jul 28 00:32:15 lvsdr1 Keepalived_healthcheckers[1813]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 28 00:32:15 lvsdr1 Keepalived_healthcheckers[1813]: Gained quorum 1+0=1 <= 2 for VS [10.100.100.200]:tcp:8080
    Jul 28 00:32:15 lvsdr1 Keepalived_healthcheckers[1813]: Activating healthchecker for service [10.100.100.101]:tcp:8080 for VS [10.100.100.200]:tcp:8080
    Jul 28 00:32:15 lvsdr1 Keepalived_healthcheckers[1813]: Activating healthchecker for service [10.100.100.102]:tcp:8080 for VS [10.100.100.200]:tcp:8080
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Registering Kernel netlink reflector
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Registering Kernel netlink command channel
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: WARNING - 1 missing '}'(s) in the config file(s)
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Assigned address 10.100.100.88 for interface ens192
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Assigned address fe80::7602:aaf3:d259:a375 for interface ens192
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: Registering gratuitous ARP shared channel
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: (VI_1) removing VIPs.
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Entering BACKUP STATE (init)
    Jul 28 00:32:15 lvsdr1 Keepalived_vrrp[1814]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(8,9)]

可以看到，没有广播VIP的ARP，并且没有绑定VIP的地址，如下

[root@lvsdr2 ~]# ip addr
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:be:8d:18 brd ff:ff:ff:ff:ff:ff
        inet 10.100.100.88/24 brd 10.100.100.255 scope global noprefixroute ens192
           valid_lft forever preferred_lft forever
        inet6 fe80::7602:aaf3:d259:a375/64 scope link noprefixroute 
           valid_lft forever preferred_lft forever

然后我们此时停掉主keepalived，看备服务器能否顺利接管VIP，日志如下

Jul 28 00:32:36 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Backup received priority 0 advertisement
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Receive advertisement timeout
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Entering MASTER STATE
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: (VI_1) setting VIPs.
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.100.100.200
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:37 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200
    Jul 28 00:32:42 lvsdr1 Keepalived_vrrp[1814]: Sending gratuitous ARP on ens192 for 10.100.100.200

可以看到，备机已经侦测到主机挂掉，并顺利接管了VIP

[root@lvsdr2 ~]# ip addr
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:be:8d:18 brd ff:ff:ff:ff:ff:ff
        inet 10.100.100.88/24 brd 10.100.100.255 scope global noprefixroute ens192
           valid_lft forever preferred_lft forever
        inet 10.100.100.200/32 scope global ens192
           valid_lft forever preferred_lft forever
        inet6 fe80::7602:aaf3:d259:a375/64 scope link noprefixroute 
           valid_lft forever preferred_lft forever

然后我们启动keepalived主服务器，观察备服务器能否主动将VIP让出

Jul 28 00:36:30 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Master received advert from 10.100.100.89 with higher priority 100, ours 80
    Jul 28 00:36:30 lvsdr1 Keepalived_vrrp[1814]: (VI_1) Entering BACKUP STATE
    Jul 28 00:36:30 lvsdr1 Keepalived_vrrp[1814]: (VI_1) removing VIPs.

可以看到，VIP已经成功从备服务器上摘除

另外，Keepalived非常的只能，在检测到real server故障之后不仅可以自动切换，自动维护LVS列表，而且可以通过调用脚本的方式执行命令，实现报警和日志记录的功能，方法很简单。

对于real server的切换，我们在/etc/keepalived目录下建立如下四个文件

web1down.sh  web1up.sh  web2down.sh  web2up.sh

内容类似下面的web1down.sh：

#!/bin/bash
    echo "`date` web1 down" >> /tmp/lvs.log

这样就实现了一个切换日志记录的功能，你也可以在脚本中加入其他的报警命令。

然后在keepalived.conf中的后端server部分加入两行，分别是notify\_up和notify\_down，如下

virtual_server 10.100.100.200 8080 {
        delay_loop 15
        lb_algo rr
        lb_kind DR
        persistence_timeout 50
        protocol TCP
     
        real_server 10.100.100.101 8080 {
            weight 1
    		notify_up /etc/keepalived/web1up.sh
            notify_down /etc/keepalived/web1down.sh
            HTTP_GET {
                url {
                  path /hello
                  digest d7f986677d9f563bd1794b09d82206a3
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    	
    	real_server 10.100.100.102 8080 {
            weight 1
    		notify_up /etc/keepalived/web2up.sh
            notify_down /etc/keepalived/web2down.sh
            HTTP_GET {
                url {
                  path /hello
                  digest 3dc49f3f365e728446c8f8df2475464d
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }

就实现了real server切换的报警和日志记录功能。同样可以对keepalived主机备机切换记录日志和报警，具体脚本写法请参考另一篇博客《[Haproxy加持KeepAlived+进程自动检测][Haproxy_KeepAlived]》

[Keepalived_NAT_LVS]: https://blog.csdn.net/lvshaorong/article/details/81205451
[Haproxy_KeepAlived]: https://blog.csdn.net/lvshaorong/article/details/81301976