基于云端虚拟机的LVS/DR+Keepalived+nginx的高可用集群架构配置（更新nginx代理功能）...

约定不等于承诺〃 2022-02-05 06:11 166阅读 0赞

基于云端虚拟机的LVS/DR+Keepalived+nginx的高可用集群架构配置

最近，公司要我部署一个集群架构，选来选取还是选择了大家都很熟悉的基于DR的LVS+Keepalived做负载分发，然后使用轻量级的nginx做中间代理层，这里呢先暂时实现一个简单web应用功能，对于代理功能和后续的web层部署、数据层安排将择机更新！

首先看一下集群框架：[![wKiom1R9pDCBdH6DAAFs7SXE3aQ326.jpg][]][wKiom1R9pDCBdH6DAAFs7SXE3aQ326.jpg]  下面开始我们的集群搭建行程：

一、IP规划：

做一件事情需要想想怎么去做既简单又有条理，我们做项目也应该如此。所以这里一定先做一个简单的规划，对于你后续测试的时候出现不能分发的情况有所解决。

负载均衡层 代理层 WEB层（待定）

LVS1：10.124.20.248 nginx1:10.124.20.250 web1 10.124.20.240

web2 10.124.20.241

nginx2:10.124.20.251 web3 10.124.20.242

web4 10.124.20.243

LVS2：10.124.20.249 nginx3:10.124.20.252 web5 10.124.20.244

VIP：10.124.20.20 对应绑定的外网IP：117.78.3.253 web6 10.124.20.245

备注：在云端虚拟机做架构，比在自己本地虚拟机做实验，要略有不同。初次接触云端虚拟机的同学可能有点犯迷糊，不知道这个ip地址是怎么分配的，怎么去管理。我的老师教给一个最能解决问题的办法：在任何服务之前，首先要了解整个网络环境，然后根据环境部署服务，更改配置、规则等。关于云端虚拟机，它在整个硬件防火墙和整个路由器封闭起来的一个大的局域网集群（这里为了好理解我们姑且认为云端就一个防火墙和一个路由器）。在这个集群中，又有很多网段的局域网，每个企业在申请虚拟机的时候一般会建立一个虚拟机组，这个机组则对应着一个网段。

这里使用的网段是10.124.20.0/24，给各台服务器分配唯一的IP地址。这里要特别说明一下，华为云技术人员建议使用的服务器固定ip在32-254之间，在2-31之间的IP地址作为候补之用，比如说VIP就选用这里面，还有临时加虚拟机的话也可以选用。所以对于IP地址的规划是很重要的。另外说一下，这也是在生产环境中的经验吧，VIP必须是和集群IP地址在一个网段，还要必须有别于集群IP地址，另外要提供网站访问服务或者测试，则需要将内网的VIP与一个公网IP进行绑定，在外部网络浏览器访问外网IP就会直接映射到内网VIP上，然后就会实现我们所熟悉的功能，VIP漂移，LVS分发。小技巧：公网IP与单纯的一个内网ip做绑定是不成功的，所以先把需要绑定公网IP的内网IP绑定到一个虚拟机上，然后再进行绑定，绑定完之后再进行更改IP地址。（比如说我要使用10.124.20.20做VIP，首先我把10.124.20.20绑定到一台虚拟机上，通过后台将外网IP117.78.3.253与10.124.20.20绑定，也就是做映射，最后再将10.124.20.20与虚拟解绑，这样就实现了预期目的）

二、源码安装&&配置

1.LVS+Keepalived搭建

1）LVS1：10.124.20.248&&LVS2：10.124.20.249

cd /usr/local/src/

wget [http://down1.chinaunix.net/distfiles/ipvsadm-1.24.tar.gz][http_down1.chinaunix.net_distfiles_ipvsadm-1.24.tar.gz]

wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz

tar xf ipvsadm-1.24.tar.gz

tar xf keepalived-1.1.20.tar.gz

ipvsadm编译安装

ln -s /usr/src/kernels/2.6.32-279.el6.x86\_64/ /usr/src/linux \#创建软连接

cd /usr/local/src/ipvsadm-1.24

make && make install

\#/sbin/ipvsadm -v \#检测ipvsadm是否安装成功(或者：\#watch ipvsadm –ln)

ipvsadm v1.24 2005/12/10 (compiled with getopt\_long and IPVS v1.2.1)

Keepalived编译安装

/usr/local/src/keepalived-1.1.20

./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-279.el6.x86\_64

返回信息：

Keepalived configuration

\------------------------

Keepalived version : 1.1.20

Compiler : gcc

Compiler flags : -g -O2

Extra Lib : -lpopt -lssl -lcrypto

Use IPVS Framework : Yes

IPVS sync daemon support : Yes

Use VRRP Framework : Yes

Use Debug flags : No

make && make install

返回信息：

install -d /usr/local/keepalived/bin

install -m 755 ../bin/genhash /usr/local/keepalived/bin/

install -d /usr/local/keepalived/share/man/man1

install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1

将Keepalived做成服务：

cd /usr/local/keepalived/

cp etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp etc/keepalived/keepalived.conf /etc/keepalived/

cp sbin/keepalived /usr/sbin/

将keepalived设置成开机自启动服务

chkconfig --add keepalived

chkconfig keepalived on

2） 修改Keepalived.conf配置文件

\#cat /etc/keepalived/keepalived.conf \#主LVS1：10.124.20.248

! Configuration File for keepalived

global\_defs \{

notification\_email \{

qingbo.song@apicloud.com

notification\_email\_from qingbo.song@apicloud.com

smtp\_server 127.0.0.1

\# smtp\_connect\_timeout 30

router\_id LVS\_DEVEL

vrrp\_sync\_group VGM \{

group \{

VI\_1

notify\_master "/opt/shell/arp.sh"

notify\_backup "/opt/shell/arp.sh"

vrrp\_script chk\_nginx \{

script "/opt/shell/check\_nginx.sh"

interval 1

weight -2

vrrp\_script chk\_nginx2 \{

script "/opt/shell/check\_nginx2.sh"

interval 1

weight -2

vrrp\_script chk\_nginx3 \{

script "/opt/shell/check\_nginx3.sh"

interval 1

weight -2

vrrp\_instance VI\_1 \{

state MASTER

interface eth0

virtual\_router\_id 51

priority 100

advert\_int 1

authentication \{

auth\_type PASS

auth\_pass yzkjTest

track\_script \{

chk\_nginx

chk\_nginx2

chk\_nginx3

virtual\_ipaddress \{

10.124.20.20

virtual\_server 10.124.20.20 80 \{

delay\_loop 5

lb\_algo sh

lb\_kind DR

persistence\_timeout 60

protocol TCP

real\_server 10.124.20.250 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

real\_server 10.124.20.251 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

real\_server 10.124.20.252 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

\#cat /etc/keepalived/keepalived.conf \#备LVS1：10.124.20.249

! Configuration File for keepalived

global\_defs \{

notification\_email \{

qingbo.song@apicloud.com

notification\_email\_from qingbo.song@apicloud.com

smtp\_server 127.0.0.1

\# smtp\_connect\_timeout 30

router\_id LVS\_DEVEL

vrrp\_sync\_group VGM \{

group \{

VI\_1

notify\_master "/opt/shell/arp.sh"

notify\_backup "/opt/shell/arp.sh"

vrrp\_script chk\_nginx \{

script "/opt/shell/check\_nginx.sh"

interval 1

weight -2

vrrp\_script chk\_nginx2 \{

script "/opt/shell/check\_nginx2.sh"

interval 1

weight -2

vrrp\_script chk\_nginx3 \{

script "/opt/shell/check\_nginx3.sh"

interval 1

weight -2

vrrp\_instance VI\_1 \{

state BACKUP

interface eth0

virtual\_router\_id 51

priority 99

advert\_int 1

authentication \{

auth\_type PASS

auth\_pass yzkjTest

track\_script \{

chk\_nginx

chk\_nginx2

chk\_nginx3

virtual\_ipaddress \{

10.124.20.20

virtual\_server 10.124.20.20 80 \{

delay\_loop 5

lb\_algo sh

lb\_kind DR

persistence\_timeout 60

protocol TCP

real\_server 10.124.20.250 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

real\_server 10.124.20.251 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

real\_server 10.124.20.252 80 \{

weight 1

TCP\_CHECK \{

connect\_timeout 10

nb\_get\_retry 3

delay\_before\_retry 3

connect\_port 80

==============================================================

相关检测脚本：

\#cat /opt/shell/arp.sh

\#!/bin/bash

VIP=10.124.20.20

GATEWAY=10.124.20.1

/sbin/arping -I eth0 -c 5 -s $VIP $GATEWAY &>/dev/null

\#cat /opt/shell/check\_nginx.sh

!/bin/bash

url="http://10.124.20.250"

status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ \{print $2\}')

if \[ "$status" != "200" \]; then

/opt/shell/arp.sh start

\#cat /opt/shell/check\_nginx2.sh

!/bin/bash

url="http://10.124.20.251"

status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ \{print $2\}')

if \[ "$status" != "200" \]; then

/opt/shell/arp.sh start

\#cat /opt/shell/check\_nginx3.sh

!/bin/bash

url="http://10.124.20.252"

status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ \{print $2\}')

if \[ "$status" != "200" \]; then

/opt/shell/arp.sh start

===============================================================

3）Keepalived服务启动（ifconfig eth0:1 10.124.20.248 netmask 255.255.255.255 up）

service keepalived start \#服务启动

\#ps -aux|grep keepalived \#检测服务启动状态

（/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/keepalived.conf -D 显示在日志记录 -f 指定配置文件目录）

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ

root 21205 0.0 0.0 36776 732 ? Ss 17:06 0:00 keepalived -D

root 21207 0.0 0.1 38880 1936 ? S 17:06 0:00 keepalived -D

root 21208 0.0 0.1 38880 1232 ? S 17:06 0:00 keepalived -D

root 21301 0.0 0.0 103240 840 pts/0 S+ 17:07 0:00 grep keepalived

4）\#设置开机启动服务

（\#\#echo “/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/keepalived.conf” >> /etc/rc.d/rc.local）

chkconfig --add keepalived

chkconfig keepalived on

service iptables stop

5）添加防火墙规则：

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT \#使80端口提供对外服务

iptables -A INPUT -d 224.0.0.0/8 -j ACCEPT

iptables -A INPUT -p vrrp -j ACCEPT \#基于DR模式，当用户发出请求后，只有DR响应ARP广播包，允许vrrp虚拟路由器冗余协议

/etc/rc.d/init.d/iptables save \#保存规则到iptables文件，使重启后生效）

6）开启转发功能；

\#vim /etc/sysctl.conf

net.ipv4.ip\_forward = 1

\#sysctl -p \#使转发能够生效

三、nginx服务器的搭建

1）安装pcre-8.36.tar.gz

cd /usr/local/src

tar xf pcre-8.36.tar.gz

cd pcre-8.36

./configure --prefix=/usr/local/pcre

make && make install

2）安装nginx-1.5.1.tar.gz

cd /usr/local/src

tar xf nginx-1.5.1.tar.gz

cd nginx-1.5.1

./configure --prefix=/usr/local/nginx --with-http\_ssl\_module --with-http\_stub\_status\_module --with-pcre=/usr/local/src/pcre-8.36 --with-openssl=/usr/local/src/openssl-1.0.0c //这里跟的目录是源码目录，而不是源码安装后的目录

make && make install

3）启动

编写启动脚本:

===============================================================

vim /etc/init.d/nginx | chmod +x /etc/init.d/nginx

\#!/bin/sh

\# nginx - this script starts and stops the nginx daemin

\# chkconfig: - 85 15

\# description: Nginx is an HTTP(S) server, HTTP(S) reverse \\

\# proxy and IMAP/POP3 proxy server

\# processname: nginx

\# config: /usr/local/nginx/conf/nginx.conf

\# pidfile: /usr/local/nginx/logs/nginx.pid

\# Source function library.

. /etc/rc.d/init.d/functions

\# Source networking configuration.

. /etc/sysconfig/network

\# Check that networking is up.

\[ "$NETWORKING" = "no" \] && exit 0

nginx="/usr/local/nginx/sbin/nginx"

prog=$(basename $nginx)

NGINX\_CONF\_FILE="/usr/local/nginx/conf/nginx.conf"

lockfile=/var/lock/subsys/nginx

start() \{

\[ -x $nginx \] || exit 5

\[ -f $NGINX\_CONF\_FILE \] || exit 6

echo -n $"Starting $prog: "

daemon $nginx -c $NGINX\_CONF\_FILE

retval=$?

echo

\[ $retval -eq 0 \] && touch $lockfile

return $retval

stop() \{

echo -n $"Stopping $prog: "

killproc $prog -QUIT

retval=$?

echo

\[ $retval -eq 0 \] && rm -f $lockfile

return $retval

restart() \{

configtest || return $?

stop

start

reload() \{

configtest || return $?

echo -n $"Reloading $prog: "

killproc $nginx -HUP

RETVAL=$?

echo

force\_reload() \{

restart

configtest() \{

$nginx -t -c $NGINX\_CONF\_FILE

rh\_status() \{

status $prog

rh\_status\_q() \{

rh\_status >/dev/null 2>&1

case "$1" in

start)

rh\_status\_q && exit 0

;;

stop)

rh\_status\_q || exit 0

;;

restart|configtest)

;;

reload)

rh\_status\_q || exit 7

;;

force-reload)

force\_reload

;;

status)

rh\_status

;;

condrestart|try-restart)

rh\_status\_q || exit 0

;;

\*)

echo $"Usage: $0 \{start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest\}"

exit 2

esac

======================================================================

service nginx start 启动服务

4）RS配置脚本

======================================================================

\#cat /opt/RS.sh

\#!/bin/bash

\# Script to start LVS DR real server.

\# description: LVS DR real server

. /etc/rc.d/init.d/functions

VIP=10.124.20.20

host=\`/bin/hostname\`

case "$1" in

start)

\# Start LVS-DR real server on this machine.

/sbin/ifconfig lo down

/sbin/ifconfig lo up

echo 1 > /proc/sys/net/ipv4/conf/lo/arp\_ignore

echo 2 > /proc/sys/net/ipv4/conf/lo/arp\_announce

echo 1 > /proc/sys/net/ipv4/conf/all/arp\_ignore

echo 2 > /proc/sys/net/ipv4/conf/all/arp\_announce

/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up

/sbin/route add -host $VIP dev lo:0

echo "RealServer Start OK"

;;

stop)

\# Stop LVS-DR real server loopback device(s).

/sbin/ifconfig lo:0 down

echo 0 > /proc/sys/net/ipv4/conf/lo/arp\_ignore

echo 0 > /proc/sys/net/ipv4/conf/lo/arp\_announce

echo 0 > /proc/sys/net/ipv4/conf/all/arp\_ignore

echo 0 > /proc/sys/net/ipv4/conf/all/arp\_announce

echo "RealServer Stoped OK"

;;

status)

\# Status of LVS-DR real server.

islothere=\`/sbin/ifconfig lo:0 | grep $VIP\`

isrothere=\`netstat -rn | grep "lo:0" | grep $VIP\`

if \[ ! "$islothere" -o ! "isrothere" \];then

\# Either the route or the lo:0 device

\# not found.

echo "LVS-DR real server Stopped."

else

echo "LVS-DR real server Running."

;;

\*)

\# Invalid entry.

echo "$0: Usage: $0 \{start|status|stop\}"

exit 1

;;

esac

===================================================================

启动RS配置：

sh /opt/RS.sh start

5)配置ngin.conf文件，开启反向代理功能（三台代理服务器，需要修改的就是需要web服务器的IP地址）

\#nginx1：10.124.20.250

vim /usr/local/nginx/conf/nginx.conf

===================================================================

worker\_processes 1;

events \{

worker\_connections 1024;

http \{

include mime.types;

default\_type application/octet-stream;

sendfile on;

keepalive\_timeout 65;

upstream YZKJ1 \{ \#定义一个代理服务群组

ip\_hash;

server 10.124.20.240 weight=1;

server 10.124.20.241 weight=1;

server \{

listen 80;

server\_name localhost;

location / \{

proxy\_pass http://YZKJ1; \#开启代理服务，直接执行代理服务群组

error\_page 500 502 503 504 /50x.html;

location = /50x.html \{

root html;

==================================================================

\#nginx1：10.124.20.251

===================================================================

worker\_processes 1;

events \{

worker\_connections 1024;

http \{

include mime.types;

default\_type application/octet-stream;

sendfile on;

keepalive\_timeout 65;

upstream YZKJ2 \{ \#定义一个代理服务群组

ip\_hash;

server 10.124.20.242 weight=1;

server 10.124.20.243 weight=1;

server \{

listen 80;

server\_name localhost;

location / \{

proxy\_pass http://YZKJ2; \#开启代理服务，直接执行代理服务群组

error\_page 500 502 503 504 /50x.html;

location = /50x.html \{

root html;

==================================================================

\#nginx1：10.124.20.251

===================================================================

worker\_processes 1;

events \{

worker\_connections 1024;

http \{

include mime.types;

default\_type application/octet-stream;

sendfile on;

keepalive\_timeout 65;

upstream YZKJ3 \{ \#定义一个代理服务群组

ip\_hash;

server 10.124.20.244 weight=1;

server 10.124.20.245 weight=1;

server 10.124.20.246 weight=1;

server \{

listen 80;

server\_name localhost;

location / \{

proxy\_pass http://YZKJ3; \#开启代理服务，直接执行代理服务群组

error\_page 500 502 503 504 /50x.html;

location = /50x.html \{

root html;

==================================================================

四、web层服务器：

这里使用的是nginx提供的web服务功能，所以，安装步骤请参考第三部分RS的配置

如果使用Apache提供的文本服务，安装流程请见下文：

=======================================================================

Apache源码编译安装流程：（http-2.4.10.tar.gz）

cd /usr/local/src/

\#依赖包安装

\*apr\*

tar xf apr-1.4.6.tar.gz

cd apr-1.4.6

./configure --prefix=/usr/local/apr

make && make install

\*apr-util\*

tar xf apr-util-1.4.1.tar.gz

cd apr-util-1.4.1

./configure --prefix=/usr/lcoal/apr-util --with-apr=/usr/local/apr

make && make intsall

\*pcre\*

tar xf pcre-8.36.tar.gz

cd pcre-8.36

./configure --prefix=/usr/local/pcre

make

make check

make install

\#Apache安装

tar xf httpd-2.4.10.tar.gz

cd httpd-2.4.10

./configure --prefix=/usr/local/apache --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util --with-pcre=/usr/local/pcre --enable-module=so --enable-deflate=shared --enable-expires=shared --enable-rewrite=shared --enable-cache --enable-file-cache --enable-men-cache --enable-disk-cache --enable-static-support --enable-static-ab --disable-userdir --with-mpm=prefork --enable-nonportable-atomics --disable-ipv6 --with-senfile

make && make install

\#Apache启动

/usr/local/src/apache/bin/apachectl start

======================================================================================

五、测试

浏览器访问测试117.78.3.253，能够返回想要的信息即为成功！

六、总结

使用LVS+Keepalived做负载均衡，首先先将内部各个服务器的iptables与selinux关闭，确保内部网络通畅，一般而言，将上述防火墙关闭之后，安装上述配置文件提供的参数进行配置，是能够实现预期功能的。我的文档中可能提到过要设置iptables规则，这个要在测试通过之后，再打开iptables，然后写上防火墙规则。通过查阅很多资料，看到很多人都在防火墙关闭的情况实验成功，看来防火墙与LVS的分发功能是多少有点冲突，对于这方面的知识学习，有待于我们在今后的工作和学习中进行探索，我也将对后续知识在被人博客上做介绍，希望以后能在这个平台与大家一起交流、学习、成长！

qingbo.song@apicloud.com

2014-12-02

2014-12-05更新

更新：防火墙设置：

修改防火墙

\-A INPUT -p vrrp -j ACCEPT \#基于DR模式，当用户发出请求后，只有DR响应ARP广播包，允许vrrp虚拟路由器冗余协议

重启防火墙：service iptables restart

2015.11.24更新

本文转自 南非波波 51CTO博客，原文链接：http://blog.51cto.com/nanfeibobo/1585684，如需转载请自行联系原作者

[wKiom1R9pDCBdH6DAAFs7SXE3aQ326.jpg]: /images/20220205/8f1a451673b3418da87ef32a54621731.png
[http_down1.chinaunix.net_distfiles_ipvsadm-1.24.tar.gz]: http://down1.chinaunix.net/distfiles/ipvsadm-1.24.tar.gz