Nginx+Keepalived简单构建高可用集群

￡神魔★判官ぃ 2022-07-14 02:22 338阅读 0赞

以前一直用heartbeat或corosync+pacemaker构建高可用集群，现在发现keepalived实现起来更简单。  
keepalived的master向backup发送广播，当backup一段时间收不到对方传来的VRRP广播时，backup会通过竞选一个master,master就会重新持有资源。具体的理论知识参见[http://bbs.ywlm.net/thread-790-1-1.html][http_bbs.ywlm.net_thread-790-1-1.html]

实验目标:2台Nginx+Keepalived 2台Lamp构建高可用Web集群

规划：

1.  ng1.laoguang.me 192.168.1.22 ng1  
2.  ng2.laoguang.me 192.168.1.23 ng2  
3.  lamp1.laoguang.me   192.168.1.24 lamp1  
4.  lamp2.laoguang.me   192.168.1.25 lamp2

拓扑：

[![104100472.jpg][]][104100472.jpg]

一.基本环境准备  
ng1,ng2上安装nginx  
lamp1,lamp2上构建LAMP或只安装httpd，我只安装了Httpd，这里不给大家演示了，有需要请看我的其它博文，更改lamp1,lamp2的index.html的内容分别为lamp1和lamp2,以容易区分，实际集群中内容应该是一致的，由共享存储提供。

二.ng1,ng2上安装配置keepalived  
下载地址:http://www.keepalived.org/download.html  
2.1 安装keepalived

1.  tar xvf keepalived-1.2.7.tar.gz  
2.  cd keepalived-1.2.7  
3.  ./configure \--prefix=/usr/local/keepalived      
4.  \#\#可能会提示安装popt-devel包，yum即可
5.  make && make install

2.2 整理配置文件与脚本

1.  mkdir /etc/keepalived  
2.  \#\#keepalived默认配置文件从/etc/keepalived下读取
3.  cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
4.  \#\#就一个二进制文件，直接拷贝过去即可，多的话就更改PATH吧
5.  cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
6.  \#\#脚本的额外配置文件读取位置  
7.  cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/  
8.  \#\#启动脚本你懂得
9.  cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
10. \#\#我们关键的keepalived配置文件

2.3 修改ng1的/etc/keepalived/keepalived.conf

1.  ! Configuration File for keepalived  
2.  global\_defs \{  
3.    notification\_email \{  
4.      ibuler@qq.com         \#\#出故障发送邮件给谁  
5.    \}    
6.    notification\_email\_from keepalived@localhost \#\#故障用哪个邮箱发送邮件  
7.    smtp\_server 127.0.0.1   \#\#SMTP\_Server IP
8.    smtp\_connect\_timeout 30 \#\#超时时间
9.    router\_id LVS\_DEVEL     \#\#服务器标识  
10. \}  
11. vrrp\_instance VI\_1 \{  
12.    state BACKUP            
13. \#\#状态，都为BACKUP，它们会推选Master，如果你写MASTER，它就会是Master，
14.    \#\#当Master故障时Backup会成为Master,当原来的Master恢复后，原来的Master会成为Master  
15.    interface eth0       \#\#发送VRRP的接口，仔细看你的是不是eth0
16.    virtual\_router\_id 51  \#\#虚拟路由标识，同一个组应该用一个，即Master与Backup同一个
17.    priority 100   \#\#重要的优先级哦  
18.    nopreempt      \#\#不抢占，一个故障时，重启后恢复后不抢占意资源
19.    advert\_int 1   \#\#同步间隔时长
20.    authentication \{             \#\#认证  
21.        auth\_type PASS            \#\#认证方式  
22.        auth\_pass www.laoguang.me \#\#密钥
23.    \}  
24.    virtual\_ipaddress \{  
25.        192.168.1.18/24 dev eth0              \#\#VIP  
26.    \}  
27. \}  
28. \#\#后面的删除吧，LVS上才有用

拷贝到ng2上一份，只修改priority 90 即可

1.  scp /etc/keepalived/keepalived.conf 192.168.1.23:/etc/keepalived/
2.  \#\#Ng2上
3.  vi /etc/keepalived/keepalived.conf  priority 90   \#\#其它一致

2.4 ng1,ng2上启动keepalived

1.  service keepalived start

查看日志

1.  tail /var/log/messages  
2.  Nov 27 08:07:54 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) Entering BACKUP STATE  
3.  Nov 27 08:07:54 localhost Keepalived\_vrrp\[41871\]: VRRP sockpool: \[ifindex(2), proto(112), fd(11,12)\]  
4.  Nov 27 08:07:54 localhost Keepalived\_healthcheckers\[41870\]: Using LinkWatch kernel netlink reflector...  
5.  Nov 27 08:07:54 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) forcing a new MASTER election  
6.  Nov 27 08:07:55 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) Transition to MASTER STATE  
7.  Nov 27 08:07:56 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) Entering MASTER STATE  
8.  Nov 27 08:07:56 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) setting protocol VIPs.  
9.  Nov 27 08:07:56 localhost Keepalived\_healthcheckers\[41870\]: Netlink reflector reports IP 192.168.1.18 added  
10. Nov 27 08:07:56 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) Sending gratuitous ARPs on eth0 for 192.168.1.18  
11. Nov 27 08:08:01 localhost Keepalived\_vrrp\[41871\]: VRRP\_Instance(VI\_1) Sending gratuitous ARPs on eth0 for 192.168.1.18

查看vip绑定到哪台机器上了

1.  ip addr     \#\#ng1上  
2.  ....省略  
3.  2: eth0: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen 1000  
4.     link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff  
5.     inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0  
6.     inet 192.168.1.18/32 scope global eth0  
7.     inet6 fe80::20c:29ff:fee8:900b/64 scope link  
8.        valid\_lft forever preferred\_lft forever

由此可知vip绑定到ng1上了  
三，Keepalived测试

3.1 关闭ng1上的keepalived或者直接关闭ng1 查看vip转移情况

1.  service keepalived stop  
2.  ip addr  
3.  2: eth0: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen 1000  
4.     link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff  
5.     inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0  
6.     inet6 fe80::20c:29ff:fee8:900b/64 scope link  
7.        valid\_lft forever preferred\_lft forever

3.2 查看ng2上是否绑定了vip

1.  ip addr  
2.  2: eth0: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen 1000  
3.     link/ether 00:0c:29:dd:00:77 brd ff:ff:ff:ff:ff:ff  
4.     inet 192.168.1.23/24 brd 192.168.1.255 scope global eth0  
5.     inet 192.168.1.18/32 scope global eth0  
6.     inet6 fe80::20c:29ff:fedd:77/64 scope link  
7.        valid\_lft forever preferred\_lft forever

由此可知ip转移正常，keepalived设置成功

四.配置Nginx做反向代理

4.1 修改nginx配置文件

1.  vi /etc/nginx/nginx.conf  
2.  user  nginx nginx;   \#\#运行nginx的用户和组
3.  worker\_processes  2; \#\#启动进程数
4.  error\_log /var/log/nginx/error.log  notice; \#\#错误日志记录
5.  pid        /tmp/nginx.pid;                   \#\#pid存放位置
6.  worker\_rlimit\_nofile 65535;                  \#\#线程最大打开文件数，须配合ulimit -SHn使用  
7.  events \{  
8.     use epoll;                 \#\#工作模型  
9.     worker\_connections  65536; \#\#单进程最大连接数
10. \}  
11. http \{                        \#\#http模块      
12.    include       mime.types;  \#\#包含进来
13.    default\_type  application/octet-stream; \#\#默认类型  
14.    log\_format  main  '$remote\_addr - $remote\_user \[$time\_local\] "$request" '  
15.                      '$status $body\_bytes\_sent "$http\_referer" '  
16.                      '"$http\_user\_agent" "$http\_x\_forwarded\_for"';  
17. \#\#日志格式
18.    access\_log  /var/logs/nginx/http.access.log  main; \#\#访问日志
19.    client\_max\_body\_size 20m;  \#\#最大请求文件大小
20.    client\_header\_buffer\_size 16k; \#\#来自客户端请求header\_buffer大小
21.    large\_client\_header\_buffers 4 16k; \#\#较大请求缓冲个数与大小
22.    sendfile       on;                 \#\#内核空间直接发送到tcp队列  
23.    tcp\_nopush     on;  
24.    tcp\_nodelay    on;  
25.    keepalive\_timeout  65;    \#\#长连接时长  
26.    gzip  on;                 \#\#启用压缩
27.    gzip\_min\_length 1k;        \#\#最小压缩大小
28.    gzip\_buffers 4 16k;        \#\#压缩缓冲  
29.    gzip\_http\_version 1.1;     \#\#支持协议  
30.    gzip\_comp\_level 2;         \#\#压缩等级  
31.    gzip\_types text/plain application/x-javascript text/css application/xml;      \#\#压缩类型  
32.    gzip\_vary on;              \#\#前端缓存服务器可以缓存压缩过的页面
33.    upstream laoguang.me \{     \#\#用upstream模块定义集群与RS
34.        server 192.168.1.24:80 max\_fails=3fail\_timeout=10s;   \#\#RS的地址，最大错误数与超时时间，超过了自动剔除  
35.        server 192.168.1.25:80 max\_fails=3fail\_timeout=10s;  
36. \}  
37. server \{  
38.        listen       80;           \#\#监听端口
39.        server\_name  192.168.1.18; \#\#servername
40.        root   html;               \#\#根目录  
41.        index  index.html index.htm; \#\#你懂得
42.        \#charset koi8-r;  
43.        access\_log  logs/192.168.1.18.access.log  main;  
44.  \#\#这个server的访问日志
45.        location / \{    
46.                proxy\_pass http://laoguang.me;  \#\#反向代理
47.                proxy\_redirect off;  
48.                proxy\_set\_header X-Real-IP $remote\_addr;  
49. \#\#真实客户ip告诉后端
50.                proxy\_set\_header X-Forwarded-For Proxy\_add\_x\_forwarded\_for;  
51.        \}  
52.        location /nginx \{  
53.                access\_log off;    
54.                stub\_status on; \#\#状态页面
55.        \}  
56.        error\_page   500 502 503 504  /50x.html;  
57. location = /50x.html \{  
58.            root   html;  
59.        \}  
60.    \}  
61. \}

4.2 拷贝到ng2上一份

1.  scp /etc/nginx/nginx.conf 192.168.1.23:/etc/nginx/

4.3 测试反向代理能否负载均衡

lamp1,lamp2启动httpd

1.  service httpd start

ng1重启nginx

1.  service nginx restart

用RealIp访问测试能否轮询  
[http://192.168.1.22][http_192.168.1.22]

同样测试ng2,如果都能实现负载均衡，那么继续

五.测试keepalived与nginx配合运行

现在192.168.1.18在 ng2上，        访问 [http://192.168.1.18][http_192.168.1.18] 测试能否轮询  
ng2上 service keepalived stop     访问测试 [http://192.168.1.18][http_192.168.1.18] 能否轮询  
关闭lamp1上的service httpd stop   访问测试[http://192.168.1.18][http_192.168.1.18] 是否会报错

到此高可用webserver构建完毕，没有单点故障，任何一点故障不影响业务。

[http_bbs.ywlm.net_thread-790-1-1.html]: http://bbs.ywlm.net/thread-790-1-1.html
[104100472.jpg]: /images/20220714/8208bca47f9146d4b080d5e2bcdd69b8.png
[http_192.168.1.22]: http://192.168.1.22/
[http_192.168.1.18]: http://192.168.1.18/