centos7中keepalived+nginx做双机热备和反向代理

- 日理万妓 2022-06-05 05:51 225阅读 0赞

我虚拟了三台centos7，要将one（192.168.56.31）、two（192.168.56.32）配置成nginx+keepalived双机热备（思路是当一台服务器的nginx挂掉时，可以自动用另一台的nginx,就是当主服务器nginx的端口不通时，自动关闭自己的keepalived服务，这样虚拟ip就会指到备用ip上），three（192.168.56.33）做tomcat服务器。实验环境下先把防火墙和selinux关闭：`setenforce 0 && systemctl stop firewalld`  
首先进行nginx的安装，我用的是最简单的yum安装方式，自动的yum源中没有nginx，我们需要安装一下yum源然后再来安装nginx：`yum -y install epel-release && yum -y install nginx`。然后再来安装一下keepalived：`yum -y install keepalived`。安装完成后可以使用`keepalived --help` 来进行keepalived的命令帮助。两者的配置文件都在etc目录下。再将jdk和tomcat上传到three中并安装，我这设置了三个tomcat实例，端口分别为：8180、8280、8380。  
然后先来配置nginx，配置文件在/etc/nginx/中：

# For more information on configuration, see:
    # * Official English Documentation: http://nginx.org/en/docs/
    # * Official Russian Documentation: http://nginx.org/ru/docs/
    
    user nginx;                          #运行用户，避免权限问题，就用root吧
    worker_processes auto;               #工作进程数，表示自动选择，cpu多的话，可以手动设置一般为cpu的倍数。
    error_log /var/log/nginx/error.log;  #错误日志路径
    pid /run/nginx.pid;                  #PID文件路径
    
    # Load dynamic modules. See /usr/share/nginx/README.dynamic. 
    include /usr/share/nginx/modules/*.conf;   #包含这个文件的信息
    
    events {       
        worker_connections 1024;         #工作连接数，指一个进程可以产生多少个连接，用 ulimit -n 来查看，但是这是理论上，没必要设置这么大
    }
    
    http {     
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '   #定义访问日志的显示格式
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;                                #访问日志的路径
    
        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 2048;
    
        include             /etc/nginx/mime.types;                                 #将指定的配置文件引入到这个文件中来
        default_type        application/octet-stream;
    
        # Load modular configuration files from the /etc/nginx/conf.d directory.
        # See http://nginx.org/en/docs/ngx_core_module.html#include
        # for more information.
        include /etc/nginx/conf.d/*.conf;
    
        server {                                          #服务器设置，可以设置多个服务器
            listen       80 default_server;               #设置监听的端口
            listen       [::]:80 default_server;            
            server_name  _;                               #服务器名称
            root         /usr/share/nginx/html;           #服务器默认网站的目录
    
            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;
    
            location / {                                  #设置请求转向
            proxy_pass http://cs;
            proxy_set_header HOST $host;                   #不添加下面三行时，有次转发后点击连接前面域名会变为cs
            proxy_set_header X-Real-IP $remote_addr; 
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
    
            }
    
            error_page 404 /404.html;
                location = /40x.html {
            }
    
            error_page 500 502 503 504 /50x.html;
                location = /50x.html {
            }
        }
    upstream cs {                      #这是一个反向连接池,指向了三个tomcat实例
     server 192.168.56.33:8180;
     server 192.168.56.33:8280;
     server 192.168.56.33:8380;
    }
    # Settings for a TLS enabled server.
    #
    # server { 
    # listen 443 ssl http2 default_server;
    # listen [::]:443 ssl http2 default_server;
    # server_name _;
    # root /usr/share/nginx/html;
    #
    # ssl_certificate "/etc/pki/nginx/server.crt";
    # ssl_certificate_key "/etc/pki/nginx/private/server.key";
    # ssl_session_cache shared:SSL:1m;
    # ssl_session_timeout 10m;
    # ssl_ciphers HIGH:!aNULL:!MD5;
    # ssl_prefer_server_ciphers on;
    #
    # # Load configuration files for the default server block.
    # include /etc/nginx/default.d/*.conf;
    #
    # location / { 
    # }
    #
    # error_page 404 /404.html;
    # location = /40x.html { 
    # }
    #
    # error_page 500 502 503 504 /50x.html;
    # location = /50x.html { 
    # }
    # }
    
    }

配置完成后把，nginx和三个tomcat服务起来后，就能成功访问了，接下来我们再来配置一下keepalived，配置文件keeplived.conf在安装好后的/etc/keeplived中，下面是配置文件的解释：

! Configuration File for keepalived
    
    global_defs {                                                       #全局定义块
       notification_email {                                             #指定keepalived在发生切换时需要发送email到的对象，一行一个。
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc            #发件人是谁
       smtp_server 192.168.200.1                                        #指定smtp服务器地址 
       smtp_connect_timeout 30                                          #指定smtp连接超时时间
       router_id LVS_DEVEL                                              #Lvs负载均衡器标识（lvs_id）。在一个网络内，它应该是唯一的。
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }

vrrp_instance VI_1 {                                               #虚拟vrrp设置模块 
        state MASTER                                                   #指定那个为master，那个为backup，如果设置了nopreempt这个值不起作用，主备靠priority决定。
        interface eth0                                                 #设置实例绑定的网卡
        virtual_router_id 51                                           #虚拟路由的id
        priority 100                                                   #优先级，高优先级竞选为master
        advert_int 1                                                   #检查间隔，默认1秒
        authentication {                                               #认证设置
            auth_type PASS                                             #认证方式
            auth_pass 1111                                             #认证密码
        }
        virtual_ipaddress {                                            #虚拟ip地址设置
            192.168.200.16
            192.168.200.17
            192.168.200.18
        }
    }

#虚拟服务器virtual_server定义块 ，该部分是用来管理LVS的，是实现keepalive和LVS相结合的模块。ipvsadm命令可以实现的管理在这里都可以通过参数配置实现。
    virtual_server 192.168.200.100 443 {                             #虚拟IP地址，要和vrrp_instance模块中的virtual_ipaddress地址一致
        delay_loop 6                                                 #健康检查时间间隔 
        lb_algo rr                                                   #lvs调度算法有rr|wrr|lc|wlc|lblc|sh|dh 
        lb_kind NAT                                                  #负载均衡转发规则NAT|DR|RUN 
        persistence_timeout 50                                       #会话保持时间
        protocol TCP                                                 #使用的协议
    
        real_server 192.168.201.100 443 {                            #真实ip地址
            weight 1                                                 #默认为1,0为失效
            SSL_GET {
                url {
                  path /
                  digest ff20ad2481f97b1754ef3e12ecd3a9cc
                }
                url {
                  path /mrtg/
                  digest 9b3a0c85a887a256d6939da88aabd8cd
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }

virtual_server 10.10.10.2 1358 { delay_loop 6 lb_algo rr lb_kind NAT persistence_timeout 50 protocol TCP sorry_server 192.168.200.200 1358 real_server 192.168.200.2 1358 { weight 1 HTTP_GET { url { path /testurl/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl2/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl3/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    
        real_server 192.168.200.3 1358 { weight 1 HTTP_GET { url { path /testurl/test.jsp digest 640205b7b0fc66c1ea91c463fac6334c }
                url { path /testurl2/test.jsp digest 640205b7b0fc66c1ea91c463fac6334c }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }
    
    virtual_server 10.10.10.3 1358 { delay_loop 3 lb_algo rr lb_kind NAT persistence_timeout 50 protocol TCP real_server 192.168.200.4 1358 { weight 1 HTTP_GET { url { path /testurl/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl2/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl3/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    
        real_server 192.168.200.5 1358 { weight 1 HTTP_GET { url { path /testurl/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl2/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                url { path /testurl3/test.jsp digest 640205b7b0fc66c1ea91c463fac6334d }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }

配置文件有点多，我们用不到的功能可以不往上配，这样能使性能更好！我的脚本如下：

global_defs { notification_email { 652179279@qq.com }
       notification_email_from
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id one
    }
    vrrp_script chk_nginx { #因为要检测nginx服务状态，所以创建一个检查脚本 script "/usr/local/check_ng.sh" interval 3 }
    vrrp_instance VI_1 { state MASTER interface enp0s8 virtual_router_id 60 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 }
    
        virtual_ipaddress { 192.168.56.100 }
        track_script { chk_nginx }
    }

接下来就是去上面配置的路径创建脚本了`vi /usr/local/check_ng.sh` ：

#!/bin/bash
    d=`date --date today +%Y%m%d_%H:%M:%S`
    n=`ps -C nginx --no-heading|wc -l`
    if [ $n -eq "0" ]; then
            systemctl start nginx
            n2=`ps -C nginx --no-heading|wc -l`
            if [ $n2 -eq "0"  ]; then
                    echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                    systemctl stop keepalived
            fi
    fi

然后增加执行权限。再配置备用机：

global_defs { notification_email { 652179279@qq.com }
       notification_email_from
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id two
    }
    vrrp_script chk_nginx { script "/usr/local/sbin/check_ng.sh" interval 3 }
    vrrp_instance VI_1 { state BACKUP interface enp0s8 virtual_router_id 60 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 }
    
        virtual_ipaddress { 192.168.56.100 }
        track_script { chk_nginx }
    }

配置检测脚本和配置主机一样！生产环境下需要开放112端口，是keepalived的！