Nginx反向代理 + Keepalived双机热备 + Tomcat服务器池 + 后端数据库

我不是女神ヾ 2022-04-13 09:28 99阅读 0赞

### 一、系统环境 ###

1、2台CentOS 7作为nginx服务器绑定同一块虚拟网卡：vnet1  
对应的IP分别为：192.168.80.100  
192.168.80.110  
2、2台CentOS 7作为tomcat服务器同样绑定同一款网卡：vnet1  
对应的IP设置为：192.168.80.120  
192.168.80.130  
3、1台CentOS 7 作为后端数据库绑定网卡：vnet1对应的IP为192.168.80.140

### 二、项目简介 ###

**1、拓扑图**  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat]  
**2、项目说明**  
（1）两台nginx服务器作反代理  
（2）两台nginx服务器实现双机热备  
（3）Nginx+tomcat实现负载均衡  
（4）在tomcat上部署商城项目实现tomcat服务器与数据库的对接  
**3、项目需求**  
实现niginx服务器互为热备份，一台服务器停止，另外一台继续运行。tomcat负载均衡，同样一台停止工作另外一台能继续工作，整个项目是为了让商城项目成功运行，并且在后台服务器down机过程中，用户无感知。

### 三、项目部署 ###

### 部署nginx服务器 ###

**1、安装nginx服务器（另外一台相同操作）**

yum install -y \        //搭建安装工具和编辑环境
    gcc \
    gcc-c++ \
    make \
    openssl-devel \
    zlib-devel \
    pcre-devel

useradd \-s /sbin/nologin -M nginx //添加帐号

tar vxf nginx-1.13.9.tar.gz //解压nginx源码包

cd nginx-1.13.9  
./configure \\ //检查环境  
–prefix=/usr/local/nginx \\ //指定安装路径  
–user=nginx  
–group=nginx

make && make install //编译并且安装

ln \-s /usr/local/nginx/sbin/nginx /usr/local/sbin/ //nginx命令做软连接方便使用

vi /etc/init.d/nginx //制作nginx启动脚本 
\#!/bin/bash 
\# chkconfig: 35 99 20 
\# description: Nginx Service Control Script 
PROG="/usr/local/nginx/sbin/nginx" 
PIDF="/usr/local/nginx/logs/nginx.pid" 
case " 1 < / s p a n > " < / s p a n > < s p a n c l a s s = " h l j s − k e y w o r d " > i n < / s p a n > s t a r t ) < s p a n c l a s s = " h l j s − v a r i a b l e " > 1" in start) 1"<spanclass="hljs−keyword">instart)<spanclass="hljs−variable">PROG 
;; 
stop) 
kill \-s QUIT ( c a t < s p a n c l a s s = " h l j s − v a r i a b l e " > (cat (cat<spanclass="hljs−variable">PIDF) 
;; 
restart) 
$0 stop 
$0 start 
;; 
reload) 
kill \-s HUP ( c a t < s p a n c l a s s = " h l j s − v a r i a b l e " > (cat (cat<spanclass="hljs−variable">PIDF) 
;; 
\*) 
echo “Usage: $0 \{start|stop|restart|reload\}” 
exit 1 
esac 
exit 0 
chmod +x /etc/init.d/nginx //更改启动脚本的权限 
chkconfig --add nginx //将脚本添加到systemctl环境中 
systemctl start nginx //启动nginx服务器

**2、验证两台nginx服务器搭建成功**  
验证第一台：192.168.80.100  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 1]  
验证第二台：192.168.80.110  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 2]

### 部署keepalived双机热备 ###

**1、nginx主服务器中安装keepalived**

yum -y install \        //安装所需环境
    popt-devel \
    kernel-devel \
    openssl-devel
    tar xvf keepalived-1.4.2.tar.gz     //解压源码安装包

cd keepalived\-1.4.2 //进目录  
./configure --prefix=/ //检查编译环境  
make && make install //编译并安装

cp keepalived/etc/init.d/keepalived /etc/init.d/ //将keepalived添加开机进程中  
systemctl enable keepalived //设置开机自启

**2、修改keepalived配置文件**

vi /etc/keepalived/keepalived.conf
    删除所有内容添加下面内容：
    ! Configuration File for keepalived
    global_defs {
            route_id NGINX-01       //服务器名称
       }
    vrrp_script nginx {
            script "/opt/nginx.sh"  //keepalived状态检查配置文件的路径
            interval 2
            weight -10  //检查失败对应优先级减少的值
    }
    vrrp_instance VI_1 {
        state MASTER        //状态为主服务器
        interface ens33 virtual_router_id 51 priority 150 //优先级设为150 advert_int 1 authentication {
            auth_type PASS  //认证方式 
            auth_pass 1111  //认证密码
        }
        track_script {
            nginx
            }
        virtual_ipaddress {
            192.168.80.90       //两台nginx服务器共同的虚拟IP也称漂移地址
        }
    }

**3、添加检查keepalived状态的配置文件**

vi /opt/nginx.sh    //该脚本方便同时将nginx和keepalived服务器同时启动
    添加以下内容
    #!/bin/bash 
    #Filename:nginx.sh
    A=$(ps -ef | grep keepalived | grep -v grep | wc -l)
    if [ $A -gt 0 ]; then
            /etc/init.d/nginx start
    else
            /etc/init.d/nginx stop
    fi

chmod +x /opt/nginx.sh //添加执行权限  
systemctl start keepalived //开启服务

**4、验证漂移地址vip是否生成**  
`ip addr show dev ens33`  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 3]  
**5、nginx从服务器所有部署如上区别如下：**  
（1）keepalived.conf配置文件中route\_id更改为NGINX-02，状态state更改为从服务器BACKUP，优先级priority更改为100。  
`vi /etc/keepalived/keepalived`  
删除所有内容添加下面内容：

! Configuration File for keepalived
    global_defs {
            route_id NGINX-02       //服务器名称为02
       }
    vrrp_script nginx {
            script "/opt/nginx.sh"  //keepalived状态检查配置文件的路径
            interval 2
            weight -10  //检查失败对应优先级减少的值
    }
    vrrp_instance VI_1 {
        state BACKUP        //状态为从服务器
        interface ens33 virtual_router_id 51 priority 100 //优先级设为100与主相差50 advert_int 1 authentication {
            auth_type PASS  //认证方式 
            auth_pass 1111  //认证密码
        }
        track_script {
            nginx
            }
        virtual_ipaddress {
            192.168.80.90       //两台nginx服务器共同的虚拟IP也称漂移地址
        }
    }

（2）nginx.sh脚本中nginx服务的启动不是根据keepalived的启动而启动，而是根据  
判断漂移地址是否存在而启动nginx服务。

vi /opt/nginx.sh
    #!/bin/bash
    #Filename:nginx.sh
    A=$(ip addr | grep 192.168.80.90/32 | grep -v grep | wc -l)
    if [ $A -gt 0 ]; then
            /etc/init.d/nginx start
    else
            /etc/init.d/nginx stop
    fi

**6、验证nginx从服务器配置是否生效**  
停止主服务器查看漂移地址  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 4]  
漂移地址出现在从服务器上  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 5]  
重启主服务器再次查看漂移地址  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 6]  
漂移地址重新回到主服务器从服务器处于热备状态  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 7]  
**7、验证漂移地址是否能访问nginx**  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 8]  
主服务器down机验证访问  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 9]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 10]

### 部署tomcat ###

**说明：**  
由于tomat服务器是运行在jdk环境中，因此首先要搭建jdk环境。  
**1、搭建jdk环境**

tar xvf jdk-8u144-linux-x64.tar.gz      //解压jdk
    cp -r jdk1.8.0_144/ /usr/local/java     //将解压好的jdk下的文件复制到java目录下
    vi /etc/profile     //添加jdk系统变量
    export JAVA_HOME=/usr/local/java
    export JRE_HOME=/usr/local/java/jre
    export PATH=$PATH:/usr/local/java/bin
    export CLASSPATH=./:/usr/local/java/lib:/usr/local/java/jre/lib
    source /etc/profile     //刷新下文件
    java -version       //查看版本成功说明环境搭建成功

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 11]  
**2、安装tomcat**

tar xvf apache-tomcat-8.5.23.tar.gz     //解压软件包
    cp -r apache-tomcat-8.5.23 /usr/local/tomcat8   //将解压好包复制到tomcat8目录下

//将tomcat启动命令软链接到系统命令的目录下重命名为tomcatup方便使用  
ln \-s /usr/local/tomcat8/bin/startup.sh /usr/bin/tomcatup  
ln \-s /usr/local/tomcat8/bin/shutdown.sh /usr/bin/tomcatdown

tomcatup //启动tomcat

**3、验证两台tomcat服务器是否成功访问（另一台tomcat部署操作上同）**  
验证访问第一台192.168.80.120:8080  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 12]  
验证访问第二台192.168.80.130:8080  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 13]

### 部署nginx反代理 ###

**1、配置nginx服务器**  
`vi /usr/local/nginx/conf/nginx.conf`  
删除location块中的内容添加反向代理  
`proxy_pass http://tomcat_pool;`  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 14]  
**2、定义上游请求实现负载均衡**  
在server模块上面添加名为tomcat\_pool地址池

upstream tomcat_pool {
            server 192.168.80.120:8080 weight=1;
    server 192.168.80.130:8080 weight=1;
    }

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 15]  
**3、检查配置文件重启nginx**

nginx -t
    systemctl restart nginx

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 16]  
**4、验证通过nginx服务器vip地址：192.168.80.90访问tomcat服务器**  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 17]

### 部署后端数据库 ###

**1、搭建安装环境**

yum -y install \
    gcc \
    gcc-c++ \
    make \
    ncurses \
    ncurses-devel \
    bison \
    Cmake

**2、添加数据库用户**  
`useradd -s /sbin/nologin mysql //添加数据库用户`  
**3、安装mysql数据库**

tar xf mysql-boost-5.7.20.tar.gz –C /opt/       //解压源码包

cmake \\ //检查安装配置环境  
\-DCMAKE\_INSTALL\_PREFIX=/usr/local/mysql  
\-DMYSQL\_UNIX\_ADDR=/usr/local/mysql/mysql.sock  
\-DSYSCONFDIR=/etc  
\-DSYSTEMD\_PID\_DIR=/usr/local/mysql  
\-DDEFAULT\_CHARSET=utf8  
\-DDEFAULT\_COLLATION=utf8\_general\_ci  
\-DWITH\_INNOBASE\_STORAGE\_ENGINE=1  
\-DWITH\_ARCHIVE\_STORAGE\_ENGINE=1  
\-DWITH\_BLACKHOLE\_STORAGE\_ENGINE=1  
\-DWITH\_PERFSCHEMA\_STORAGE\_ENGINE=1  
\-DMYSQL\_DATADIR=/usr/local/mysql/data  
\-DWITH\_BOOST=boost  
\-DWITH\_SYSTEMD=1

make && make install //编译安装

chown -R mysql.mysql /usr/local/mysql/ //将数据库安装目录的使用权限给用户mysql

vi /etc/my.cnf //将主配置文件清空添加以下内容  
\[client\]  
port = 3306  
default\-character\-set=utf8  
socket = /usr/local/mysql/mysql.sock

\[mysql\]  
port = 3306  
default\-character\-set=utf8  
socket = /usr/local/mysql/mysql.sock

\[mysqld\]  
user = mysql  
basedir = /usr/local/mysql  
datadir = /usr/local/mysql/data  
port = 3306  
character\_set\_server=utf8  
pid-file = /usr/local/mysql/mysqld.pid  
socket = /usr/local/mysql/mysql.sock  
server\-id = 1

sql\_mode=NO\_ENGINE\_SUBSTITUTION,STRICT\_TRANS\_TABLES,NO\_AUTO\_CREATE\_USER,NO\_AUTO\_VALUE\_ON\_ZERO,NO\_ZERO\_IN\_DATE,NO\_ZERO\_DATE,ERROR\_FOR\_DIVISION\_BY\_ZERO,PIPES\_AS\_CONCAT,ANSI\_QUOTES

chown mysql:mysql /etc/my.cnf //更改主配置文件的权限

echo ‘PATH=/usr/local/mysql/bin:/usr/local/mysql/lib:$PATH’ >> /etc/profile  
echo ‘export PATH’ >> /etc/profile //添加系统环境变量  
source /etc/profile //刷新立即生效

cd /usr/local/mysql/ //进入安装目录初始化数据库 
bin/mysqld 
–initialize-insecure 
–user=mysql 
–basedir=/usr/local/mysql 
–datadir=/usr/local/mysql/data

cp usr/lib/systemd/system/mysqld.service /usr/lib/systemd/system/ //制作数据库系统启动命令  
systemctl daemon-reload //刷新识别启动脚本  
systemctl start mysqld //启动数据库  
netstat -anpt | grep 3306 //查看端口是否在监听状态

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 18]

systemctl enable mysqld     //开机自启
    mysqladmin -u root -p password "abc123"     //添加数据库用户
    mysql -u root -p        //进入数据库

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 19]

### 部署商城项目 ###

**1、将商城项目上传到两台tomcat服务器中(两台配置一样以其中一台为例）** 
`tar xf SLSaleSystem.tar.gz //解压写好的商城项目` 
**2、修改tomcat主配置文件** 
`vi /usr/local/tomcat8/conf/server.xml` 
在webapps站点下指定项目的路径 
<Context docBase="/root/SLSaleSystem" path="" reloadable="flase"/> 
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 20] 
**3、tomcat与后端数据库对接实现用户登陆**

mysql -u root –p //进入数据库
 create database slsaledb; //为商城项目创建数据库
 GRANT all ON slsaledb.* TO 'testuser'@'%' IDENTIFIED BY 'admin123'; //授权testuser 用户连接slsaledb数据库
 flush privileges;
 mysql -u root -p < slsaledb-2014-4-10.sql //商城项目对应的数据库文件上传到slsaledb数据库中

**4、更改商城项目中的数据库对接配置文件**  
`vi SLSaleSystem/WEB-INF/classes/jdbc.properties`  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 21]  
**5、重启tomcat服务器并用本地服务器地址访问登陆**

tomcatdown
    tomcatup

访问192.168.80.120:8080  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 22]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 23]  
**6、验证nginx代理服务器vip访问**  
由于商城项目有登陆会话界面因此要想登陆成功需要在nginx服务器的代理地址池中添加会话保持功能，ip\_hash；  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 24]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 25]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 26]  
**7、测试用户down机用户无感知访问**  
先down掉主nginx服务器验证访问

systemctl stop keepalived
    killall -9 nginx
    netstat -anput | grep nginx

![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 27]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 28]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 29]  
同时down掉一台tomcat服务器验证访问(down掉192.168.80.120服务器)  
`tomcatdown`  
并用tomcat本地IP验证down机成功  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 30]  
访问vip:192.168.80.90  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 31]  
![群集架构篇——nginx反向代理+keepalived双机热备+tomcat服务器池+后端数据库][nginx_keepalived_tomcat 32]

### 四、总结 ###

本文只做了简单的架构，现实中还需在nginx服务器和tomcat服务器中做各种优化，以便让客户感受最佳的访问效果。

[nginx_keepalived_tomcat]: /images/20220413/5c320bc9934e4fcb9c094c6941712f62.png
[nginx_keepalived_tomcat 1]: /images/20220413/08e6edac57ef44dfa420ec817dfb4edd.png
[nginx_keepalived_tomcat 2]: /images/20220413/a53837a8ea284c0eac79065d1f00b97f.png
[nginx_keepalived_tomcat 3]: /images/20220413/8a5956e4b2b245d99b920f0c49eda475.png
[nginx_keepalived_tomcat 4]: /images/20220413/d9cba15df8a44056afbf7b277878f3bb.png
[nginx_keepalived_tomcat 5]: /images/20220413/59c9d47800e64ac0a13103899df8b511.png
[nginx_keepalived_tomcat 6]: /images/20220413/9b917f1431554c13883852c344dbdd26.png
[nginx_keepalived_tomcat 7]: /images/20220413/10de24adeef74fc7b702b7e38a5bf008.png
[nginx_keepalived_tomcat 8]: /images/20220413/21802615fef642768a889cf97b083607.png
[nginx_keepalived_tomcat 9]: /images/20220413/1ae0fd02b2e042f1850b74d4370917e9.png
[nginx_keepalived_tomcat 10]: /images/20220413/ba8764889ece4f78a689aeab41cdd505.png
[nginx_keepalived_tomcat 11]: /images/20220413/7a7e0c30e2aa45678a01cd7a97c59c15.png
[nginx_keepalived_tomcat 12]: /images/20220413/903dcfb0c751498eab15f9881e2c3947.png
[nginx_keepalived_tomcat 13]: /images/20220413/a5f2991d60a342fc97170beb4e25f0de.png
[nginx_keepalived_tomcat 14]: /images/20220413/c666436c1be647be8e5cc6d5478f41a4.png
[nginx_keepalived_tomcat 15]: /images/20220413/830ca33f2edc4d369ae796c3a5f55904.png
[nginx_keepalived_tomcat 16]: /images/20220413/ce7892633e5f4a7fbe669f4e0ab77375.png
[nginx_keepalived_tomcat 17]: /images/20220413/81c86f402de84c89bbd37fc5a31551bb.png
[nginx_keepalived_tomcat 18]: /images/20220413/70981a8fec714389914074dadd3fc941.png
[nginx_keepalived_tomcat 19]: /images/20220413/0b8e781046a54d10aabd5086fdc10d66.png
[nginx_keepalived_tomcat 20]: /images/20220413/b34716b228dd4c049d0c10c899a1d523.png
[nginx_keepalived_tomcat 21]: /images/20220413/2d0aa2ca00fa4283b90657bd42d18cdd.png
[nginx_keepalived_tomcat 22]: /images/20220413/0fa1e95377614816a07aabf8c157f8b6.png
[nginx_keepalived_tomcat 23]: /images/20220413/035146b2d7934154875b0483ab497349.png
[nginx_keepalived_tomcat 24]: /images/20220413/defc822dcb9b4a70acbebb9f040fe418.png
[nginx_keepalived_tomcat 25]: /images/20220413/4f359eae6dcc4f8b9ddee1d670b391b1.png
[nginx_keepalived_tomcat 26]: /images/20220413/a5ad32d26a7d49e3bb1e055a4ee3e3b1.png
[nginx_keepalived_tomcat 27]: /images/20220413/c86bab62983e4210a22812cc1d3012e2.png
[nginx_keepalived_tomcat 28]: /images/20220413/3b828306e38440d6956a8a988948da44.png
[nginx_keepalived_tomcat 29]: /images/20220413/a5454f1ae6a944da91fca5e5003059c2.png
[nginx_keepalived_tomcat 30]: /images/20220413/f2d4303eb7ad4e82a4c58db7c047f5e2.png
[nginx_keepalived_tomcat 31]: /images/20220413/ae0af3ed575542738468a4122de5115d.png
[nginx_keepalived_tomcat 32]: /images/20220413/6bb11a743da244eeaf7877b4d966f551.png