Memcached+Magent+Keepalived高可用架构部署

傷城~ 2022-01-11 11:09 320阅读 0赞

## Memcached集群介绍 ##

*  ## 现实生产环境中，我们要实现memcached主从复制避免单点故障风险，往往需要部署高可用群集。这里我们使用magent代理memcached实现主从备份，从而来保证缓存数据的完整性。工作原理如下： ##

> ### 1）数据在到达memcached服务器前会经过magent，每次写数据到主memcached服务器，magent会同时写到从 [服务器][Link 1] 上，并且主从服务器写的算法一样。 ###

> ### 2）当主memcached服务器宕掉之后，magent会向从memcached服务器中读取数据 ###
> 
> ### 3）当memcached恢复后，magent将重新从主memcached中读取数据，此时由于主memcached刚恢复正常，所以在宕掉期间的数据是无法获取的，这也是magent的缺点 ###

*  ## 再通过keepalived设置VIP地址，提供给Memcached API客户端以及magent连接访问。 ##

> ### 1）memcached主从复制这种架构，在程序连接时不知道应该连接哪个主服务器，所以需要用keepalived在前端加VIP地址实现高可用； ###
> 
> ### 2）keepalived不断检测memcached主服务器的11211端口，如果检测到memcached服务发生宕机或者死机等情况，就会将VIP从主服务器移至从服务器，从而实现高可用。 ###

## Memcached高可用架构图： ##

[![image][]][image 1]

## 实验环境介绍： ##

<table> 
 <tbody> 
  <tr> 
   <td><h3>服务器主机</h3></td> 
   <td><h3>IP地址</h3><br></td> 
   <td><h3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 需要安装的软件包&nbsp;</h3></td> 
   <td><h3>操作系统</h3></td> 
  </tr> 
  <tr> 
   <td><h3>master</h3></td> 
   <td><h3>192.168.30.55</h3><h3>VIP:192.168.30.16</h3></td> 
   <td><h3>libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz,</h3><h3>magent-0.5.tar.gz，keepalived</h3></td> 
   <td><h3>CentOS7</h3></td> 
  </tr> 
  <tr> 
   <td><h3>backup</h3></td> 
   <td><h3>192.168.30.10</h3><h3>VIP:192.168.30.16</h3></td> 
   <td><h3>libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz，</h3><h3>keepalived</h3></td> 
   <td><h3>CentOS7</h3></td> 
  </tr> 
  <tr> 
   <td><h3>client</h3></td> 
   <td><h3>192.168.30.15</h3></td> 
   <td><h3>telnet</h3></td> 
   <td><h3>RedHat6.5</h3></td> 
  </tr> 
 </tbody> 
</table>

## 百度网盘免费提供软件包： ##

> ## libevent-2.1.8-stable.tar.gz：[https://pan.baidu.com/s/1TBVEyndXVHcMbxst-kpCuA][https_pan.baidu.com_s_1TBVEyndXVHcMbxst-kpCuA] ##
> 
> ## memcached-1.5.6.tar.gz：[https://pan.baidu.com/s/1oO\_mC6J3SV6PLVepjmi2eQ][https_pan.baidu.com_s_1oO_mC6J3SV6PLVepjmi2eQ] ##
> 
> ## magent-0.5.tar.gz：[https://pan.baidu.com/s/1xoT3Px14RHu6QE-66nLtMQ][https_pan.baidu.com_s_1xoT3Px14RHu6QE-66nLtMQ] ##

## 实验部署： ##

## 首先所有主机关闭防火墙，及selinux ##

> systemctl stop firewalld.service（CentOS7上）或者service iptables stop（RedHat6.5上）  
> setenforce 0

## 配置memcached主缓存节点和从缓存节点（主，从基本步骤相同，个别不同步骤下面会单独说明） ##

*  ## 将事先准备的软件包进行解压 ##

> tar xvf memcached-1.5.6.tar.gz -C /opt/
> 
> tar xvf libevent-2.1.8-stable.tar.gz -C /opt/ \#事件通知库

*  ## 安装编译环境包 ##

> ##### yum install gcc gcc-c++ make -y #####

#####  #####

## 编译安装libvent ##

> cd /opt/libevent-2.1.8-stable  
> ./configure --prefix=/usr \#指定工作目录
> 
> make && make install

*  ## 编译安装memcached ##

> cd /opt/memcached-1.5.6
> 
> ./configure \\  
> \--prefix=/usr/local/memcached \\ \#指定工作目录  
> \--with-libevent=/usr \#指定libevent的安装路径
> 
> make && make install
> 
> ln -s /usr/local/memcached/bin/\* /usr/local/bin/ \#让系统识别命令

*  ## 在主服务器安装magent，从服务器不用安装 ##

> mkdir /opt/magent
> 
> cd ～/  
> tar xvf magent-0.5.tar.gz -C /opt/magent/  
> ketama.c  
> magent.c  
> ketama.h  
> Makefile
> 
> cd /opt/magent/
> 
> ## 编辑配置文件 ##
> 
> vim ketama.h
> 
> \#ifndef SSIZE\_MAX \#将原来的参数修改为SSIZE\_MAX  
> \#define SSIZE\_MAX 32767 \#将原来的参数修改为SSIZE\_MAX 32767
> 
> [![5][]][5 1]
> 
> ## 修改Makefile ##
> 
> vim Makefile
> 
> LIBS = -levent -lm \#在后面加上-lm
> 
> [![6][]][6 1]
> 
> ## 编译（完成后会生成一个magent的可执行文件） ##
> 
> make
> 
> [![7][]][7 1]
> 
> ## 复制mgent命令到系统管理中，便于使用 ##
> 
> #### cp magent /usr/bin/ \#将magent可执行文件加入usr可执行命令中 ####

*  ## 把产生的magent文件远程复制到从服务器上 ##

> \[root@localhost magent\]\# **scp magent 192.168.30.10:/usr/bin**  
> The authenticity of host '192.168.30.10 (192.168.30.10)' can't be established.  
> ECDSA key fingerprint is SHA256:/5yAWf89joXesnoqczcLtVetMK2DIseKpZOcWkC+7V4.  
> ECDSA key fingerprint is MD5:a1:a4:ef:ff:84:46:0b:0c:fb:1d:de:56:4e:36:89:15.  
> Are you sure you want to continue connecting (yes/no)? **yes \#输入yes，同意复制**  
> Warning: Permanently added '192.168.30.10' (ECDSA) to the list of known hosts.  
> root@192.168.30.10's password: **\#输入从服务器的root密码**  
> magent 100% 112KB 4.9MB/s 00:00

*  ## 开启memcached缓存数据库（主，从都开启） ##

> memcached -d -m 32m -p 11211 -u root

*  ## 检查memcached进程是否运行 ##

> netstat -antp | grep 11211  
> tcp 0 0 0.0.0.0:11211 0.0.0.0:\* LISTEN 57782/memcached  
> tcp6 0 0 :::11211 :::\* LISTEN 57782/memcached

## 接着在主从服务器上安装配置keepalived ##

> yum install keepalived -y

*  ## 修改配置文件keepalived.conf ##

> vim /etc/keepalived/keepalived.conf  
> 其余设置项删除掉，就留下面的设置
> 
> ! Configuration File for keepalived
> 
> global\_defs \{  
> notification\_email \{  
> acassen@firewall.loc  
> failover@firewall.loc  
> sysadmin@firewall.loc  
> \}  
> notification\_email\_from Alexandre.Cassen@firewall.loc  
> smtp\_server 192.168.200.1  
> smtp\_connect\_timeout 30  
> router\_id MAGENT\_HA \#指定router\_id， 从服务器为MAGENT\_HB  
> \}
> 
> vrrp\_script magent \{ `#定义函数`  
> script "/opt/shell/magent.sh"  
> interval 2  
> \}
> 
> vrrp\_instance VI\_1 \{  
> state MASTER \#从服务器为BACKUP  
> interface ens33 `#本地网卡名称`  
> virtual\_router\_id 51 \#从服务器id不同  
> priority 100 \#从服务器优先级要小于主的  
> advert\_int 1  
> authentication \{  
> auth\_type PASS  
> auth\_pass 1111  
> \}  
> track\_script \{ \#调用函数  
> magent  
> \}  
> virtual\_ipaddress \{  
> 192.168.30.16 \#设置VIP  
> \}

*  ## 主服务器上的keepalived配置 ##

> \[root@localhost magent\]\# vim /etc/keepalived/keepalived.conf
> 
> ! Configuration File for keepalived
> 
> global\_defs \{  
> notification\_email \{  
> acassen@firewall.loc  
> failover@firewall.loc  
> sysadmin@firewall.loc  
> \}  
> notification\_email\_from Alexandre.Cassen@firewall.loc  
> smtp\_server 192.168.200.1  
> smtp\_connect\_timeout 30  
> **router\_id MAGENT\_HA  
> \}** 
> 
> **vrrp\_script magent \{  
> script "/opt/shell/magent.sh"  
> interval 2  
> \}** 
> 
> vrrp\_instance VI\_1 \{  
> state **MASTER**  
> interface **ens33**  
> virtual\_router\_id **51**  
>  priority **100**  
>  advert\_int 1  
> authentication \{  
> auth\_type PASS  
> auth\_pass 1111  
> \}
> 
> **track\_script \{  
> magent  
> \}**  
>  virtual\_ipaddress \{  
> **192.168.30.16**  
>  \}  
> \}

*  ## 从服务器上的keepalived配置 ##

> \[root@localhost memcached-1.5.6\]\# vim /etc/keepalived/keepalived.conf
> 
> ! Configuration File for keepalived
> 
> global\_defs \{  
> notification\_email \{  
> acassen@firewall.loc  
> failover@firewall.loc  
> sysadmin@firewall.loc  
> \}  
> notification\_email\_from Alexandre.Cassen@firewall.loc  
> smtp\_server 192.168.200.1  
> smtp\_connect\_timeout 30  
> router\_id **MAGENT\_HB**  
> \}
> 
> **vrrp\_script magent \{  
> script "/opt/shell/magent.sh"  
> interval 2  
> \}** 
> 
> vrrp\_instance VI\_1 \{  
> state **BACKUP**  
>  interface **ens33**  
>  virtual\_router\_id **52**  
>  priority **99**  
>  advert\_int 1  
> authentication \{  
> auth\_type PASS  
> auth\_pass 1111  
> \}
> 
> **track\_script \{  
> magent  
> \}**  
>  virtual\_ipaddress \{  
> **192.168.30.16**  
>  \}  
> \}

## 在主服务器上建立/opt/shell目录，创建magent.sh脚本 ##

> mkdir /opt/shell  
> vim /opt/shell/magent.sh
> 
> \#!/bin/bash  
> K=\`ps -ef | grep keepalived | grep -v grep | wc -l\`  
> if \[ $K -gt 0 \]; then  
> magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211  
> else  
> pkill -9 magent  
> fi

*  ## 其中的参数说明 ##

> \-n 51200 //定义用户最大连接数  
> \-l 192.168.175.188 //指定虚拟IP  
> \-p 12000 //指定端口号  
> \-s //指定主缓存服务器  
> \-b //指定从缓存服务器

*  ## 赋予脚本执行权限 ##

> chmod +x magent.sh

*  ## 启动keepalived服务   systemctl start keepalived.service ##
 *  ## 查看12000端口的进程（magent）是否运行 ##

> netstat -antp | grep 12000  
> tcp 0 0 192.168.30.16:12000 0.0.0.0:\* LISTEN 22180/magent

*  ## 查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用 ##

> vim /var/log/messages
> 
> Sep 26 12:20:42 localhost Keepalived\_vrrp\[24260\]: VRRP\_Instance(VI\_1) **Transition to MASTER STATE**  
> Sep 26 12:20:43 localhost Keepalived\_vrrp\[24260\]: VRRP\_Instance(VI\_1) **Entering MASTER STATE**  
> Sep 26 12:20:43 localhost Keepalived\_vrrp\[24260\]: VRRP\_Instance(VI\_1) **setting protocol VIPs.**  
> Sep 26 12:20:43 localhost Keepalived\_vrrp\[24260\]: Sending gratuitous ARP on ens33 for 192.168.30.16  
> Sep 26 12:20:43 localhost Keepalived\_vrrp\[24260\]: VRRP\_Instance(VI\_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.16

## 同样在从服务器上建立/opt/shell目录，创建magent.sh脚本 ##

> mkdir /opt/shell  
> vim /opt/shell/magent.sh
> 
> \#!/bin/bash  
> K=\`ip addr | grep 192.168.30.16 | grep -v grep | wc -l\`  
> if \[ $K -gt 0 \]; then  
> magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211  
> else  
> pkill -9 magent  
> fi

*  ## 赋予脚本执行权限 ##

> chmod +x magent.sh

*  ## 启动keepalived服务   systemctl start keepalived.service ##
 *  ## 查看12000端口的进程（magent）是否运行 ##

> netstat -antp | grep 12000  
> tcp 0 0 192.168.30.16:12000 0.0.0.0:\* LISTEN 22180/magent

*  ## 查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用 ##

> vim /var/log/messages

> Sep 26 12:32:50 localhost Keepalived\_vrrp\[22129\]: **VRRP\_Instance(VI\_1) Entering BACKUP STATE**  
> Sep 26 12:32:50 localhost Keepalived\_vrrp\[22129\]: VRRP sockpool: \[ifindex(2), proto(112), unicast(0), fd(10,11)\]
> 
> Sep 26 12:32:51 localhost Keepalived\_vrrp\[22129\]: **VRRP\_Script(magent) succeeded**

## 在client上测试 ##

*  ## 安装telnet， ##

> yum -y install telnet

*  ## 连接VIP，端口12000进行远程登录memcached群集，测试主从复制同步，添加键值username ##

> ## \[root@localhost ~\]\# **telnet 192.168.30.16 12000** ##
> 
> ## Trying 192.168.30.16... ##
> 
> ## Connected to 192.168.30.16. ##
> 
> ## Escape character is '^\]'. ##
> 
> ## **add username 0 0 6** ##
> 
> ## **abcdef** ##
> 
> ## **STORED** ##
> 
> ## quit ##
> 
> ## Connection closed by foreign host. ##

*  ## 连接master主服务器，查看username，同步成功   \[root@localhost ~\]\# **telnet 192.168.30.55 11211**   Trying 192.168.30.55...   Connected to 192.168.30.55.   Escape character is '^\]'.   **get username**   VALUE username 0 6   **abcdef**   END   quit   Connection closed by foreign host. ##
 *  ## 连接backup从服务器，查看username，同步成功   \[root@localhost ~\]\# **telnet 192.168.30.10 11211**   Trying 192.168.30.10...   Connected to 192.168.30.10.   Escape character is '^\]'.   **get username**   VALUE username 0 6   **abcdef**   END   quit   Connection closed by foreign host. ##

## 测试故障转移 ##

*  ## 在主服务器端关闭memcached服务 ##

[![30][]][30 1]

*  ## 确认从服务器端memcached服务正常运行 ##

[![31][]][31 1]

*  ## 在client端通过VIP连接群集，检测群集仍然正常运行 ##

[![32][]][32 1]

转载于:https://blog.51cto.com/13728740/2286022

[Link 1]: http://www.codercto.com/category/server.html
[image]: https://s1.51cto.com/images/blog/201809/26/3093d3327aec822f69d3a719c4534801.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[image 1]: https://s1.51cto.com/images/blog/201809/26/946c6b025d6b3cb83fe732982b112f5f.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[https_pan.baidu.com_s_1TBVEyndXVHcMbxst-kpCuA]: https://pan.baidu.com/s/1TBVEyndXVHcMbxst-kpCuA
[https_pan.baidu.com_s_1oO_mC6J3SV6PLVepjmi2eQ]: https://pan.baidu.com/s/1oO_mC6J3SV6PLVepjmi2eQ
[https_pan.baidu.com_s_1xoT3Px14RHu6QE-66nLtMQ]: https://pan.baidu.com/s/1xoT3Px14RHu6QE-66nLtMQ
[5]: https://s1.51cto.com/images/blog/201809/26/22d0bea513ad6c9c71291c818d17f7f2.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[5 1]: https://s1.51cto.com/images/blog/201809/26/caef411894e5881bec40d8f267b7fe6b.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[6]: https://s1.51cto.com/images/blog/201809/26/246857e13ba80f973e61d80a80eec8a6.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[6 1]: https://s1.51cto.com/images/blog/201809/26/adfdfb79193382efb61e1221ac34f6f0.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[7]: https://s1.51cto.com/images/blog/201809/26/a9dec821923e2d3d04e9e1d9ecad5734.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[7 1]: https://s1.51cto.com/images/blog/201809/26/0cd61ae6ca4f3c5a21d9537c230be221.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[30]: https://s1.51cto.com/images/blog/201809/26/4af309de2376203df74259813478e5e7.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[30 1]: https://s1.51cto.com/images/blog/201809/26/a94204f0578fb301f5ca74d0eeffdb6a.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[31]: https://s1.51cto.com/images/blog/201809/26/48fd79a4bfdc07b89ea037e4155eb91d.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[31 1]: https://s1.51cto.com/images/blog/201809/26/ce7474da2a41a064eff7b1ddef816199.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[32]: https://s1.51cto.com/images/blog/201809/26/54b835d5d487452d5f3dd7efe5c428fa.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[32 1]: https://s1.51cto.com/images/blog/201809/26/e7dc431bd45d4ec6e605d6b3b006b20d.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=