Redis-4.0.11集群配置

本是古典何须时尚 2022-05-13 07:20 611阅读 0赞

版本：redis-3.0.5 redis-3.2.0  redis-3.2.9  redis-4.0.11

参考：[http://redis.io/topics/cluster-tutorial][http_redis.io_topics_cluster-tutorial]。

集群部署交互式命令行工具：[https://github.com/eyjian/redis-tools/tree/master/deploy][https_github.com_eyjian_redis-tools_tree_master_deploy]

集群运维命令行工具：[https://github.com/eyjian/redis-tools/tree/master][https_github.com_eyjian_redis-tools_tree_master]

批量操作工具：[https://github.com/eyjian/libmooon/releases][https_github.com_eyjian_libmooon_releases]

# **目录** #

目录 1

1. 前言 2

2. 部署计划 2

3. 目录结构 2

4. 编译安装 3

5. 修改系统参数 3

5.1. 修改最大可打开文件数 3

5.2. TCP监听队列大小 4

5.3. OOM相关：vm.overcommit\_memory 5

5.4. /sys/kernel/mm/transparent\_hugepage/enabled 5

6. 配置redis 5

7. 启动redis实例 8

8. 创建和启动redis cluster前的准备工作 9

8.1. 安装ruby 9

8.2. 安装rubygems 9

8.3. 安装redis-3.0.0.gem 9

9. redis-trib.rb 10

10. 创建和启动redis集群 10

10.1. 复制redis-trib.rb 10

10.2. 创建redis cluster 11

10.3. ps aux|grep redis 12

11. redis cluster client 13

11.1. 命令行工具redis-cli 13

11.2. 从slaves读数据 13

11.3. jedis（java cluster client） 13

11.4. r3c（C++ cluster client） 14

12. 新增节点 14

12.1. 添加一个新主（master）节点 14

12.2. 添加一个新从（slave）节点 15

13. 删除节点 15

14. master机器硬件故障 17

15. 检查节点状态 17

16. 变更主从关系 17

17. slots相关命令 17

17.1. 迁移slosts 18

17.2. redis-trib.rb rebalance 18

18. 人工主备切换 18

19. 查看集群信息 19

20. 禁止指定命令 20

21. 各版本配置文件 20

22. 大压力下Redis参数调整要点 20

23. 问题排查 22

# **1. 前言** #

本文参考官方文档而成：[http://redis.io/topics/cluster-tutorial][http_redis.io_topics_cluster-tutorial]。经测试，安装过程也适用于redis-3.2.0、redis-4.0.11等。

Redis运维工具和部署工具：[https://github.com/eyjian/redis-tools][https_github.com_eyjian_redis-tools]。

# **2. 部署计划** #

依据官网介绍，部署6个redis节点，为3主3从。3台物理机每台都创建2个redis节点：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:114.65pt;"> 服务端口 </td> 
 <td style="vertical-align:top;width:127.2pt;"> IP地址 </td> 
 <td style="vertical-align:top;width:184.25pt;"> 配置文件名 </td> 
 </tr> 
 <tr> 
 <td style="background-color:#99cdff;width:114.65pt;"> 6379 </td> 
 <td style="background-color:#99cdff;width:127.2pt;"> 192.168.0.251 </td> 
 <td style="background-color:#99cdff;width:184.25pt;"> redis-6379.conf </td> 
 </tr> 
 <tr> 
 <td style="background-color:#99cdff;width:114.65pt;"> 6379 </td> 
 <td style="background-color:#99cdff;width:127.2pt;"> 192.168.0.252 </td> 
 <td style="background-color:#99cdff;width:184.25pt;"> redis-6379.conf </td> 
 </tr> 
 <tr> 
 <td style="background-color:#99cdff;width:114.65pt;"> 6379 </td> 
 <td style="background-color:#99cdff;width:127.2pt;"> 192.168.0.253 </td> 
 <td style="background-color:#99cdff;width:184.25pt;"> redis-6379.conf </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:114.65pt;"> 6380 </td> 
 <td style="vertical-align:top;width:127.2pt;"> 192.168.0.251 </td> 
 <td style="vertical-align:top;width:184.25pt;"> redis-6380.conf </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:114.65pt;"> 6380 </td> 
 <td style="vertical-align:top;width:127.2pt;"> 192.168.0.252 </td> 
 <td style="vertical-align:top;width:184.25pt;"> redis-6380.conf </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:114.65pt;"> 6380 </td> 
 <td style="vertical-align:top;width:127.2pt;"> 192.168.0.253 </td> 
 <td style="vertical-align:top;width:184.25pt;"> redis-6380.conf </td> 
 </tr> 
 </tbody> 
</table>

***疑问：3台物理机，会不会主和从节点分布在同一个物理机上？***

# **3. 目录结构** #

redis.conf为从[https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf][https_raw.githubusercontent.com_antirez_redis_3.0_redis.conf]下载的配置文件。redis-6379.conf和redis-6380.conf指定了服务端口，两者均通过include复用（包含）了redis.conf。

本文将redis安装在**/data/redis**（每台机器完全相同，同一台机器上的多个节点对应相同的目录和文件，并建议将bin目录加入到环境变量PATH中，以简化后续的使用）：

# **4. 编译安装** #

打开redis的Makefile文件，可以看到如下内容：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> PREFIX?=/usr/local INSTALL_BIN=$(PREFIX)/bin INSTALL=install </td> 
 </tr> 
 </tbody> 
</table>

Makefile中的“**?=**”表示，如果该变量之前没有定义过，则赋值为/usr/local，否则什么也不做。

如果不设置环境变量PREFIX或不修改Makefile中的值，则默认安装到/usr/local/bin目录下。建议不要使用默认配置，而是指定安装目录，如/data/redis-3.0.5：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;make $&nbsp;make&nbsp;install&nbsp;PREFIX=/data/redis-3.0.5 $&nbsp;ln&nbsp;-s&nbsp;/data/redis-3.0.5&nbsp;/data/redis $&nbsp;mkdir&nbsp;/data/redis/conf $&nbsp;mkdir&nbsp;/data/redis/log $&nbsp;mkdir&nbsp;/data/redis/data </td> 
 </tr> 
 </tbody> 
</table>

# **5. 修改系统参数** #

## **5.1. 修改最大可打开文件数** ##

修改文件/etc/security/limits.conf，加入以下两行：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> *&nbsp;soft&nbsp;nofile&nbsp;102400 *&nbsp;hard&nbsp;nofile&nbsp;102400 &nbsp; #&nbsp;End&nbsp;of&nbsp;file </td> 
 </tr> 
 </tbody> 
</table>

其中**102400**为一个进程最大可以打开的文件个数，当与RedisServer的连接数多时，需要设定为合适的值。

有些环境修改后，root用户需要重启机器才生效，而普通用户重新登录后即生效。如果是crontab，则需要重启crontab，如：service crond restart，有些平台可能是service cron restart。

有些环境下列设置即可让root重新登录即生效，而不用重启机器：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> root&nbsp;soft&nbsp;nofile&nbsp;102400 root&nbsp;hard&nbsp;nofile&nbsp;102400 &nbsp; #&nbsp;End&nbsp;of&nbsp;file </td> 
 </tr> 
 </tbody> 
</table>

但是要**小心**，有些环境上面这样做，可能导致无法ssh登录，所以在修改时最好打开两个窗口，万一登录不了还可自救。

如何确认更改对一个进程生效？按下列方法（其中$PID为被查的进程ID）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;cat&nbsp;/proc/$PID/limits </td> 
 </tr> 
 </tbody> 
</table>

系统关于/etc/security/limits.conf文件的说明：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> #This&nbsp;file&nbsp;sets&nbsp;the&nbsp;resource&nbsp;limits&nbsp;for&nbsp;the&nbsp;users&nbsp;logged&nbsp;in&nbsp;via&nbsp;PAM. #It&nbsp;does&nbsp;not&nbsp;affect&nbsp;resource&nbsp;limits&nbsp;of&nbsp;the&nbsp;system&nbsp;services. </td> 
 </tr> 
 </tbody> 
</table>

PAM：全称“Pluggable Authentication Modules”，中文名“插入式认证模块”。/etc/security/limits.conf实际为**pam\_limits.so**（位置：/lib/security/pam\_limits.so）的配置文件，只针对单个会话。要使用limits.conf生效，必须保证pam\_limits.so被加入到了启动文件中。

注释说明只对通过**PAM**登录的用户生效，与PAM相关的文件（均位于/etc/**pam.d**目录下）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> /etc/pam.d/login /etc/pam.d/sshd /etc/pam.d/crond </td> 
 </tr> 
 </tbody> 
</table>

如果需要设置Linux用户的密码策略，可以修改文件/etc/login.defs，但这个只对新增的用户有效，如果要影响已有用户，可使用命令chage。

## **5.2. TCP监听队列大小** ##

即TCP listen的backlog大小，“/proc/sys/net/core/**somaxconn**”的默认值一般较小如128，需要修改大一点，比如改成32767。立即生效还可以使用命令：**sysctl** **-w** net.core.somaxconn=32767。

要想永久生效，需要在文件/etc/sysctl.conf中增加一行：net.core.somaxconn = 32767，然后执行命令“sysctl -p”以生效。

Redis配置项**tcp-backlog**的值不能超过**somaxconn**的大小。

## **5.3. OOM相关：vm.overcommit\_memory** ##

如果“/proc/sys/vm/overcommit\_memory”的值为0，则会表示开启了OOM。可以设置为1关闭OOM，设置方法请参照net.core.somaxconn完成。

## **5.4. /sys/kernel/mm/transparent\_hugepage/enabled** ##

默认值为“\[always\] madvise never”，建议设置为never，以开启内核的“Transparent Huge Pages (**THP**)”特性，设置后redis进程需要重启。为了永久生效，请将“echo never > /sys/kernel/mm/transparent\_hugepage/enabled”加入到文件/etc/rc.local中。

什么是Transparent Huge Pages？为提升性能，通过大内存页来替代传统的4K页，使用得管理虚拟地址数变少，加快从虚拟地址到物理地址的映射，以及摒弃内存页面的换入换出以提高内存的整体性能。内核Kernel将程序缓存内存中，每页内存以2M为单位。相应的系统进程为khugepaged。

在Linux中，有两种方式使用Huge Pages，一种是2.6内核引入的HugeTLBFS，另一种是2.6.36内核引入的THP。HugeTLBFS主要用于数据库，THP广泛应用于应用程序。

一般可以在rc.local或/etc/default/grub中对Huge Pages进行设置。

# **6. 配置redis** #

从[https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf][https_raw.githubusercontent.com_antirez_redis_3.0_redis.conf]下载配置文件（也可直接复制源代码包中的redis.conf，然后在它的基础上进行修改），在这个基础上，进行如下表所示的修改（配置文件名redis-**6379**.conf中的**6379**建议设置为实际使用的端口号）：

<table style="width:426.05pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> 配置项 </td> 
 <td style="vertical-align:top;width:130.8pt;"> 值 </td> 
 <td style="vertical-align:top;width:81.6pt;"> 配置文件 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 说明 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> include </td> 
 <td style="vertical-align:top;width:130.8pt;"> redis.conf </td> 
 <td style="width:81.6pt;"> redis-6379.conf </td> 
 <td style="vertical-align:top;width:119.4pt;"> 引用公共的配置文件，建议为全路径值 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> port </td> 
 <td style="vertical-align:top;width:130.8pt;"> 6379 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 客户端连接端口，并且总有一个刚好大于10000的端口，这个大的端口用于主从复制和集群内部通讯。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-config-file </td> 
 <td style="vertical-align:top;width:130.8pt;"> nodes-6379.conf </td> 
 <td style="vertical-align:top;width:119.4pt;"> 默认放在dir指定的目录 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> pidfile </td> 
 <td style="vertical-align:top;width:130.8pt;"> /var/run/redis-6379.pid </td> 
 <td style="vertical-align:top;width:119.4pt;"> 只有当daemonize值为yes时，才有意义；并且这个要求对目录/var/run有写权限，否则可以考虑设置为/tmp/redis-6379.pid。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> dir </td> 
 <td style="vertical-align:top;width:130.8pt;"> /data/redis/data/6379 </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> dbfilename </td> 
 <td style="vertical-align:top;width:130.8pt;"> dump-6379.rdb </td> 
 <td style="vertical-align:top;width:119.4pt;"> 位于dir指定的目录下 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> appendonly </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> appendfilename </td> 
 <td style="vertical-align:top;width:130.8pt;"> "appendonly-6379.aof" </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> logfile </td> 
 <td style="vertical-align:top;width:130.8pt;"> /data/redis/log/redis-6379.log </td> 
 <td style="vertical-align:top;width:119.4pt;"> 日志文件，包含目录和文件名，注意redis不会自动滚动日志文件 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> include </td> 
 <td style="vertical-align:top;width:130.8pt;"> redis.conf </td> 
 <td style="width:81.6pt;"> redis-6380.conf </td> 
 <td style="vertical-align:top;width:119.4pt;"> 引用公共的配置文件 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> port </td> 
 <td style="vertical-align:top;width:130.8pt;"> 6380 </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-config-file </td> 
 <td style="vertical-align:top;width:130.8pt;"> nodes-6380.conf </td> 
 <td style="vertical-align:top;width:119.4pt;"> 默认放在dir指定的目录 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> pidfile </td> 
 <td style="vertical-align:top;width:130.8pt;"> /var/run/redis-6380.pid </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> dir </td> 
 <td style="vertical-align:top;width:130.8pt;"> /data/redis/data/6380 </td> 
 <td style="vertical-align:top;width:119.4pt;"> AOF和RDB文件存放目录 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> dbfilename </td> 
 <td style="vertical-align:top;width:130.8pt;"> dump-6380.rdb </td> 
 <td style="vertical-align:top;width:119.4pt;"> RDB文件名 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> appendfilename </td> 
 <td style="vertical-align:top;width:130.8pt;"> appendonly-6380.aof </td> 
 <td style="vertical-align:top;width:119.4pt;"> AOF文件名 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> logfile </td> 
 <td style="vertical-align:top;width:130.8pt;"> /data/redis/log/redis-6380.log </td> 
 <td style="vertical-align:top;width:119.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> loglevel </td> 
 <td style="vertical-align:top;width:130.8pt;"> verbose </td> 
 <td style="vertical-align:top;width:81.6pt;"> redis.conf （公共配置文件） </td> 
 <td style="vertical-align:top;width:119.4pt;"> 日志级别，建议为notice，另外注意redis是不会滚动日志文件的，每次写日志都是先打开日志文件再写日志再关闭方式 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> maxclients </td> 
 <td style="vertical-align:top;width:130.8pt;"> 10000 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 最大连接数 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> timeout </td> 
 <td style="vertical-align:top;width:130.8pt;"> 0 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 客户端多长（秒）时间没发包过来关闭它，0表示永不关闭 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-enabled </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 表示以集群方式运行，为no表示以非集群方式运行 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-node-timeout </td> 
 <td style="vertical-align:top;width:130.8pt;"> 15000 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 单位为毫秒： repl-ping-slave-period+ (cluster-node-timeout* cluster-slave-validity-factor) 判断节点失效（fail）之前，允许不可用的最大时长（毫秒），如果master不可用时长超过此值，则会被failover。不能太小，建议默认值15000 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-slave-validity-factor </td> 
 <td style="vertical-align:top;width:130.8pt;"> 0 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 如果要最大的可用性，值设置为0。定义slave和master失联时长的倍数，如果值为0，则只要失联slave总是尝试failover，而不管与master失联多久。失联最大时长：(cluster-slave-validity-factor*cluster-node-timeout) </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> repl-timeout </td> 
 <td style="vertical-align:top;width:130.8pt;"> 10 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 该配置项的值要求大于repl-ping-slave-period的值 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> repl-ping-slave-period </td> 
 <td style="vertical-align:top;width:130.8pt;"> 1 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 定义slave多久（秒）ping一次master，如果超过repl-timeout指定的时长都没有收到响应，则认为master挂了 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> slave-read-only </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> slave是否只读 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> slave-serve-stale-data </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 当slave与master断开连接，slave是否继续提供服务 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> slave-priority </td> 
 <td style="vertical-align:top;width:130.8pt;"> 100 </td> 
 <td style="vertical-align:top;width:119.4pt;"> slave权重值，当master挂掉，只有权重最大的slave接替master </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> aof-use-rdb-preamble </td> 
 <td style="vertical-align:top;width:130.8pt;"> &nbsp; </td> 
 <td style="vertical-align:top;width:119.4pt;"> 4.0新增配置项，用于控制是否启用RDB-AOF混用，值为no表示关闭 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> appendonly </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 当同时写AOF或RDB，则redis启动时只会加载AOF，AOF包含了全量数据。如果当队列使用，入队压力又很大，建议设置为no </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> appendfsync </td> 
 <td style="vertical-align:top;width:130.8pt;"> no </td> 
 <td style="vertical-align:top;width:119.4pt;"> 可取值everysec，其中no表示由系统自动，当写压力很大时，建议设置为no，否则容易造成整个集群不可用 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> daemonize </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 相关配置项pidfile </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> protected-mode </td> 
 <td style="vertical-align:top;width:130.8pt;"> no </td> 
 <td style="vertical-align:top;width:119.4pt;"> 3.2.0新增的配置项，默认值为yes，限制从其它机器登录Redis&nbsp;server，而只能从127.0.0.1登录。为保证redis-trib.rb工具的正常运行，需要设置为no，完成后可以改回yes，但每次使用redis-trib.rb都需要改回为no。要想从非127.0.0.1访问也需要改为no。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> tcp-backlog </td> 
 <td style="vertical-align:top;width:130.8pt;"> 32767 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 取值不能超过系统的/proc/sys/net/core/somaxconn </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> auto-aof-rewrite-percentage </td> 
 <td style="vertical-align:top;width:130.8pt;"> 100 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 设置自动rewite&nbsp;AOF文件（手工rewrite只需要调用命令BGREWRITEAOF） </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> auto-aof-rewrite-min-size </td> 
 <td style="vertical-align:top;width:130.8pt;"> 64mb </td> 
 <td style="vertical-align:top;width:119.4pt;"> 触发rewrite的AOF文件大小，只有大于此大小时才会触发rewrite </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> no-appendfsync-on-rewrite </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 子进程在做rewrite时，主进程不调用fsync（由内核默认调度） </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> stop-writes-on-bgsave-error </td> 
 <td style="vertical-align:top;width:130.8pt;"> yes </td> 
 <td style="vertical-align:top;width:119.4pt;"> 如果因为磁盘故障等导致保存rdb失败，停止写操作，可设置为NO。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-require-full-coverage </td> 
 <td style="vertical-align:top;width:130.8pt;"> no </td> 
 <td style="vertical-align:top;width:119.4pt;"> 为no表示有slots不可服务时其它slots仍然继续服务 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> maxmemory </td> 
 <td style="vertical-align:top;width:130.8pt;"> 26843545600 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 设置最大的内存，单位为字节 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> maxmemory-policy </td> 
 <td style="vertical-align:top;width:130.8pt;"> volatile-lru </td> 
 <td style="vertical-align:top;width:119.4pt;"> 设置达到最大内存时的淘汰策略 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> client-output-buffer-limit </td> 
 <td style="vertical-align:top;width:130.8pt;"> &nbsp; </td> 
 <td style="vertical-align:top;width:119.4pt;"> 设置master端的客户端缓存，三种：normal、slave和pubsub </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> cluster-migration-barrier </td> 
 <td style="vertical-align:top;width:130.8pt;"> 1 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 最少slave数，用来保证集群中不会有裸奔的master。当某个master节点的slave节点挂掉裸奔后，会从其他富余的master节点分配一个slave节点过来，确保每个master节点都有至少一个slave节点，不至于因为master节点挂掉而没有相应slave节点替换为master节点导致集群崩溃不可用。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> repl-backlog-size </td> 
 <td style="vertical-align:top;width:130.8pt;"> 1mb </td> 
 <td style="vertical-align:top;width:119.4pt;"> 当slave失联时的，环形复制缓区大小，值越大可容忍更长的slave失联时长 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> repl-backlog-ttl </td> 
 <td style="vertical-align:top;width:130.8pt;"> &nbsp; </td> 
 <td style="vertical-align:top;width:119.4pt;"> slave失联的时长达到该值时，释放backlog缓冲区 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:94.25pt;"> save </td> 
 <td style="vertical-align:top;width:130.8pt;"> save&nbsp;900&nbsp;1 save&nbsp;300&nbsp;10 save&nbsp;60&nbsp;10000 </td> 
 <td style="vertical-align:top;width:119.4pt;"> 刷新快照（RDB）到磁盘的策略，根据实际调整值，“save&nbsp;900&nbsp;1”表示900秒后至少有1个key被修改才触发save操作，其它类推。 注意执行flushall命令也会产生RDB文件，不过是空文件。 如果不想生成RDB文件，可以将save全注释掉。 </td> 
 </tr> 
 </tbody> 
</table>

# **7. 启动redis实例** #

登录3台物理机，启动两个redis实例（启动之前，需要创建好配置中的各目录）：

1) redis-server redis-**6379**.conf

2) redis-server redis-**6380**.conf

可以写一个启动脚本start-redis-cluster.sh：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> #!/bin/sh &nbsp; REDIS_HOME=/data/redis $REDIS_HOME/bin/redis-server&nbsp;$REDIS_HOME/conf/redis-6379.conf $REDIS_HOME/bin/redis-server&nbsp;$REDIS_HOME/conf/redis-6380.conf </td> 
 </tr> 
 </tbody> 
</table>

# **8. 创建和启动redis cluster前的准备工作** #

上一步启动的redis只是单机版本，在启动redis cluster之前，需要完成如下一些依赖的安装。在此之后，才可以创建和启动redis cluster。

## **8.1. 安装ruby** ##

安装命令：yum install ruby

安装过程中，如提示“\[**y**/d/N\]”，请选“**y**”然后回车。

查看版本：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;ruby&nbsp;--version ruby&nbsp;2.0.0p353&nbsp;(2013-11-22)&nbsp;[x86_64-linux] </td> 
 </tr> 
 </tbody> 
</table>

也可以从Ruby官网[https://www.ruby-lang.org][https_www.ruby-lang.org]下载安装包（如[ruby-2.3.1.tar.gz][]）来安装Ruby。截至2016/5/13，Ruby的最新稳定版本为Ruby 2.3.1。

## **8.2. 安装rubygems** ##

安装命令：yum install rubygems

如果不使用yum安装，也可以手动安装RubyGems，RubyGems是一个Ruby包管理框架，它的下载网址：[https://rubygems.org/pages/download][https_rubygems.org_pages_download]。

比如下载安装包rubygems-2.6.4.zip后解压，然后进入解压生成的目录，里面有个setup.rb文件，以root用户执行：ruby setup.rb安装RubyGems。

## **8.3. 安装redis-3.0.0.gem** ##

安装命令：gem install -l redis-3.0.0.gem

安装之前，需要先下载好redis-3.0.0.gem。

redis-3.0.0.gem官网：[https://rubygems.org/gems/redis/versions/3.0.0][https_rubygems.org_gems_redis_versions_3.0.0]

redis-3.0.0.gem下载网址：[https://rubygems.org/downloads/redis-3.0.0.gem][https_rubygems.org_downloads_redis-3.0.0.gem]

redis-3.3.0.gem官网：[https://rubygems.org/gems/redis/versions/3.3.0][https_rubygems.org_gems_redis_versions_3.3.0]

redis-3.3.3.gem官网：[https://rubygems.org/gems/redis/versions/3.3.3][https_rubygems.org_gems_redis_versions_3.3.3]

redis-4.0.1.gem官网：[https://rubygems.org/gems/redis/versions/4.0.1][https_rubygems.org_gems_redis_versions_4.0.1]

集群的创建只需要在一个节点上操作，所以只需要在一个节点上安装redis-X.X.X.gem即可。

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> #&nbsp;gem&nbsp;install&nbsp;-l&nbsp;redis-3.3.3.gem Successfully&nbsp;installed&nbsp;redis-3.3.3 Parsing&nbsp;documentation&nbsp;for&nbsp;redis-3.3.3 Installing&nbsp;ri&nbsp;documentation&nbsp;for&nbsp;redis-3.3.3 Done&nbsp;installing&nbsp;documentation&nbsp;for&nbsp;redis&nbsp;after&nbsp;1&nbsp;seconds 1&nbsp;gem&nbsp;installed </td> 
 </tr> 
 </tbody> 
</table>

# **9. redis-trib.rb** #

redis-trib.rb是redis官方提供的redis cluster管理工具，使用ruby实现。

# **10. 创建和启动redis集群** #

## **10.1. 复制redis-trib.rb** ##

将redis**源代码**的src目录下的**集群管理程序****redis-trib.rb**复制到/data/redis/bin目录，并将bin目录加入到环境变量PATH中，以简化后续的操作。

redis-trib.rb用法（不带任何参数执行redis-trib.rb即显示用法）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;./redis-trib.rb Usage:&nbsp;redis-trib&nbsp;&lt;command&gt;&nbsp;&lt;options&gt;&nbsp;&lt;arguments&nbsp;...&gt; &nbsp; &nbsp;&nbsp;rebalance&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--auto-weights &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--timeout&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--pipeline&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--use-empty-masters &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--weight&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--threshold&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--simulate &nbsp;&nbsp;add-node&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;new_host:new_port&nbsp;existing_host:existing_port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--slave &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--master-id&nbsp;&lt;arg&gt; &nbsp;&nbsp;reshard&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--timeout&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--pipeline&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--yes &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--slots&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--to&nbsp;&lt;arg&gt; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--from&nbsp;&lt;arg&gt; &nbsp;&nbsp;check&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;set-timeout&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port&nbsp;milliseconds &nbsp;&nbsp;call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port&nbsp;command&nbsp;arg&nbsp;arg&nbsp;..&nbsp;arg &nbsp;&nbsp;fix&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--timeout&nbsp;&lt;arg&gt; &nbsp;&nbsp;info&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;create&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host1:port1&nbsp;...&nbsp;hostN:portN &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--replicas&nbsp;&lt;arg&gt; &nbsp;&nbsp;import&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--replace &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--copy &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;--from&nbsp;&lt;arg&gt; &nbsp;&nbsp;help&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(show&nbsp;this&nbsp;help) &nbsp;&nbsp;del-node&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host:port&nbsp;node_id &nbsp; For&nbsp;check,&nbsp;fix,&nbsp;reshard,&nbsp;del-node,&nbsp;set-timeout&nbsp;you&nbsp;can&nbsp;specify&nbsp;the&nbsp;host&nbsp;and&nbsp;port&nbsp;of&nbsp;any&nbsp;working&nbsp;node&nbsp;in&nbsp;the&nbsp;cluster. </td> 
 </tr> 
 </tbody> 
</table>

## **10.2. 创建redis cluster** ##

创建命令（3主3从）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> redis-trib.rb&nbsp;create&nbsp;--replicas&nbsp;1&nbsp;192.168.0.251:6379&nbsp;192.168.0.252:6379&nbsp;192.168.0.253:6379&nbsp;192.168.0.251:6380&nbsp;192.168.0.252:6380&nbsp;192.168.0.253:6380 </td> 
 </tr> 
 </tbody> 
</table>

**Ø ****参数说明：**

**1) ****create**

表示创建一个redis cluster集群。

**2) ****--replicas ****1**

表示为集群中的每一个主节点指定一个从节点，即一比一的复制。\\

运行过程中，会有个提示，输入**yes**回车即可。从屏幕输出，可以很容易地看出哪些是主（master）节点，哪些是从（slave）节点：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> &gt;&gt;&gt;&nbsp;Creating&nbsp;cluster Connecting&nbsp;to&nbsp;node&nbsp;192.168.0.251:6379:&nbsp;OK /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182:&nbsp;warning:&nbsp;wrong&nbsp;element&nbsp;type&nbsp;nil&nbsp;at&nbsp;0&nbsp;(expected&nbsp;array) /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182:&nbsp;warning:&nbsp;ignoring&nbsp;wrong&nbsp;elements&nbsp;is&nbsp;deprecated,&nbsp;remove&nbsp;them&nbsp;explicitly /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182:&nbsp;warning:&nbsp;this&nbsp;causes&nbsp;ArgumentError&nbsp;in&nbsp;the&nbsp;next&nbsp;release &gt;&gt;&gt;&nbsp;Performing&nbsp;hash&nbsp;slots&nbsp;allocation&nbsp;on&nbsp;6&nbsp;nodes... Using&nbsp;3&nbsp;masters: 192.168.0.251:6379 192.168.0.252:6379 192.168.0.253:6379 Adding&nbsp;replica&nbsp;192.168.0.252:6380&nbsp;to&nbsp;192.168.0.251:6379 Adding&nbsp;replica&nbsp;192.168.0.251:6380&nbsp;to&nbsp;192.168.0.252:6379 Adding&nbsp;replica&nbsp;192.168.0.253:6380&nbsp;to&nbsp;192.168.0.253:6379 M:&nbsp;150f77d1000003811fb3c38c3768526a0b25ec31&nbsp;192.168.0.251:6379 &nbsp;&nbsp;&nbsp;slots:0-5460&nbsp;(5461&nbsp;slots)&nbsp;master M:&nbsp;de461d3337b17d2119b79024d57d8b119e7320a6&nbsp;192.168.0.252:6379 &nbsp;&nbsp;&nbsp;slots:5461-10922&nbsp;(5462&nbsp;slots)&nbsp;master M:&nbsp;faf50658fb7b0bae64cee5371da782e0f4919eee&nbsp;192.168.0.253:6379 &nbsp;&nbsp;&nbsp;slots:10923-16383&nbsp;(5461&nbsp;slots)&nbsp;master S:&nbsp;c567db02cc40eebf577f71f703214dd2f4f26dfb&nbsp;192.168.0.251:6380 &nbsp;&nbsp;&nbsp;replicates&nbsp;de461d3337b17d2119b79024d57d8b119e7320a6 S:&nbsp;284f8196b250ad9ac272316db84a07bebf661ab7&nbsp;192.168.0.252:6380 &nbsp;&nbsp;&nbsp;replicates&nbsp;150f77d1000003811fb3c38c3768526a0b25ec31 S:&nbsp;39fdef9fd5778dc94d8add819789d7d73ca06899&nbsp;192.168.0.253:6380 &nbsp;&nbsp;&nbsp;replicates&nbsp;faf50658fb7b0bae64cee5371da782e0f4919eee Can&nbsp;I&nbsp;set&nbsp;the&nbsp;above&nbsp;configuration?&nbsp;(type&nbsp;'yes'&nbsp;to&nbsp;accept):&nbsp;yes &gt;&gt;&gt;&nbsp;Nodes&nbsp;configuration&nbsp;updated &gt;&gt;&gt;&nbsp;Assign&nbsp;a&nbsp;different&nbsp;config&nbsp;epoch&nbsp;to&nbsp;each&nbsp;node &gt;&gt;&gt;&nbsp;Sending&nbsp;CLUSTER&nbsp;MEET&nbsp;messages&nbsp;to&nbsp;join&nbsp;the&nbsp;cluster Waiting&nbsp;for&nbsp;the&nbsp;cluster&nbsp;to&nbsp;join.... &gt;&gt;&gt;&nbsp;Performing&nbsp;Cluster&nbsp;Check&nbsp;(using&nbsp;node&nbsp;192.168.0.251:6379) M:&nbsp;150f77d1000003811fb3c38c3768526a0b25ec31&nbsp;192.168.0.251:6379 &nbsp;&nbsp;&nbsp;slots:0-5460&nbsp;(5461&nbsp;slots)&nbsp;master M:&nbsp;de461d3337b17d2119b79024d57d8b119e7320a6&nbsp;192.168.0.252:6379 &nbsp;&nbsp;&nbsp;slots:5461-10922&nbsp;(5462&nbsp;slots)&nbsp;master M:&nbsp;faf50658fb7b0bae64cee5371da782e0f4919eee&nbsp;192.168.0.253:6379 &nbsp;&nbsp;&nbsp;slots:10923-16383&nbsp;(5461&nbsp;slots)&nbsp;master M:&nbsp;c567db02cc40eebf577f71f703214dd2f4f26dfb&nbsp;192.168.0.251:6380 &nbsp;&nbsp;&nbsp;slots:&nbsp;(0&nbsp;slots)&nbsp;master &nbsp;&nbsp;&nbsp;replicates&nbsp;de461d3337b17d2119b79024d57d8b119e7320a6 M:&nbsp;284f8196b250ad9ac272316db84a07bebf661ab7&nbsp;192.168.0.252:6380 &nbsp;&nbsp;&nbsp;slots:&nbsp;(0&nbsp;slots)&nbsp;master &nbsp;&nbsp;&nbsp;replicates&nbsp;150f77d1000003811fb3c38c3768526a0b25ec31 M:&nbsp;39fdef9fd5778dc94d8add819789d7d73ca06899&nbsp;192.168.0.253:6380 &nbsp;&nbsp;&nbsp;slots:&nbsp;(0&nbsp;slots)&nbsp;master &nbsp;&nbsp;&nbsp;replicates&nbsp;faf50658fb7b0bae64cee5371da782e0f4919eee [OK]&nbsp;All&nbsp;nodes&nbsp;agree&nbsp;about&nbsp;slots&nbsp;configuration. &gt;&gt;&gt;&nbsp;Check&nbsp;for&nbsp;open&nbsp;slots... &gt;&gt;&gt;&nbsp;Check&nbsp;slots&nbsp;coverage... [OK]&nbsp;All&nbsp;16384&nbsp;slots&nbsp;covered. </td> 
 </tr> 
 </tbody> 
</table>

## **10.3. ps aux|grep redis** ##

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> [test@test-168-251&nbsp;~]$&nbsp;ps&nbsp;aux|grep&nbsp;redis test&nbsp;&nbsp;&nbsp;3824&nbsp;&nbsp;0.7&nbsp;&nbsp;5.9&nbsp;6742404&nbsp;3885144&nbsp;?&nbsp;&nbsp;&nbsp;&nbsp;Ssl&nbsp;&nbsp;&nbsp;2015&nbsp;1639:13&nbsp;/data/redis/bin/redis-server&nbsp;*:6379&nbsp;[cluster] test&nbsp;&nbsp;&nbsp;3831&nbsp;&nbsp;0.5&nbsp;&nbsp;3.9&nbsp;6709636&nbsp;2618536&nbsp;?&nbsp;&nbsp;&nbsp;&nbsp;Ssl&nbsp;&nbsp;&nbsp;2015&nbsp;1235:43&nbsp;/data/redis/bin/redis-server&nbsp;*:6380&nbsp;[cluster] </td> 
 </tr> 
 </tbody> 
</table>

停止redis实例，直接使用kill命令即可，如：kill **3831**，重启和单机版相同，经过上述一系列操作后，重启会自动转换成cluster模式。。

# **11. redis cluster client** #

## **11.1. 命令行工具redis-cli** ##

官方提供的命令行客户端工具，在单机版redis基础上指定参数“**-c**”即可。以下是在192.168.0.251上执行redis-cli的记录：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;./redis-cli&nbsp;-c&nbsp;-p&nbsp;6379 127.0.0.1:6379&gt;&nbsp;set&nbsp;foo&nbsp;bar -&gt;&nbsp;Redirected&nbsp;to&nbsp;slot&nbsp;[12182]&nbsp;located&nbsp;at&nbsp;192.168.0.253:6379 OK 192.168.0.253:6379&gt;&nbsp;set&nbsp;hello&nbsp;world -&gt;&nbsp;Redirected&nbsp;to&nbsp;slot&nbsp;[866]&nbsp;located&nbsp;at&nbsp;192.168.0.251:6379 OK 192.168.0.251:6379&gt;&nbsp;get&nbsp;foo -&gt;&nbsp;Redirected&nbsp;to&nbsp;slot&nbsp;[12182]&nbsp;located&nbsp;at&nbsp;192.168.0.253:6379 "bar" 192.168.0.253:6379&gt;&nbsp;get&nbsp;hello -&gt;&nbsp;Redirected&nbsp;to&nbsp;slot&nbsp;[866]&nbsp;located&nbsp;at&nbsp;192.168.0.251:6379 "world" &nbsp; 查看集群中的节点： 192.168.0.251:6379&gt;&nbsp;cluster&nbsp;nodes </td> 
 </tr> 
 </tbody> 
</table>

## **11.2. 从slaves读数据** ##

默认不能从slaves读取数据，但建立连接后，执行一次命令[READONLY][] ，即可从slaves读取数据。如果想再次恢复不能从slaves读取数据，可以执行下命令READWRITE。

## **11.3. jedis（java cluster client）** ##

官网：[https://github.com/xetorthio/jedis][https_github.com_xetorthio_jedis]

编程示例：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> Set&lt;HostAndPort&gt;&nbsp;jedisClusterNodes&nbsp;=&nbsp;new&nbsp;HashSet&lt;HostAndPort&gt;(); //Jedis&nbsp;Cluster&nbsp;will&nbsp;attempt&nbsp;to&nbsp;discover&nbsp;cluster&nbsp;nodes&nbsp;automatically jedisClusterNodes.add(new&nbsp;HostAndPort("127.0.0.1",&nbsp;7379)); JedisCluster&nbsp;jc&nbsp;=&nbsp;new&nbsp;JedisCluster(jedisClusterNodes); jc.set("foo",&nbsp;"bar"); String&nbsp;value&nbsp;=&nbsp;jc.get("foo"); </td> 
 </tr> 
 </tbody> 
</table>

## **11.4. r3c（C++ cluster client）** ##

官网：[https://github.com/eyjian/r3c][https_github.com_eyjian_r3c]

# **12. 新增节点** #

## **12.1. 添加一个新主（master）节点** ##

先以单机版配置和启动好redis-server，然后执行命令：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> ./redis-trib.rb&nbsp;add-node&nbsp;127.0.0.1:7006&nbsp;127.0.0.1:7000 </td> 
 </tr> 
 </tbody> 
</table>

执行上面这条命令时，可能遇到错误“\[ERR\] Sorry, can't connect to node 127.0.0.1:7006”。引起该问题的原因可能是因为ruby的版本过低（运行ruby -v可以查看ruby的版本），可以尝试升级ruby再尝试，比如ruby 1.8.7版本就需要升级。对于Redis 3.0.5和Redis 3.2.0，使用Ruby 2.3.1操作正常。请注意升级到最新版本的ruby也可能遇到这个错误。

另一个会引起这个问题的原因是从Redis 3.2.0版本开始引入了“保护模式（protected mode），防止redis-cli远程访问”，仅限redis-cli绑定到127.0.0.1才可以连接Redis server。

为了完成添加新主节点，可以暂时性的关闭保护模式，使用redis-cli，不指定-h参数（但可以指定-p参数，或者-h参数值为127.0.0.1）进入操作界面：**CONFIG SET** **protected-mode** **no**。

注意**7006**是新增的节点，而**7000**是已存在的节点（可为master或slave）。如果需要将7006变成某master的slave节点，执行命令：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> cluster&nbsp;replicate&nbsp;3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e </td> 
 </tr> 
 </tbody> 
</table>

新加入的master节点上没有任何数据（slots，运行redis命令cluster nodes可以看到这个情况）。当一个slave想成为master时，由于这个新的master节点不管理任何slots，它不参与选举。

可以使用工具redis-trib.rb的resharding特性为这个新master节点分配slots，如：

**redis-trib.rb** **reshard **127.0.0.1:**7000**，其中**7000**为集群中任意一个节点即可，redis-trib.rb将自动发现其它节点。

在reshard过程中，将会询问reshard多少slots：

How many **slots **do you want to move (from 1 to 16384)?，取值范围为1~16384，其中16384为redis cluster的拥有的slots总数，比如想只移动100个，输入100即可。如果迁移的slots数量多，应当设置redis-trib.rb的超时参数--timeout值大一点。否则，迁移过程中易遇到超时错误“\[ERR\] Calling MIGRATE: IOERR error or timeout reading to target instance”，导致只完成部分，可能会造成数据丢失。

接着，会提示“What is the receiving node ID?”，输入新加入的master节点ID。过程中如果遇到错误“Sorry, can't connect to node 10.225.168.253:6380”，则可能需要暂时先关闭相应的保护模式。

如果在迁移过程遇到下面这样的错误：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> &gt;&gt;&gt;&nbsp;Check&nbsp;for&nbsp;open&nbsp;slots... [WARNING]&nbsp;Node&nbsp;192.168.0.3:6379&nbsp;has&nbsp;slots&nbsp;in&nbsp;importing&nbsp;state&nbsp;(5461). [WARNING]&nbsp;Node&nbsp;192.168.0.5:6380&nbsp;has&nbsp;slots&nbsp;in&nbsp;migrating&nbsp;state&nbsp;(5461). [WARNING]&nbsp;The&nbsp;following&nbsp;slots&nbsp;are&nbsp;open:&nbsp;5461 </td> 
 </tr> 
 </tbody> 
</table>

可以考虑使用命令“redis-trib.rb fix 192.168.0.3:6379”尝试修复。需要显示有节点处于migrating或importing状态，可以登录到相应的节点，使用命令“cluster setslot **5461 **stable”修改，参数**5461**为问题显示的slot的ID。

## **12.2. 添加一个新从（slave）节点** ##

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> ./redis-trib.rb&nbsp;add-node&nbsp;--slave&nbsp;127.0.0.1:7006&nbsp;127.0.0.1:7000 </td> 
 </tr> 
 </tbody> 
</table>

注意这种方式，如果添加了多个slave节点，可能导致master的slaves不均衡，比如一些有3个slave，其它只1个slave。可以在slave节点上执行redis命令“CLUSTER REPLICATE”进行调整，让它成为其它master的slave。“CLUSTER REPLICATE”带一个参数，即master ID，注意使用redis-cli -c登录到slave上执行。

上面方法没有指定7006的master，而是随机指定。下面方法可以明确指定为哪个master的slave：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> ./redis-trib.rb&nbsp;add-node&nbsp;--slave&nbsp;--master-id&nbsp;3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e&nbsp;127.0.0.1:7006&nbsp;127.0.0.1:7000 </td> 
 </tr> 
 </tbody> 
</table>

其中“127.0.0.1:7006”是新节点，“127.0.0.1:7000”是集群中已有的节点。

# **13. 删除节点** #

从集群中删除一个节点：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> ./redis-trib.rb&nbsp;del-node&nbsp;127.0.0.1:7000&nbsp;&lt;node-id&gt; </td> 
 </tr> 
 </tbody> 
</table>

第一个参数为集群中任意一个节点，第二个参数为需要删除节点的ID。

成功删除后，提示：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $./redis-trib.rb&nbsp;del-node&nbsp;127.0.0.1:6380&nbsp;f49a2bda05e81aa343adb9924775ba95a1f4236e &gt;&gt;&gt;&nbsp;Removing&nbsp;node&nbsp;f49a2bda05e81aa343adb9924775ba95a1f4236e&nbsp;from&nbsp;cluster&nbsp;127.0.0.1:6379 /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182:&nbsp;warning:&nbsp;wrong&nbsp;element&nbsp;type&nbsp;nil&nbsp;at&nbsp;0&nbsp;(expected&nbsp;array) 。。。&nbsp;。。。 /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182:&nbsp;warning:&nbsp;this&nbsp;causes&nbsp;ArgumentError&nbsp;in&nbsp;the&nbsp;next&nbsp;release &gt;&gt;&gt;&nbsp;Sending&nbsp;CLUSTER&nbsp;FORGET&nbsp;messages&nbsp;to&nbsp;the&nbsp;cluster... &gt;&gt;&gt;&nbsp;SHUTDOWN&nbsp;the&nbsp;node.&nbsp;在这里会停顿几分钟，通知并等待被删除节点退出（exit），被删除节点在将数据写到RDB文件中后退出，所以停顿时长和写RDB文件时长有关，数据量越大时间就越长。5~6G的RAID1的SATA盘数据大概需要45秒左右。 </td> 
 </tr> 
 </tbody> 
</table>

被删除节点日志：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 15577:S&nbsp;06&nbsp;Sep&nbsp;20:06:37.774&nbsp;-&nbsp;Accepted&nbsp;10.49.126.98:14669 15577:S&nbsp;06&nbsp;Sep&nbsp;20:06:38.741&nbsp;#&nbsp;User&nbsp;requested&nbsp;shutdown... 15577:S&nbsp;06&nbsp;Sep&nbsp;20:06:38.741&nbsp;*&nbsp;Calling&nbsp;fsync()&nbsp;on&nbsp;the&nbsp;AOF&nbsp;file. 15577:S&nbsp;06&nbsp;Sep&nbsp;20:06:38.742&nbsp;*&nbsp;Saving&nbsp;the&nbsp;final&nbsp;RDB&nbsp;snapshot&nbsp;before&nbsp;exiting. 15577:S&nbsp;06&nbsp;Sep&nbsp;20:07:19.683&nbsp;*&nbsp;DB&nbsp;saved&nbsp;on&nbsp;disk 15577:S&nbsp;06&nbsp;Sep&nbsp;20:07:19.683&nbsp;*&nbsp;Removing&nbsp;the&nbsp;pid&nbsp;file. 15577:S&nbsp;06&nbsp;Sep&nbsp;20:07:19.683&nbsp;#&nbsp;Redis&nbsp;is&nbsp;now&nbsp;ready&nbsp;to&nbsp;exit,&nbsp;bye&nbsp;bye... </td> 
 </tr> 
 </tbody> 
</table>

成功后不用再调用“**CLUSTER ****FORGET**”，否则报错：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> $&nbsp;redis-cli&nbsp;-c&nbsp;CLUSTER&nbsp;FORGET&nbsp;aa6754a093ea4047f92cc0ea77f1859553bc5c57 (error)&nbsp;ERR&nbsp;Unknown&nbsp;node&nbsp;aa6754a093ea4047f92cc0ea77f1859553bc5c57 </td> 
 </tr> 
 </tbody> 
</table>

如果待删除节点已经不能连接，则调用**CLUSTER ****FORGET**剔除（可能需要在所有机器上执行一次FORGET）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> CLUSTER&nbsp;FORGET&nbsp;&lt;node-id&gt; </td> 
 </tr> 
 </tbody> 
</table>

注意如果是删除一个master节点，则需要先将它管理的slots的迁走，然后才可以删除它。

如果是master或slave机器不能连接，比如硬件故障导致无法启动，这个时候做不了**del-node**，只需要直接做**CLUSTER ** 即可，在FORGET后，节点状态变成handshake。

！！！请注意，需要在**所有node**上执行一次“**CLUSTER ****FORGET**”，否则可能遇到被剔除node的总是处于handshake状态。

如果有部分node没有执行到FORGET，导致有部分node还处于fail状态，则在一些node将看到待剔除节点仍然处于handshake状态，并且nodeid在不断变化，所以需要在所有node上执行“**CLUSTER ****FORGET**”。

如果一个节点处于“**:0 master,fail,noaddr**”状态，执行“**del-node**”会报错：

\[ERR\] No such node ID 80560d0d97a0b3fa975203350516437b58251745

这种情况下，只需要执行“**CLUSTER ****FORGET**”将其剔除即可（注意，需要在所有节点上执行一次，不然未执行的节点上可能仍然看得到“**:0 master,fail,noaddr**”）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> #&nbsp;redis-cli&nbsp;-c&nbsp;-p&nbsp;1383&nbsp;cluster&nbsp;nodes 80560d0d97a0b3fa975203350516437b58251745&nbsp;:0&nbsp;master,fail,noaddr&nbsp;-&nbsp;1528947205054&nbsp;1528947203553&nbsp;0&nbsp;disconnected fa7bbbf7d48389409ce05d303272078c3a6fd44f&nbsp;127.0.0.1:1379&nbsp;slave&nbsp;689f7c1ae71ea294c4ad7c5d1b32ae4e78e27915&nbsp;0&nbsp;1535871825187&nbsp;138&nbsp;connected c1a9d1d23438241803ec97fbd765737df80f402a&nbsp;127.0.0.1:1381&nbsp;slave&nbsp;f03b1008988acbb0f69d96252decda9adf747be9&nbsp;0&nbsp;1535871826189&nbsp;143&nbsp;connected 50003ccd5885771196e717e27011140e7d6c94e0&nbsp;127.0.0.1:1385&nbsp;slave&nbsp;f03b1008988acbb0f69d96252decda9adf747be9&nbsp;0&nbsp;1535871825688&nbsp;143&nbsp;connected f6080015129eada3261925cc1b466f1824263358&nbsp;127.0.0.1:1380&nbsp;slave&nbsp;4e932f2a3d80de29798660c5ea62e473e63a6630&nbsp;0&nbsp;1535871825388&nbsp;145&nbsp;connected 4e932f2a3d80de29798660c5ea62e473e63a6630&nbsp;127.0.0.1:1383&nbsp;myself,master&nbsp;-&nbsp;0&nbsp;0&nbsp;145&nbsp;connected&nbsp;5458-10922 689f7c1ae71ea294c4ad7c5d1b32ae4e78e27915&nbsp;127.0.0.1:1382&nbsp;master&nbsp;-&nbsp;0&nbsp;1535871826490&nbsp;138&nbsp;connected&nbsp;0-1986&nbsp;1988-5457 f03b1008988acbb0f69d96252decda9adf747be9&nbsp;127.0.0.1:1384&nbsp;master&nbsp;-&nbsp;0&nbsp;1535871825187&nbsp;143&nbsp;connected&nbsp;1987&nbsp;10923-16383 </td> 
 </tr> 
 </tbody> 
</table>

# **14. master机器硬件故障** #

这种情况下，master机器可能无法启动，导致其上的master无法连接，master将一直处于“master,fail”状态，如果是slave则处于“slave,fail”状态。

如果是master，则会它的slave变成了master，因此只需要添加一个新的从节点作为原slave（已变成master）的slave节点。完成后，通过**CLUSTER ****FORGET**将故障的master或slave从集群中剔除即可。

！！！请注意，需要在**所有node**上执行一次“**CLUSTER ****FORGET**”，否则可能遇到被剔除node的总是处于handshake状态。

# **15. 检查节点状态** #

redis-trib.rb check 127.0.0.1:6380

如发现如下这样的错误：

\[WARNING\] Node **1****92.168.0.11****:6380** has slots in migrating state (5461).

\[WARNING\] The following slots are open: **5461**

可以使用redis命令取消slots迁移（**5461**为slot的ID）：

cluster **setslot ****5461** stable

需要注意，须登录到**1****92.168.0.11****:6380**上执行redis的**setslot**子命令。

# **16. 变更主从关系** #

使用命令cluster replicate，参数为master节点ID，注意不是IP和端口，在被迁移的slave上执行该命令。

# **17. slots相关命令** #

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> CLUSTER&nbsp;ADDSLOTS&nbsp;slot1&nbsp;[slot2]&nbsp;...&nbsp;[slotN] CLUSTER&nbsp;DELSLOTS&nbsp;slot1&nbsp;[slot2]&nbsp;...&nbsp;[slotN] CLUSTER&nbsp;SETSLOT&nbsp;slot&nbsp;NODE&nbsp;node CLUSTER&nbsp;SETSLOT&nbsp;slot&nbsp;MIGRATING&nbsp;node CLUSTER&nbsp;SETSLOT&nbsp;slot&nbsp;IMPORTING&nbsp;node </td> 
 </tr> 
 </tbody> 
</table>

## **17.1. 迁移slosts** ##

官方参考：[https://redis.io/commands/cluster-setslot][https_redis.io_commands_cluster-setslot]。

示例：将值为8的slot从源节点A迁移到目标节点B，有如下两种方法：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 在目标节点B上执行：CLUSTER&nbsp;SETSLOT&nbsp;8&nbsp;IMPORTING&nbsp;src-A-node-id 或 在源节点A上执行：CLUSTER&nbsp;SETSLOT&nbsp;8&nbsp;MIGRATING&nbsp;dst-B-node-id </td> 
 </tr> 
 </tbody> 
</table>

上述操作只是将slot标记为迁移状态，完成迁移还需要执行（在目标node上执行）：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> CLUSTER&nbsp;SETSLOT&nbsp;&lt;slot&gt;&nbsp;NODE&nbsp;&lt;dst-node-id&gt; </td> 
 </tr> 
 </tbody> 
</table>

其中node-id为目标的Node ID，**取消迁移**使用“CLUSTER SETSLOT <slot> **STABLE**”。

操作示例：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> #&nbsp;将值为11677的slot迁到192.168.31.3:6379 $&nbsp;redis-cli&nbsp;-c&nbsp;-h&nbsp;192.168.31.3&nbsp;-p&nbsp;6379&nbsp;CLUSTER&nbsp;SETSLOT&nbsp;11677&nbsp;IMPORTING&nbsp;216e0069af11eca91465394b2ad7bf1c27f5f7fe OK $&nbsp;redis-cli&nbsp;-c&nbsp;-h&nbsp;192.168.31.3&nbsp;-p&nbsp;6379&nbsp;CLUSTER&nbsp;SETSLOT&nbsp;11677&nbsp;NODE&nbsp;4e149c72aff2b6651370ead476dd70c8cf9e3e3c OK </td> 
 </tr> 
 </tbody> 
</table>

## **17.2. redis-trib.rb rebalance** ##

当有增减节点时，可以使用命令：

redis-trib.rb rebalance 192.168.0.31:6379 --auto-weights

做一次均衡，简单点可以只指定两个参数：“192.168.0.31:6379”为集群中已知的任何一个节点，参数“-auto-weights”表示自动权重。

# **18. 人工主备切换** #

在需要的slaves节点上执行命令：CLUSTER **FAILOVER**。如果人工发起failover，则其它master会收到“Failover auth **granted **to 4291f18b5e9729e832ed15ceb6324ce5dfc2ffbe for **epoch **31”，每次**epoch**值增一。

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 23038:M&nbsp;06&nbsp;Sep&nbsp;20:31:24.815&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;4291f18b5e9729e832ed15ceb6324ce5dfc2ffbe&nbsp;for&nbsp;epoch&nbsp;31 &nbsp; 当出现下面两条日志时，表示failover完成： 23038:M&nbsp;06&nbsp;Sep&nbsp;20:32:44.019&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ea28f68438e5bb79c26a9cb2135241f11d7a50ba&nbsp;about&nbsp;5e6ffacb2c5d5761e39aba5270fbf48f296cb5ee 23038:M&nbsp;06&nbsp;Sep&nbsp;20:32:58.487&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;5e6ffacb2c5d5761e39aba5270fbf48f296cb5ee:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. </td> 
 </tr> 
 </tbody> 
</table>

**原master**收到failover后的日志：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:43.396&nbsp;-&nbsp;DB&nbsp;0:&nbsp;16870482&nbsp;keys&nbsp;(7931571&nbsp;volatile)&nbsp;in&nbsp;50331648&nbsp;slots&nbsp;HT. 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:43.396&nbsp;-&nbsp;1954&nbsp;clients&nbsp;connected&nbsp;(1&nbsp;slaves),&nbsp;5756515544&nbsp;bytes&nbsp;in&nbsp;use 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:48.083&nbsp;#&nbsp;Manual&nbsp;failover&nbsp;requested&nbsp;by&nbsp;slave&nbsp;58a40dbe01e1563773724803854406df04c62724. 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:48.261&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;58a40dbe01e1563773724803854406df04c62724&nbsp;for&nbsp;epoch&nbsp;32 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:48.261&nbsp;-&nbsp;Client&nbsp;closed&nbsp;connection &nbsp; 10.51.147.216:7388为failover前的slave， 10.51.147.216:7388的ID为58a40dbe01e1563773724803854406df04c62724 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:48.261&nbsp;#&nbsp;Connection&nbsp;with&nbsp;slave&nbsp;10.51.147.216:7388&nbsp;lost. 35475:M&nbsp;06&nbsp;Sep&nbsp;20:35:48.278&nbsp;#&nbsp;Configuration&nbsp;change&nbsp;detected.&nbsp;Reconfiguring&nbsp;myself&nbsp;as&nbsp;a&nbsp;replica&nbsp;of&nbsp;58a40dbe01e1563773724803854406df04c62724 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.280&nbsp;-&nbsp;Client&nbsp;closed&nbsp;connection 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;-&nbsp;DB&nbsp;0:&nbsp;16870296&nbsp;keys&nbsp;(7931385&nbsp;volatile)&nbsp;in&nbsp;50331648&nbsp;slots&nbsp;HT. 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;-&nbsp;1953&nbsp;clients&nbsp;connected&nbsp;(0&nbsp;slaves),&nbsp;5722753736&nbsp;bytes&nbsp;in&nbsp;use 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;*&nbsp;Connecting&nbsp;to&nbsp;MASTER&nbsp;10.51.147.216:7388 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;*&nbsp;MASTER&nbsp;&lt;-&gt;&nbsp;SLAVE&nbsp;sync&nbsp;started 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;*&nbsp;Non&nbsp;blocking&nbsp;connect&nbsp;for&nbsp;SYNC&nbsp;fired&nbsp;the&nbsp;event. 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;*&nbsp;Master&nbsp;replied&nbsp;to&nbsp;PING,&nbsp;replication&nbsp;can&nbsp;continue... 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.408&nbsp;*&nbsp;Partial&nbsp;resynchronization&nbsp;not&nbsp;possible&nbsp;(no&nbsp;cached&nbsp;master) 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.459&nbsp;*&nbsp;Full&nbsp;resync&nbsp;from&nbsp;master:&nbsp;36beb63d32b3809039518bf4f3e4e10de227f3ee:16454238619 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.493&nbsp;-&nbsp;Client&nbsp;closed&nbsp;connection 35475:S&nbsp;06&nbsp;Sep&nbsp;20:35:48.880&nbsp;-&nbsp;Client&nbsp;closed&nbsp;connection </td> 
 </tr> 
 </tbody> 
</table>

# **19. 查看集群信息** #

对应的redis命令为：cluster info，示例：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 127.0.0.1:6381&gt;&nbsp;cluster&nbsp;info cluster_state:ok&nbsp;所有slots正常则显示为OK，否则为error cluster_slots_assigned:16384&nbsp;多少slots被分配了，即多少被master管理了，16384为全部slots cluster_slots_ok:16384&nbsp;有多少slots是正常的 cluster_slots_pfail:0&nbsp;有多少slots可能处于异常状态，处于这个状态并不表示有问题，仍能继续提供服务 cluster_slots_fail:0&nbsp;有多少slots处于异常状态，需要修复才能服务 cluster_known_nodes:10&nbsp;集群中的节点数 cluster_size:3&nbsp;集群中master个数 cluster_current_epoch:11&nbsp;本地的当前时间变量，用于故障切换时生成独一无二的增量版本号 cluster_my_epoch:0 cluster_stats_messages_sent:4049&nbsp;通过集群消息总线发送的消息总数 cluster_stats_messages_received:4051&nbsp;通过过集通过群消息总线收到的消息总数 </td> 
 </tr> 
 </tbody> 
</table>

# **20. 禁止指定命令** #

KEYS命令很耗时，FLUSHDB和FLUSHALL命令可能导致误删除数据，所以线上环境最好禁止使用，可以在Redis配置文件增加如下配置：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> rename-command&nbsp;KEYS&nbsp;"" rename-command&nbsp;FLUSHDB&nbsp;"" rename-command&nbsp;FLUSHALL&nbsp;"" </td> 
 </tr> 
 </tbody> 
</table>

# **21. 各版本配置文件** #

[https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf][https_raw.githubusercontent.com_antirez_redis_3.0_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/3.2.9/redis.conf][https_raw.githubusercontent.com_antirez_redis_3.2.9_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0.1/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0.1_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0.3/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0.3_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0.5/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0.5_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0.9/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0.9_redis.conf]

[https://raw.githubusercontent.com/antirez/redis/4.0.11/redis.conf][https_raw.githubusercontent.com_antirez_redis_4.0.11_redis.conf]

# **22. 大压力下Redis参数调整要点** #

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> 参数 </td> 
 <td style="vertical-align:top;width:87.6pt;"> 建议最小值 </td> 
 <td style="vertical-align:top;width:219.35pt;"> 说明 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> repl-ping-slave-period </td> 
 <td style="vertical-align:top;width:87.6pt;"> 10 </td> 
 <td style="vertical-align:top;width:219.35pt;"> 每10秒ping一次 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> repl-timeout </td> 
 <td style="vertical-align:top;width:87.6pt;"> 60 </td> 
 <td style="vertical-align:top;width:219.35pt;"> 60秒超时，也就是ping十次 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> cluster-node-timeout </td> 
 <td style="vertical-align:top;width:87.6pt;"> 15000 </td> 
 <td style="vertical-align:top;width:219.35pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> repl-backlog-size </td> 
 <td style="vertical-align:top;width:87.6pt;"> 1GB </td> 
 <td style="vertical-align:top;width:219.35pt;"> Master对slave的队列大小 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> appendfsync </td> 
 <td style="vertical-align:top;width:87.6pt;"> no </td> 
 <td style="vertical-align:top;width:219.35pt;"> 让系统自动刷 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> save </td> 
 <td style="vertical-align:top;width:87.6pt;"> &nbsp; </td> 
 <td style="vertical-align:top;width:219.35pt;"> 大压力下，调大参数值，以减少写RDB带来的压力： "900&nbsp;20&nbsp;300&nbsp;200&nbsp;60&nbsp;200000" </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:119.15pt;"> appendonly </td> 
 <td style="vertical-align:top;width:87.6pt;"> &nbsp; </td> 
 <td style="vertical-align:top;width:219.35pt;"> 对于队列，建议单独建立集群，并且设置该值为no </td> 
 </tr> 
 </tbody> 
</table>

为何大压力下要这样调整？

最重要的原因之一Redis的主从复制，两者复制共享同一线程，虽然是异步复制的，但因为是单线程，所以也十分有限。如果主从间的网络延迟不是在0.05左右，比如达到0.6，甚至1.2等，那么情况是非常糟糕的，因此同一Redis集群一定要部署在同一机房内。

这些参数的具体值，要视具体的压力而定，而且和消息的大小相关，比如一条200~500KB的流水数据可能比较大，主从复制的压力也会相应增大，而10字节左右的消息，则压力要小一些。大压力环境中开启appendfsync是十分不可取的，容易导致整个集群不可用，在不可用之前的典型表现是QPS毛刺明显。

这么做的目的是让Redis集群尽可能的避免master正常时触发主从切换，特别是容纳的数据量很大时，和大压力结合在一起，集群会雪崩。

当Redis日志中，出现大量如下信息，即可能意味着相关的参数需要调整了：

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:05.388&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;e438a338e9d9834a6745c12931950da87e360ca2 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:07.551&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:08.438&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f&nbsp;for&nbsp;epoch&nbsp;285&nbsp;（We&nbsp;can&nbsp;vote&nbsp;for&nbsp;this&nbsp;slave） &nbsp; 有投票资格的node： 1）为master 2）至少有一个slot 3）投票node的epoch不能小于node自己当前的epoch（reqEpoch&nbsp;&lt;&nbsp;curEpoch） 4）node没有投票过该epoch（already&nbsp;voted&nbsp;for&nbsp;epoch） 5）投票node不能为master（it&nbsp;is&nbsp;a&nbsp;master&nbsp;node） 6）投票node必须有一个master（I&nbsp;don't&nbsp;know&nbsp;its&nbsp;master） 7）投票node的master处于fail状态（its&nbsp;master&nbsp;is&nbsp;up） &nbsp; 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:19.844&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;denied&nbsp;to&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a:&nbsp;already&nbsp;voted&nbsp;for&nbsp;epoch&nbsp;285 &nbsp; 如果一个node又联系上了，则它当是一个slave，或者无slots的master时，直接清除FAIL标志；但如果是一个master，则当“(now&nbsp;-&nbsp;node-&gt;fail_time)&nbsp;&gt;&nbsp;(server.cluster_node_timeout&nbsp;*&nbsp;CLUSTER_FAIL_UNDO_TIME_MULT)”时，也清除FAIL标志，定义在cluster.h中（cluster.h:#define&nbsp;CLUSTER_FAIL_UNDO_TIME_MULT&nbsp;2&nbsp;/*&nbsp;Undo&nbsp;fail&nbsp;if&nbsp;master&nbsp;is&nbsp;back.&nbsp;*/） 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:29.243&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. &nbsp; 如果消息类型为fail。 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:31.995&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f&nbsp;about&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:32.496&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:32.968&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec&nbsp;for&nbsp;epoch&nbsp;286 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:33.177&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0&nbsp;for&nbsp;epoch&nbsp;287 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:36.336&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:36.855&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.419&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;e438a338e9d9834a6745c12931950da87e360ca2:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:54.954&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:56.697&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.705&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7&nbsp;for&nbsp;epoch&nbsp;288 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.890&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 22135:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.892&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. </td> 
 </tr> 
 </tbody> 
</table>

# **23. 问题排查** #

1) 如果最后一条日志为“16367:M 08 Jun 14:48:15.560 \# Server started, Redis version 3.2.0”，节点状态始终终于fail状态，则可能是aof文件损坏了，这时可以使用工具edis-check-aof --fix进行修改，如：

../../bin/redis-check-aof --fix appendonly-6380.aof

0x        a1492b9b: **Expected prefix '**

AOF analyzed: size=2705928192, ok\_up\_to=2705927067, diff=1125

This will shrink the AOF from 2705928192 bytes, with 1125 bytes, to 2705927067 bytes

Continue? \[y/N\]: **y**

2) in \`call': ERR Slot 16011 is already busy (Redis::CommandError)

将所有节点上的配置项cluster-config-file指定的文件删除，然后重新启；或者在所有节点上执行下FLUSHALL命令。

另外，如果使用主机名而不是IP，也可能遇到这个错误，如：“redis-trib.rb create --replicas 1 redis1:6379 redis2:6379 redis3:6379 redis4:6379 redis5:6379 redis6:6379”，可能也会得到错误“ERR Slot 16011 is already busy (Redis::CommandError)”。

**3) ****for lack of backlog** (Slave request was: 51875158284)

默认值：

\# redis-cli config get repl-timeout

A) "repl-timeout"

B) "10"

\# redis-cli config get client-output-buffer-limit

A) "client-output-buffer-limit"

B) "normal 0 0 0 slave **268435456 67108864 **60 pubsub 33554432 8388608 60"

增大：

redis-cli config set "client-output-buffer-limit" "normal 0 0 0 slave **2684354560 671088640 60** pubsub 33554432 8388608 60"

4) 复制中断场景

A) master的slave缓冲区达到限制的硬或软限制大小，与参数client-output-buffer-limit相关；

B) 复制时间超过repl-timeout指定的值，与参数repl-timeout相关。

slave反复循环从master复制，如果调整以上参数仍然解决不了，可以尝试删除slave上的aof和rdb文件，然后再重启进程复制，这个时候可能能正常完成复制。

5) 日志文件出现：Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

考虑优化以下配置项：

no-appendfsync-on-rewrite值设为yes

repl-backlog-size和client-output-buffer-limit调大一点

6) 日志文件出现：MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

考虑设置stop-writes-on-bgsave-error值为“no”。

7) Failover auth granted to

当日志大量反反复复出现下列内容时，很可能表示master和slave间同步和通讯不顺畅，导致无效的failover和状态变更，这个时候需要调大相关参数值，容忍更长的延迟，因此也特别注意集群内所有节点间的网络延迟要尽可能的小，最好达到0.02ms左右的水平，调大参数的代价是主备切换变迟钝。

**Slave日志：**

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:24.169&nbsp;*&nbsp;Connecting&nbsp;to&nbsp;MASTER&nbsp;10.5.14.8:6379 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:24.169&nbsp;*&nbsp;MASTER&nbsp;&lt;-&gt;&nbsp;SLAVE&nbsp;sync&nbsp;started 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:24.169&nbsp;#&nbsp;Start&nbsp;of&nbsp;election&nbsp;delayed&nbsp;for&nbsp;854&nbsp;milliseconds&nbsp;(rank&nbsp;#0,&nbsp;offset&nbsp;5127277817). 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:24.169&nbsp;*&nbsp;Non&nbsp;blocking&nbsp;connect&nbsp;for&nbsp;SYNC&nbsp;fired&nbsp;the&nbsp;event. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:25.069&nbsp;#&nbsp;Starting&nbsp;a&nbsp;failover&nbsp;election&nbsp;for&nbsp;epoch&nbsp;266. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.190&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.191&nbsp;*&nbsp;Master&nbsp;replied&nbsp;to&nbsp;PING,&nbsp;replication&nbsp;can&nbsp;continue... 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.191&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.192&nbsp;*&nbsp;Trying&nbsp;a&nbsp;partial&nbsp;resynchronization&nbsp;(request&nbsp;ea2261c827fbc54135a95f707046581a55dff133:5127277818). 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.192&nbsp;*&nbsp;Successful&nbsp;partial&nbsp;resynchronization&nbsp;with&nbsp;master. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.192&nbsp;*&nbsp;MASTER&nbsp;&lt;-&gt;&nbsp;SLAVE&nbsp;sync:&nbsp;Master&nbsp;accepted&nbsp;a&nbsp;Partial&nbsp;Resynchronization. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:29.811&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;e438a338e9d9834a6745c12931950da87e360ca2:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:37.680&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;5b41f7860cc800e65932e92d1d97c6c188138e56&nbsp;about&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:43.710&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:48.119&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b&nbsp;about&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:49.410&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;5b41f7860cc800e65932e92d1d97c6c188138e56&nbsp;about&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:53.352&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:07:57.147&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:08:36.516&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec 31019:S&nbsp;06&nbsp;Sep&nbsp;11:08:41.900&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:08:46.380&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261&nbsp;about&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 31019:S&nbsp;06&nbsp;Sep&nbsp;11:08:46.531&nbsp;*&nbsp;Marking&nbsp;node&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8&nbsp;as&nbsp;failing&nbsp;(quorum&nbsp;reached). 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:01.882&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:01.883&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:06.538&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;e438a338e9d9834a6745c12931950da87e360ca2&nbsp;about&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:06.538&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;e438a338e9d9834a6745c12931950da87e360ca2&nbsp;about&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:12.555&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:12.555&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:15.565&nbsp;*&nbsp;Marking&nbsp;node&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a&nbsp;as&nbsp;failing&nbsp;(quorum&nbsp;reached). 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:16.599&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;0a92bd7472c9af3e52f9185eac1bd1bbf36146e6&nbsp;about&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:22.262&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:27.906&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:50.744&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:55.141&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;5b41f7860cc800e65932e92d1d97c6c188138e56&nbsp;about&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:55.362&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b&nbsp;about&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:55.557&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:55.578&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;144347d5a51acf047887fe81f22e8f7705c911ec 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:56.521&nbsp;*&nbsp;Marking&nbsp;node&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a&nbsp;as&nbsp;failing&nbsp;(quorum&nbsp;reached). 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:57.996&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:09:58.329&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;5b41f7860cc800e65932e92d1d97c6c188138e56&nbsp;about&nbsp;0a92bd7472c9af3e52f9185eac1bd1bbf36146e6 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:09.239&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;144347d5a51acf047887fe81f22e8f7705c911ec:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:09.812&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:13.549&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:13.590&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;716f2e2dd9792eaf4ee486794c9797fa6e1c9650&nbsp;about&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:13.591&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f&nbsp;about&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:14.316&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7:&nbsp;is&nbsp;reachable&nbsp;again&nbsp;and&nbsp;nobody&nbsp;is&nbsp;serving&nbsp;its&nbsp;slots&nbsp;after&nbsp;some&nbsp;time. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:15.108&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:17.588&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:32.622&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;0a92bd7472c9af3e52f9185eac1bd1bbf36146e6:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:32.623&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;5b41f7860cc800e65932e92d1d97c6c188138e56&nbsp;about&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386 31019:S&nbsp;06&nbsp;Sep&nbsp;11:10:32.623&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. </td> 
 </tr> 
 </tbody> 
</table>

**Master日志：**

<table style="width:426.1pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:426.1pt;"> 31014:M&nbsp;06&nbsp;Sep&nbsp;14:08:54.083&nbsp;*&nbsp;Background&nbsp;saving&nbsp;terminated&nbsp;with&nbsp;success 31014:M&nbsp;06&nbsp;Sep&nbsp;14:09:55.093&nbsp;*&nbsp;10000&nbsp;changes&nbsp;in&nbsp;60&nbsp;seconds.&nbsp;Saving... 31014:M&nbsp;06&nbsp;Sep&nbsp;14:09:55.185&nbsp;*&nbsp;Background&nbsp;saving&nbsp;started&nbsp;by&nbsp;pid&nbsp;41395 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:00.269&nbsp;#&nbsp;Disconnecting&nbsp;timedout&nbsp;slave:&nbsp;10.15.40.9:6018 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:00.269&nbsp;#&nbsp;Connection&nbsp;with&nbsp;slave&nbsp;10.15.40.9:6018&nbsp;lost. 41395:C&nbsp;06&nbsp;Sep&nbsp;14:11:01.141&nbsp;*&nbsp;DB&nbsp;saved&nbsp;on&nbsp;disk 41395:C&nbsp;06&nbsp;Sep&nbsp;14:11:01.259&nbsp;*&nbsp;RDB:&nbsp;5&nbsp;MB&nbsp;of&nbsp;memory&nbsp;used&nbsp;by&nbsp;copy-on-write 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:01.472&nbsp;*&nbsp;Background&nbsp;saving&nbsp;terminated&nbsp;with&nbsp;success 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:11.525&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:23.039&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6&nbsp;about&nbsp;d78845370c98b3ce4cfc02e8d3e233a9a1d84a83 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:23.541&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:23.813&nbsp;*&nbsp;Slave&nbsp;10.15.40.9:6018&nbsp;asks&nbsp;for&nbsp;synchronization 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:23.813&nbsp;*&nbsp;Partial&nbsp;resynchronization&nbsp;request&nbsp;from&nbsp;10.15.40.9:6018&nbsp;accepted.&nbsp;Sending&nbsp;46668&nbsp;bytes&nbsp;of&nbsp;backlog&nbsp;starting&nbsp;from&nbsp;offset&nbsp;5502672944. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:23.888&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b&nbsp;for&nbsp;epoch&nbsp;283 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:32.464&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652&nbsp;about&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:47.616&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d78845370c98b3ce4cfc02e8d3e233a9a1d84a83:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:55.515&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652&nbsp;about&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 31014:M&nbsp;06&nbsp;Sep&nbsp;14:11:57.135&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;for&nbsp;epoch&nbsp;284 31014:M&nbsp;06&nbsp;Sep&nbsp;14:12:01.766&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:12:08.753&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;3114cec541c5bcd36d712cd6c9f4c5055510e386:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:16:02.070&nbsp;*&nbsp;10&nbsp;changes&nbsp;in&nbsp;300&nbsp;seconds.&nbsp;Saving... 31014:M&nbsp;06&nbsp;Sep&nbsp;14:16:02.163&nbsp;*&nbsp;Background&nbsp;saving&nbsp;started&nbsp;by&nbsp;pid&nbsp;13832 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:18.443&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:18.443&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f&nbsp;for&nbsp;epoch&nbsp;285 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:29.272&nbsp;#&nbsp;Connection&nbsp;with&nbsp;slave&nbsp;client&nbsp;id&nbsp;#40662&nbsp;lost. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:29.273&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;denied&nbsp;to&nbsp;534b93af6ba45a7033dbf38c8f47cd688514125a:&nbsp;already&nbsp;voted&nbsp;for&nbsp;epoch&nbsp;285 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:29.278&nbsp;*&nbsp;Slave&nbsp;10.15.40.9:6018&nbsp;asks&nbsp;for&nbsp;synchronization 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:29.278&nbsp;*&nbsp;Partial&nbsp;resynchronization&nbsp;request&nbsp;from&nbsp;10.15.40.9:6018&nbsp;accepted.&nbsp;Sending&nbsp;117106&nbsp;bytes&nbsp;of&nbsp;backlog&nbsp;starting&nbsp;from&nbsp;offset&nbsp;5502756264. 13832:C&nbsp;06&nbsp;Sep&nbsp;14:17:29.850&nbsp;*&nbsp;DB&nbsp;saved&nbsp;on&nbsp;disk 13832:C&nbsp;06&nbsp;Sep&nbsp;14:17:29.970&nbsp;*&nbsp;RDB:&nbsp;7&nbsp;MB&nbsp;of&nbsp;memory&nbsp;used&nbsp;by&nbsp;copy-on-write 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.449&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f&nbsp;about&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.449&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.449&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;denied&nbsp;to&nbsp;938d9ae2de278938beda1d39185608b02d3b31ec:&nbsp;reqEpoch&nbsp;(286)&nbsp;&lt;&nbsp;curEpoch(287) 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.449&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;d9dadf3342006e2c92def3071ca0a76390be62b0&nbsp;for&nbsp;epoch&nbsp;287 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.449&nbsp;*&nbsp;Background&nbsp;saving&nbsp;terminated&nbsp;with&nbsp;success 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.450&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d7942cfe636b25219c6d56aa72828fcfde2ee261:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.450&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;1ba437fa1683a8caafd38ff977e5fbabdaf84fd6:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:38.452&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;d6eb06e9d118c120d3961a659972a1d0191a8652:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:54.985&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467&nbsp;about&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:56.729&nbsp;*&nbsp;FAIL&nbsp;message&nbsp;received&nbsp;from&nbsp;1d07e208db56cfd7395950ca66e03589278b8e12&nbsp;about&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.737&nbsp;#&nbsp;Failover&nbsp;auth&nbsp;granted&nbsp;to&nbsp;e1c202d89ffe1c61b682e28071627635974c84a7&nbsp;for&nbsp;epoch&nbsp;288 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.922&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5:&nbsp;master&nbsp;without&nbsp;slots&nbsp;is&nbsp;reachable&nbsp;again. 31014:M&nbsp;06&nbsp;Sep&nbsp;14:17:57.923&nbsp;*&nbsp;Clear&nbsp;FAIL&nbsp;state&nbsp;for&nbsp;node&nbsp;7990d146cece7dc83eaf08b3e12cbebb2223f5f8:&nbsp;slave&nbsp;is&nbsp;reachable&nbsp;again. </td> 
 </tr> 
 </tbody> 
</table>

[http_redis.io_topics_cluster-tutorial]: http://redis.io/topics/cluster-tutorial
[https_github.com_eyjian_redis-tools_tree_master_deploy]: https://github.com/eyjian/redis-tools/tree/master/deploy
[https_github.com_eyjian_redis-tools_tree_master]: https://github.com/eyjian/redis-tools/tree/master
[https_github.com_eyjian_libmooon_releases]: https://github.com/eyjian/libmooon/releases
[https_github.com_eyjian_redis-tools]: https://github.com/eyjian/redis-tools
[https_raw.githubusercontent.com_antirez_redis_3.0_redis.conf]: https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf
[https_www.ruby-lang.org]: https://www.ruby-lang.org
[ruby-2.3.1.tar.gz]: https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz
[https_rubygems.org_pages_download]: https://rubygems.org/pages/download
[https_rubygems.org_gems_redis_versions_3.0.0]: https://rubygems.org/gems/redis/versions/3.0.0
[https_rubygems.org_downloads_redis-3.0.0.gem]: https://rubygems.org/downloads/redis-3.0.0.gem
[https_rubygems.org_gems_redis_versions_3.3.0]: https://rubygems.org/gems/redis/versions/3.3.0
[https_rubygems.org_gems_redis_versions_3.3.3]: https://rubygems.org/gems/redis/versions/3.3.3
[https_rubygems.org_gems_redis_versions_4.0.1]: https://rubygems.org/gems/redis/versions/4.0.1
[READONLY]: http://redis.io/commands/readonly
[https_github.com_xetorthio_jedis]: https://github.com/xetorthio/jedis
[https_github.com_eyjian_r3c]: https://github.com/eyjian/r3c
[https_redis.io_commands_cluster-setslot]: https://redis.io/commands/cluster-setslot
[https_raw.githubusercontent.com_antirez_redis_3.2.9_redis.conf]: https://raw.githubusercontent.com/antirez/redis/3.2.9/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0.1_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0.1/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0.3_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0.3/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0.5_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0.5/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0.9_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0.9/redis.conf
[https_raw.githubusercontent.com_antirez_redis_4.0.11_redis.conf]: https://raw.githubusercontent.com/antirez/redis/4.0.11/redis.conf