redis集群节点宕机 fail状态 redis的投票机制实测

刺骨的言语ヽ痛彻心扉 2023-07-05 13:07 13阅读 0赞

## redis集群节点宕机 fail状态 redis的投票机制实测 ##

redis集群是有很多个redis一起工作，那么就需要这个集群不是那么容易挂掉，所以呢，理论上就应该给集群中的每个节点至少一个备用的redis服务。这个备用的redis称为从节点（slave）。  
 *.本文测试 redis集群节点宕机 fail状态以上篇博文为基础[Redis集群策略及集群实例][Redis]。*

在[Redis集群策略及集群实例][Redis]中通过单机集群创建多实例（3主3从）的方式搭建了redis集群，以此为基础做实测  
 首先查看一下集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8001
    192.168.0.108:8001> cluster nodes
    3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581696889554 3 connected 10923-16383
    fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 slave 1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 0 1581696889553 8 connected
    8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581696889000 4 connected
    1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581696889754 5 connected
    1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master - 0 1581696889000 8 connected 5461-10922
    76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave 1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 0 1581696888545 8 connected
    5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581696888000 1 connected 0-5460
    192.168.0.108:8001>

查看控制台我们可以看出有3个主节点，4个从节点

> master8001-----------slave=8005  
> master8002-----------slave=8006 8008  
> master8003-----------slave=8004

--------------------

### 集群是如何判断是否有某个节点挂掉（宕机） ###

首先要说的是，每一个节点都存有这个集群所有主节点以及从节点的信息。它们之间通过互相的ping-pong判断是否节点可以连接上。如果有一半以上的节点去ping一个节点的时候没有回应，集群就认为这个节点宕机了，然后去连接它的备用节点。  
***由于没有更多环境测试，本文测试手动停掉主节点作为宕机测试验证***

### redis的投票机制 ###

我将其中一个主节点实例服务停掉看一下他的状态  
 停掉8002这个主节点，它有两个从节点

192.168.0.108:8001> 
    [root@localhost redis-4.0.10]# ps -ef|grep redis
    root      54357      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8001 [cluster]
    root      54362      1  0 Feb14 ?        00:01:24 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8002 [cluster]
    root      54367      1  0 Feb14 ?        00:01:20 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
    root      54372      1  0 Feb14 ?        00:01:22 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
    root      54377      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8005 [cluster]
    root      54382      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
    root      54649      1  0 Feb14 ?        00:01:03 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
    root      56002  55813  0 00:21 pts/7    00:00:00 grep --color=auto redis
    [root@localhost redis-4.0.10]# kill -9 54362

查看一下集群状态

192.168.0.108:8001> cluster nodes
    3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581697369000 3 connected 10923-16383
    fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581697368570 9 connected 5461-10922
    8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581697370392 4 connected
    1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581697369000 5 connected
    1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master,fail - 1581697316611 1581697314597 8 disconnected  
    76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581697369379 9 connected
    5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581697369000 1 connected 0-5460
    192.168.0.108:8001>

我们可以看到`1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master,fail - 1581697316611 1581697314597 8 disconnected` 8002节点实例fail，那他的从节点8006，8008呢，  
 接着看控制到`fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581697368570 9 connected 5461-10922` 8006从一个**从节点变为新的主节点**继续工作，再看8008`76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581697369379 9 connected` 8008成为新的主节点8006的从节点了，  
  **tip：集群中的每个节点都有1个至N个复制品，其中一个为主节点，其余的为从节点，如果主节点下线了，集群就会把这个主节点的一个从节点设置为新的主节点继续工作，这样集群就不会因为一个主节点的下线而无法正常工作。**  
三 我将8002节点服务器重新启动，来看一下会有什么反应

[root@localhost redis-4.0.10]# src/redis-server /usr/local/redis-cluster/8002/redis.conf 
    56009:C 15 Feb 00:37:57.883 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
    56009:C 15 Feb 00:37:57.883 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=56009, just started
    56009:C 15 Feb 00:37:57.883 # Configuration loaded
    [root@localhost redis-4.0.10]#

查看集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8001 
    192.168.0.108:8001> cluster nodes
    3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581698340487 3 connected 10923-16383
    fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581698339000 9 connected 5461-10922
    8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581698339479 4 connected
    1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581698338471 5 connected
    1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698339479 9 connected
    76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698338974 9 connected
    5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581698338000 1 connected 0-5460
    192.168.0.108:8001>

查看控制台`1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698339479 9 connected` 原来的8002主节点成为了`fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581698339000 9 connected 5461-10922` 新主节点8006的从节点。  
 **tip:旧主重新连接后将其变为新主的从服务器**

**总结：redis的投票机制  
   投票过程是集群中所有master参与,如果半数以上master节点与master节点通信超时(cluster-node-timeout),认为当前master节点挂掉。  
   选举的依据依次是：网络连接正常->5秒内回复过INFO命令->10\*down-after-milliseconds内与主连接过的->从服务器优先级->复制偏移量->运行id较小的。选出之后通过slaveif no ont将该从服务器升为新主服务器。  
   通过slaveof ip port命令让其他从服务器复制该信主服务器。  
   最后当旧主重新连接后将其变为新主的从服务器**

### 集群进入fail状态的必要条件 ###

1、某个主节点和所有从节点全部挂掉，我们集群就进入faill状态。  
 2、如果集群超过半数以上master挂掉，无论是否有slave，集群进入fail状态.  
 3、如果集群任意master挂掉,且当前master没有slave.集群进入fail状态  
** 测试，我将主节点8001和它的从节点8005手动停掉，看看什么状态**

[root@localhost redis-4.0.10]# ps -ef|grep redis
    root      54357      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8001 [cluster]
    root      54367      1  0 Feb14 ?        00:01:27 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
    root      54372      1  0 Feb14 ?        00:01:29 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
    root      54377      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8005 [cluster]
    root      54382      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
    root      54649      1  0 Feb14 ?        00:01:11 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
    root      56010      1  0 00:37 ?        00:00:02 src/redis-server 192.168.0.108:8002 [cluster]
    root      56043  55813  0 00:55 pts/7    00:00:00 grep --color=auto redis
    [root@localhost redis-4.0.10]# kill -9 54357 54377
    [root@localhost redis-4.0.10]#

查看服务进程

[root@localhost redis-4.0.10]# ps -ef|grep redis 
    root      54367      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
    root      54372      1  0 Feb14 ?        00:01:30 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
    root      54382      1  0 Feb14 ?        00:01:29 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
    root      54649      1  0 Feb14 ?        00:01:11 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
    root      56010      1  0 00:37 ?        00:00:03 src/redis-server 192.168.0.108:8002 [cluster]
    root      56046  55813  0 00:57 pts/7    00:00:00 grep --color=auto redis
    [root@localhost redis-4.0.10]#

目前8001，8005 已关闭，查看集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8003
    192.168.0.108:8003> cluster nodes
    3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 myself,master - 0 1581699578000 3 connected 10923-16383
    8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581699579164 4 connected
    76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581699579000 9 connected
    fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581699578051 9 connected 5461-10922
    5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 master,fail - 1581699388186 1581699386575 1 disconnected 0-5460
    1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave,fail 5ffc6c0d316f3352443f1979239c852e6c8e4540 1581699388186 1581699386000 5 disconnected
    1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581699578000 9 connected
    192.168.0.108:8003>

查看控制台`5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 master,fail - 1581699388186 1581699386575 1 disconnected 0-5460 1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave,fail 5ffc6c0d316f3352443f1979239c852e6c8e4540 1581699388186 1581699386000 5 disconnected`  
  主节点8001和从节点8005都已,fail ，8001节点redis集群就会停止工作了。

### 主存复制数据问题 ###

1、如果某一个主节点和他所有的从节点都下线的话，redis集群就会停止工作了。redis集群不保证数据的强一致性，在特定的情况下，redis集群会丢失已经被执行过的写命令。  
 2、使用异步复制（asynchronous replication）是redis 集群可能会丢失写命令的其中一个原因，有时候由于网络原因，如果网络断开时间太长，redis集群就会启用新的主节点，之前发给主节点的数据就会丢失。

参看文档：[https://www.cnblogs.com/dadonggg/p/8628735.html][https_www.cnblogs.com_dadonggg_p_8628735.html]

[Redis]: https://blog.csdn.net/iT_MaNongking/article/details/104301854
[https_www.cnblogs.com_dadonggg_p_8628735.html]: https://www.cnblogs.com/dadonggg/p/8628735.html