Redis 6.0 源码阅读笔记(10)-主从复制 Master 节点流程分析

￡神魔★判官ぃ 2023-10-05 12:52 3阅读 0赞

#### 文章目录 ####

*  1. 主从复制中 Master 节点的处理
 *  2. 处理流程分析
 *   *   *  2.1 psync 命令的大致处理
         *  2.2 rdb 数据的生成与发送
         *   *  2.2.1 直接通过 socket 发送 rdb 数据
             *  2.2.2 发送 rdb 文件

## 1. 主从复制中 Master 节点的处理 ##

[Redis 主从复制][Redis] 一节中粗略介绍过 Redis 的主从复制存在两种模式，以下则是 Master 节点对于主从复制的处理：

> 1.  接收 Slave 节点的 psync 命令
> 2.  判断是否可以执行部分复制，可以的话回复 `+CONTINUE` 给 Slave 节点，并将缓冲区命令直接发送给 Slave 节点，完成同步；不可以执行部分复制，只能全量复制，则回复 `+FULLRESYNC` 给 Slave 节点，根据情况做下一步处理
> 3.  Master 节点存在可以复用的 rdb 数据，则直接将其发送给 Slave 节点，否则新起一个进程异步生成 rdb 数据
> 4.  如果 Slave 节点同步配置可以直接通过 socket 发送 rdb 数据则直接发送，否则 Master 节点需要保存 rdb 数据到 rdb 文件
> 5.  Master 节点定时任务周期检查 rdb 文件是否保存完毕，完成则发送 rdb 文件到 Slave 节点
> 6.  最后发送 rdb 文件生成过程中缓存的 redis 执行命令给 Slave 节点

![在这里插入图片描述][20201106113036907.jpg_pic_center]

## 2. 处理流程分析 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTUwNTMxMw_size_16_color_FFFFFF_t_70_pic_center]

#### 2.1 psync 命令的大致处理 ####

1.  Redis 中对 `psync` 命令的处理由 `replication.c#syncCommand()` 函数完成，这个函数的实现源码较为复杂，不过大致逻辑可分为以下几步：
    
    > 1.  首先是一般性的校验，比如验证接收 `psync` 命令的服务端是从节点的话，直接返回；另外如果当前服务端的主节点 host 有值但是同步状态不在 `REPL_STATE_CONNECTED` 时，说明当前从节点没有与主节点连接上，也直接返回
    > 2.  接下来解析传输过来的命令，如果是 `psync` 命令，则调用 `masterTryPartialResynchronization()` 函数尝试进行从节点部分复制
    > 3.  主节点接收的命令不是 `psync` 或者从节点部分复制失败了，则需要进行全量复制。如果这是主节点第一次全量复制，还需要改变`server.replid`，并创建复制缓冲区
    > 4.  此时主节点需要根据自身的场景做处理：
    >     
    >     1.  如果此时已经有保存 rdb 数据的 `BGSAVE` 进程在运行，并且该进程会把 rdb 数据保存为 rdb 文件，则需要检查当前要求与主节点同步的从节点是否可以复用其他从节点即将传输的 rdb 数据，可以的话直接把 rdb 数据复制到当前从节点的缓存区，并调用 `replicationSetupSlaveForFullResync()` 函数通知 `+FULLRESYNC` 给从节点，主节点轮询到该从节点连接可写时会把 rdb 数据发送过去
    >     2.  如果此时已经有保存 rdb 数据的进程在运行，但是该进程生成的 rdb 数据会直接通过 socket 发送到对端，则这份 rdb 数据不可复用，当前从节点等待下一次 `BGSAVE`
    >     3.  如果此时没有保存 rdb 数据的进程在运行，并且从节点不支持直接通过 socket 接收 rdb 数据，则调用 `startBgsaveForReplication()` 函数启动一个进程后台保存 rdb 数据
    
        /* SYNC and PSYNC command implemenation. */
        void syncCommand(client *c) {
              
         /* ignore SYNC if already slave or in monitor mode */
         if (c->flags & CLIENT_SLAVE) return;
        
         /* Refuse SYNC requests if we are a slave but the link with our master
          * is not ok... */
         if (server.masterhost && server.repl_state != REPL_STATE_CONNECTED) {
              
             addReplySds(c,sdsnew("-NOMASTERLINK Can't SYNC while not connected with my master\r\n"));
             return;
         }
        
         /* SYNC can't be issued when the server has pending data to send to
          * the client about already issued commands. We need a fresh reply
          * buffer registering the differences between the BGSAVE and the current
          * dataset, so that we can copy to other slaves if needed. */
         if (clientHasPendingReplies(c)) {
              
             addReplyError(c,"SYNC and PSYNC are invalid with pending output");
             return;
         }
        
         serverLog(LL_NOTICE,"Replica %s asks for synchronization",
             replicationGetSlaveName(c));
        
         /* Try a partial resynchronization if this is a PSYNC command.
          * If it fails, we continue with usual full resynchronization, however
          * when this happens masterTryPartialResynchronization() already
          * replied with:
          *
          * +FULLRESYNC <replid> <offset>
          *
          * So the slave knows the new replid and offset to try a PSYNC later
          * if the connection with the master is lost. */
         if (!strcasecmp(c->argv[0]->ptr,"psync")) {
              
             if (masterTryPartialResynchronization(c) == C_OK) {
              
                 server.stat_sync_partial_ok++;
                 return; /* No full resync needed, return. */
             } else {
              
                 char *master_replid = c->argv[1]->ptr;
        
                 /* Increment stats for failed PSYNCs, but only if the
                  * replid is not "?", as this is used by slaves to force a full
                  * resync on purpose when they are not albe to partially
                  * resync. */
                 if (master_replid[0] != '?') server.stat_sync_partial_err++;
             }
         } else {
              
             /* If a slave uses SYNC, we are dealing with an old implementation
              * of the replication protocol (like redis-cli --slave). Flag the client
              * so that we don't expect to receive REPLCONF ACK feedbacks. */
             c->flags |= CLIENT_PRE_PSYNC;
         }
        
         /* Full resynchronization. */
         server.stat_sync_full++;
        
         /* Setup the slave as one waiting for BGSAVE to start. The following code
          * paths will change the state if we handle the slave differently. */
         c->replstate = SLAVE_STATE_WAIT_BGSAVE_START;
         if (server.repl_disable_tcp_nodelay)
             connDisableTcpNoDelay(c->conn); /* Non critical if it fails. */
         c->repldbfd = -1;
         c->flags |= CLIENT_SLAVE;
         listAddNodeTail(server.slaves,c);
        
         /* Create the replication backlog if needed. */
         if (listLength(server.slaves) == 1 && server.repl_backlog == NULL) {
              
             /* When we create the backlog from scratch, we always use a new
              * replication ID and clear the ID2, since there is no valid
              * past history. */
             changeReplicationId();
             clearReplicationId2();
             createReplicationBacklog();
             serverLog(LL_NOTICE,"Replication backlog created, my new "
                                 "replication IDs are '%s' and '%s'",
                                 server.replid, server.replid2);
         }
        
         /* CASE 1: BGSAVE is in progress, with disk target. */
         if (server.rdb_child_pid != -1 &&
             server.rdb_child_type == RDB_CHILD_TYPE_DISK)
         {
              
             /* Ok a background save is in progress. Let's check if it is a good
              * one for replication, i.e. if there is another slave that is
              * registering differences since the server forked to save. */
             client *slave;
             listNode *ln;
             listIter li;
        
             listRewind(server.slaves,&li);
             while((ln = listNext(&li))) {
              
                 slave = ln->value;
                 if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_END) break;
             }
             /* To attach this slave, we check that it has at least all the
              * capabilities of the slave that triggered the current BGSAVE. */
             if (ln && ((c->slave_capa & slave->slave_capa) == slave->slave_capa)) {
              
                 /* Perfect, the server is already registering differences for
                  * another slave. Set the right state, and copy the buffer. */
                 copyClientOutputBuffer(c,slave);
                 replicationSetupSlaveForFullResync(c,slave->psync_initial_offset);
                 serverLog(LL_NOTICE,"Waiting for end of BGSAVE for SYNC");
             } else {
              
                 /* No way, we need to wait for the next BGSAVE in order to
                  * register differences. */
                 serverLog(LL_NOTICE,"Can't attach the replica to the current BGSAVE. Waiting for next BGSAVE for SYNC");
             }
        
         /* CASE 2: BGSAVE is in progress, with socket target. */
         } else if (server.rdb_child_pid != -1 &&
                    server.rdb_child_type == RDB_CHILD_TYPE_SOCKET)
         {
              
             /* There is an RDB child process but it is writing directly to
              * children sockets. We need to wait for the next BGSAVE
              * in order to synchronize. */
             serverLog(LL_NOTICE,"Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC");
        
         /* CASE 3: There is no BGSAVE is progress. */
         } else {
              
             if (server.repl_diskless_sync && (c->slave_capa & SLAVE_CAPA_EOF)) {
              
                 /* Diskless replication RDB child is created inside
                  * replicationCron() since we want to delay its start a
                  * few seconds to wait for more slaves to arrive. */
                 if (server.repl_diskless_sync_delay)
                     serverLog(LL_NOTICE,"Delay next BGSAVE for diskless SYNC");
             } else {
              
                 /* Target is disk (or the slave is not capable of supporting
                  * diskless replication) and we don't have a BGSAVE in progress,
                  * let's start one. */
                 if (!hasActiveChildProcess()) {
              
                     startBgsaveForReplication(c->slave_capa);
                 } else {
              
                     serverLog(LL_NOTICE,
                         "No BGSAVE in progress, but another BG operation is active. "
                         "BGSAVE for replication delayed");
                 }
             }
         }
         return;
        }
2.  `reolication.c#masterTryPartialResynchronization()` 函数负责从节点部分复制的处理，其重要逻辑如下：
    
    > 1.  首先是参数检查，如果从节点带过来的 `master_replid` 和主节点的 `server.replid` 不相等，并且 `master_replid` 和主节点的 `server.replid2` 不相等，则不能进行部分复制
    > 2.  如果从节点带过来的复制偏移量 `psync_offset` 小于主节点复制缓存区起始点 `server.repl_backlog_off` 或者 `psync_offset` 大于主节点复制缓存区结束点，则也不能进行部分复制
    > 3.  排除了以上场景，则可以进行部分复制。这部分主要有以下几个操作：
    >     
    >     1.  将从节点的同步状态标记为 `SLAVE_STATE_ONLINE`，并将其添加到主节点保存的从节点列表`server.slaves`的尾部
    >     2.  给从节点发送 `+CONTINUE` 通知其可以进行部分复制
    >     3.  调用 `addReplyReplicationBacklog()` 函数将主节点复制缓存区指定偏移量的数据添加到从节点的缓存区，等待其可写时发送过去
    
        int masterTryPartialResynchronization(client *c) {
              
         long long psync_offset, psync_len;
         char *master_replid = c->argv[1]->ptr;
         char buf[128];
         int buflen;
        
         /* Parse the replication offset asked by the slave. Go to full sync
          * on parse error: this should never happen but we try to handle
          * it in a robust way compared to aborting. */
         if (getLongLongFromObjectOrReply(c,c->argv[2],&psync_offset,NULL) !=
            C_OK) goto need_full_resync;
        
         /* Is the replication ID of this master the same advertised by the wannabe
          * slave via PSYNC? If the replication ID changed this master has a
          * different replication history, and there is no way to continue.
          *
          * Note that there are two potentially valid replication IDs: the ID1
          * and the ID2. The ID2 however is only valid up to a specific offset. */
         if (strcasecmp(master_replid, server.replid) &&
             (strcasecmp(master_replid, server.replid2) ||
              psync_offset > server.second_replid_offset))
         {
              
             /* Run id "?" is used by slaves that want to force a full resync. */
             if (master_replid[0] != '?') {
              
                 if (strcasecmp(master_replid, server.replid) &&
                     strcasecmp(master_replid, server.replid2))
                 {
              
                     serverLog(LL_NOTICE,"Partial resynchronization not accepted: "
                         "Replication ID mismatch (Replica asked for '%s', my "
                         "replication IDs are '%s' and '%s')",
                         master_replid, server.replid, server.replid2);
                 } else {
              
                     serverLog(LL_NOTICE,"Partial resynchronization not accepted: "
                         "Requested offset for second ID was %lld, but I can reply "
                         "up to %lld", psync_offset, server.second_replid_offset);
                 }
             } else {
              
                 serverLog(LL_NOTICE,"Full resync requested by replica %s",
                     replicationGetSlaveName(c));
             }
             goto need_full_resync;
         }
        
         /* We still have the data our slave is asking for? */
         if (!server.repl_backlog ||
             psync_offset < server.repl_backlog_off ||
             psync_offset > (server.repl_backlog_off + server.repl_backlog_histlen))
         {
              
             serverLog(LL_NOTICE,
                 "Unable to partial resync with replica %s for lack of backlog (Replica request was: %lld).", replicationGetSlaveName(c), psync_offset);
             if (psync_offset > server.master_repl_offset) {
              
                 serverLog(LL_WARNING,
                     "Warning: replica %s tried to PSYNC with an offset that is greater than the master replication offset.", replicationGetSlaveName(c));
             }
             goto need_full_resync;
         }
        
         /* If we reached this point, we are able to perform a partial resync:
          * 1) Set client state to make it a slave.
          * 2) Inform the client we can continue with +CONTINUE
          * 3) Send the backlog data (from the offset to the end) to the slave. */
         c->flags |= CLIENT_SLAVE;
         c->replstate = SLAVE_STATE_ONLINE;
         c->repl_ack_time = server.unixtime;
         c->repl_put_online_on_ack = 0;
         listAddNodeTail(server.slaves,c);
         /* We can't use the connection buffers since they are used to accumulate
          * new commands at this stage. But we are sure the socket send buffer is
          * empty so this write will never fail actually. */
         if (c->slave_capa & SLAVE_CAPA_PSYNC2) {
              
             buflen = snprintf(buf,sizeof(buf),"+CONTINUE %s\r\n", server.replid);
         } else {
              
             buflen = snprintf(buf,sizeof(buf),"+CONTINUE\r\n");
         }
         if (connWrite(c->conn,buf,buflen) != buflen) {
              
             freeClientAsync(c);
             return C_OK;
         }
         psync_len = addReplyReplicationBacklog(c,psync_offset);
         serverLog(LL_NOTICE,
             "Partial resynchronization request from %s accepted. Sending %lld bytes of backlog starting from offset %lld.",
                 replicationGetSlaveName(c),
                 psync_len, psync_offset);
         /* Note that we don't need to set the selected DB at server.slaveseldb
          * to -1 to force the master to emit SELECT, since the slave already
          * has this state from the previous connection with the master. */
        
         refreshGoodSlavesCount();
        
         /* Fire the replica change modules event. */
         moduleFireServerEvent(REDISMODULE_EVENT_REPLICA_CHANGE,
                               REDISMODULE_SUBEVENT_REPLICA_CHANGE_ONLINE,
                               NULL);
        
         return C_OK; /* The caller can return, no full resync needed. */
        
        need_full_resync:
         /* We need a full resync for some reason... Note that we can't
          * reply to PSYNC right now if a full SYNC is needed. The reply
          * must include the master offset at the time the RDB file we transfer
          * is generated, so we need to delay the reply to that moment. */
         return C_ERR;
        }
3.  `reolication.c#replicationSetupSlaveForFullResync()` 函数主要的作用发送 `+FULLRESYNC` 通知从节点需要进行全量复制，没有其他的逻辑
    
        int replicationSetupSlaveForFullResync(client *slave, long long offset) {
              
         char buf[128];
         int buflen;
        
         slave->psync_initial_offset = offset;
         slave->replstate = SLAVE_STATE_WAIT_BGSAVE_END;
         /* We are going to accumulate the incremental changes for this
          * slave as well. Set slaveseldb to -1 in order to force to re-emit
          * a SELECT statement in the replication stream. */
         server.slaveseldb = -1;
        
         /* Don't send this reply to slaves that approached us with
          * the old SYNC command. */
         if (!(slave->flags & CLIENT_PRE_PSYNC)) {
              
             buflen = snprintf(buf,sizeof(buf),"+FULLRESYNC %s %lld\r\n",
                               server.replid,offset);
             if (connWrite(slave->conn,buf,buflen) != buflen) {
              
                 freeClientAsync(slave);
                 return C_ERR;
             }
         }
         return C_OK;
        }
4.  `replication.c#startBgsaveForReplication()` 函数负责拉起一个保存 rdb 数据的任务，其内部根据从节点的能力确定 rdb 数据的传输形式
    
    > 1.  如果主节点配置了直接通过 socket 发送 rdb 数据并且从节点也能够处理直接通过 socket 传输过来的 rdb 数据，则调用 `rdbSaveToSlavesSockets()` 函数生成并传输 rdb 数据，否则调用 `rdbSaveBackground()` 将 rdb 数据保存到 rdb 文件里
    > 2.  如果本次会生成 rdb 文件，则将主节点保存的从节点列表中处于`SLAVE_STATE_WAIT_BGSAVE_START`状态的从节点客户端都取出来，调用 `replicationSetupSlaveForFullResync()` 函数通知其`+FULLRESYNC`，也就是将进行全量复制
    
        int startBgsaveForReplication(int mincapa) {
              
         int retval;
         int socket_target = server.repl_diskless_sync && (mincapa & SLAVE_CAPA_EOF);
         listIter li;
         listNode *ln;
        
         serverLog(LL_NOTICE,"Starting BGSAVE for SYNC with target: %s",
             socket_target ? "replicas sockets" : "disk");
        
         rdbSaveInfo rsi, *rsiptr;
         rsiptr = rdbPopulateSaveInfo(&rsi);
         /* Only do rdbSave* when rsiptr is not NULL,
          * otherwise slave will miss repl-stream-db. */
         if (rsiptr) {
              
             if (socket_target)
                 retval = rdbSaveToSlavesSockets(rsiptr);
             else
                 retval = rdbSaveBackground(server.rdb_filename,rsiptr);
         } else {
              
             serverLog(LL_WARNING,"BGSAVE for replication: replication information not available, can't generate the RDB file right now. Try later.");
             retval = C_ERR;
         }
        
         /* If we succeeded to start a BGSAVE with disk target, let's remember
          * this fact, so that we can later delete the file if needed. Note
          * that we don't set the flag to 1 if the feature is disabled, otherwise
          * it would never be cleared: the file is not deleted. This way if
          * the user enables it later with CONFIG SET, we are fine. */
         if (retval == C_OK && !socket_target && server.rdb_del_sync_files)
             RDBGeneratedByReplication = 1;
        
         /* If we failed to BGSAVE, remove the slaves waiting for a full
          * resynchronization from the list of slaves, inform them with
          * an error about what happened, close the connection ASAP. */
         if (retval == C_ERR) {
              
             serverLog(LL_WARNING,"BGSAVE for replication failed");
             listRewind(server.slaves,&li);
             while((ln = listNext(&li))) {
              
                 client *slave = ln->value;
        
                 if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_START) {
              
                     slave->replstate = REPL_STATE_NONE;
                     slave->flags &= ~CLIENT_SLAVE;
                     listDelNode(server.slaves,ln);
                     addReplyError(slave,
                         "BGSAVE failed, replication can't continue");
                     slave->flags |= CLIENT_CLOSE_AFTER_REPLY;
                 }
             }
             return retval;
         }
        
         /* If the target is socket, rdbSaveToSlavesSockets() already setup
          * the slaves for a full resync. Otherwise for disk target do it now.*/
         if (!socket_target) {
              
             listRewind(server.slaves,&li);
             while((ln = listNext(&li))) {
              
                 client *slave = ln->value;
        
                 if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_START) {
              
                         replicationSetupSlaveForFullResync(slave,
                                 getPsyncInitialOffset());
                 }
             }
         }
        
         /* Flush the script cache, since we need that slave differences are
          * accumulated without requiring slaves to match our cached scripts. */
         if (retval == C_OK) replicationScriptCacheFlush();
         return retval;
        }

#### 2.2 rdb 数据的生成与发送 ####

##### 2.2.1 直接通过 socket 发送 rdb 数据 #####

1.  `rdb.c#rdbSaveToSlavesSockets()` 函数处理可以直接通过 socket 传输 rdb 数据的场景，其重要步骤如下：
    
    > 1.  创建 pipe 管道，将读写端分别绑定到 `server.rdb_pipe_read` 和 `server.rdb_pipe_write`，并且将主节点的从节点列表中在 `SLAVE_STATE_WAIT_BGSAVE_START` 状态的从节点挑出来，调用`replicationSetupSlaveForFullResync()` 函数通知其将进行全量复制
    > 2.  调用 `redisFork()` 函数创建子进程，这里分两种情况，rdb 数据格式为 $EOF:40字节的随机字符串\\r\\n文件流内容40字节的随机字符串
    >     
    >     1.  如果现在就是在子进程中，`rioInitWithFd()` 函数将父进程管道写端`server.rdb_pipe_write` 与文件句柄 rdb 绑定，调用`rdbSaveRioWithEOFMark()`函数进行 rdb 数据的生成并将其写入管道，之后调用`sendChildCOWInfo()` 函数将生成 rdb 数据期间 `CopyOnWrite` 使用的内存相关数据传输到父进程中
    >     2.  如果现在在父进程中，其创建子进程没有出错，则通过 `aeCreateFileEvent()` 函数注册 `rdbPipeReadHandler()`函数处理管道读端 `server.rdb_pipe_read` 读取到的数据
    
        int rdbSaveToSlavesSockets(rdbSaveInfo *rsi) {
              
         listNode *ln;
         listIter li;
         pid_t childpid;
         int pipefds[2];
        
         if (hasActiveChildProcess()) return C_ERR;
        
         /* Even if the previous fork child exited, don't start a new one until we
          * drained the pipe. */
         if (server.rdb_pipe_conns) return C_ERR;
        
         /* Before to fork, create a pipe that is used to transfer the rdb bytes to
          * the parent, we can't let it write directly to the sockets, since in case
          * of TLS we must let the parent handle a continuous TLS state when the
          * child terminates and parent takes over. */
         if (pipe(pipefds) == -1) return C_ERR;
         server.rdb_pipe_read = pipefds[0];
         server.rdb_pipe_write = pipefds[1];
         anetNonBlock(NULL, server.rdb_pipe_read);
        
         /* Collect the connections of the replicas we want to transfer
          * the RDB to, which are i WAIT_BGSAVE_START state. */
         server.rdb_pipe_conns = zmalloc(sizeof(connection *)*listLength(server.slaves));
         server.rdb_pipe_numconns = 0;
         server.rdb_pipe_numconns_writing = 0;
         listRewind(server.slaves,&li);
         while((ln = listNext(&li))) {
              
             client *slave = ln->value;
             if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_START) {
              
                 server.rdb_pipe_conns[server.rdb_pipe_numconns++] = slave->conn;
                 replicationSetupSlaveForFullResync(slave,getPsyncInitialOffset());
             }
         }
        
         /* Create the child process. */
         openChildInfoPipe();
         if ((childpid = redisFork()) == 0) {
              
             /* Child */
             int retval;
             rio rdb;
        
             rioInitWithFd(&rdb,server.rdb_pipe_write);
        
             redisSetProcTitle("redis-rdb-to-slaves");
             redisSetCpuAffinity(server.bgsave_cpulist);
        
             retval = rdbSaveRioWithEOFMark(&rdb,NULL,rsi);
             if (retval == C_OK && rioFlush(&rdb) == 0)
                 retval = C_ERR;
        
             if (retval == C_OK) {
              
                 sendChildCOWInfo(CHILD_INFO_TYPE_RDB, "RDB");
             }
        
             rioFreeFd(&rdb);
             close(server.rdb_pipe_write); /* wake up the reader, tell it we're done. */
             exitFromChild((retval == C_OK) ? 0 : 1);
         } else {
              
             /* Parent */
             if (childpid == -1) {
              
                 serverLog(LL_WARNING,"Can't save in background: fork: %s",
                     strerror(errno));
        
                 /* Undo the state change. The caller will perform cleanup on
                  * all the slaves in BGSAVE_START state, but an early call to
                  * replicationSetupSlaveForFullResync() turned it into BGSAVE_END */
                 listRewind(server.slaves,&li);
                 while((ln = listNext(&li))) {
              
                     client *slave = ln->value;
                     if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_END) {
              
                         slave->replstate = SLAVE_STATE_WAIT_BGSAVE_START;
                     }
                 }
                 close(server.rdb_pipe_write);
                 close(server.rdb_pipe_read);
                 zfree(server.rdb_pipe_conns);
                 server.rdb_pipe_conns = NULL;
                 server.rdb_pipe_numconns = 0;
                 server.rdb_pipe_numconns_writing = 0;
                 closeChildInfoPipe();
             } else {
              
                 serverLog(LL_NOTICE,"Background RDB transfer started by pid %d",
                     childpid);
                 server.rdb_save_time_start = time(NULL);
                 server.rdb_child_pid = childpid;
                 server.rdb_child_type = RDB_CHILD_TYPE_SOCKET;
                 close(server.rdb_pipe_write); /* close write in parent so that it can detect the close on the child. */
                 if (aeCreateFileEvent(server.el, server.rdb_pipe_read, AE_READABLE, rdbPipeReadHandler,NULL) == AE_ERR) {
              
                     serverPanic("Unrecoverable error creating server.rdb_pipe_read file event.");
                 }
             }
             return (childpid == -1) ? C_ERR : C_OK;
         }
         return C_OK; /* Unreached. */
        }
2.  `replication.c#rdbPipeReadHandler()` 函数会读取子进程生成的 rdb 数据，并为从节点连接设置写数据的回调。该函数的处理流程如下：
    
    > 1.  首先处理管道读端读取数据出错和数据读取已经结束的情况
    > 2.  遍历等待 rdb 数据的从节点列表， 将 rdb 数据读取到 `server.rdb_pipe_buff` 中，调用 `connSetWriteHandler()` 函数在从节点连接上设置写处理函数 `rdbPipeWriteHandler()`，在其可写时发送 rdb 数据
    
        void rdbPipeReadHandler(struct aeEventLoop *eventLoop, int fd, void *clientData, int mask) {
              
         UNUSED(mask);
         UNUSED(clientData);
         UNUSED(eventLoop);
         int i;
         if (!server.rdb_pipe_buff)
             server.rdb_pipe_buff = zmalloc(PROTO_IOBUF_LEN);
         serverAssert(server.rdb_pipe_numconns_writing==0);
        
         while (1) {
              
             server.rdb_pipe_bufflen = read(fd, server.rdb_pipe_buff, PROTO_IOBUF_LEN);
             if (server.rdb_pipe_bufflen < 0) {
              
                 if (errno == EAGAIN || errno == EWOULDBLOCK)
                     return;
                 serverLog(LL_WARNING,"Diskless rdb transfer, read error sending DB to replicas: %s", strerror(errno));
                 for (i=0; i < server.rdb_pipe_numconns; i++) {
              
                     connection *conn = server.rdb_pipe_conns[i];
                     if (!conn)
                         continue;
                     client *slave = connGetPrivateData(conn);
                     freeClient(slave);
                     server.rdb_pipe_conns[i] = NULL;
                 }
                 killRDBChild();
                 return;
             }
        
             if (server.rdb_pipe_bufflen == 0) {
              
                 /* EOF - write end was closed. */
                 int stillUp = 0;
                 aeDeleteFileEvent(server.el, server.rdb_pipe_read, AE_READABLE);
                 for (i=0; i < server.rdb_pipe_numconns; i++)
                 {
              
                     connection *conn = server.rdb_pipe_conns[i];
                     if (!conn)
                         continue;
                     stillUp++;
                 }
                 serverLog(LL_WARNING,"Diskless rdb transfer, done reading from pipe, %d replicas still up.", stillUp);
                 RdbPipeCleanup();
                 return;
             }
        
             int stillAlive = 0;
             for (i=0; i < server.rdb_pipe_numconns; i++)
             {
              
                 int nwritten;
                 connection *conn = server.rdb_pipe_conns[i];
                 if (!conn)
                     continue;
        
                 client *slave = connGetPrivateData(conn);
                 if ((nwritten = connWrite(conn, server.rdb_pipe_buff, server.rdb_pipe_bufflen)) == -1) {
              
                     if (connGetState(conn) != CONN_STATE_CONNECTED) {
              
                         serverLog(LL_WARNING,"Diskless rdb transfer, write error sending DB to replica: %s",
                             connGetLastError(conn));
                         freeClient(slave);
                         server.rdb_pipe_conns[i] = NULL;
                         continue;
                     }
                     /* An error and still in connected state, is equivalent to EAGAIN */
                     slave->repldboff = 0;
                 } else {
              
                     slave->repldboff = nwritten;
                     server.stat_net_output_bytes += nwritten;
                 }
                 /* If we were unable to write all the data to one of the replicas,
                  * setup write handler (and disable pipe read handler, below) */
                 if (nwritten != server.rdb_pipe_bufflen) {
              
                     server.rdb_pipe_numconns_writing++;
                     connSetWriteHandler(conn, rdbPipeWriteHandler);
                 }
                 stillAlive++;
             }
        
             if (stillAlive == 0) {
              
                 serverLog(LL_WARNING,"Diskless rdb transfer, last replica dropped, killing fork child.");
                 killRDBChild();
                 RdbPipeCleanup();
             }
             /*  Remove the pipe read handler if at least one write handler was set. */
             if (server.rdb_pipe_numconns_writing || stillAlive == 0) {
              
                 aeDeleteFileEvent(server.el, server.rdb_pipe_read, AE_READABLE);
                 break;
             }
         }
        }
3.  `replication.c#rdbPipeWriteHandler()` 函数负责将 `server.rdb_pipe_buff` 中读取的 rdb 数据发送给从节点
    
        void rdbPipeWriteHandler(struct connection *conn) {
              
         serverAssert(server.rdb_pipe_bufflen>0);
         client *slave = connGetPrivateData(conn);
         int nwritten;
         if ((nwritten = connWrite(conn, server.rdb_pipe_buff + slave->repldboff,
                                   server.rdb_pipe_bufflen - slave->repldboff)) == -1)
         {
              
             if (connGetState(conn) == CONN_STATE_CONNECTED)
                 return; /* equivalent to EAGAIN */
             serverLog(LL_WARNING,"Write error sending DB to replica: %s",
                 connGetLastError(conn));
             freeClient(slave);
             return;
         } else {
              
             slave->repldboff += nwritten;
             server.stat_net_output_bytes += nwritten;
             if (slave->repldboff < server.rdb_pipe_bufflen)
                 return; /* more data to write.. */
         }
         rdbPipeWriteHandlerConnRemoved(conn);
        }

##### 2.2.2 发送 rdb 文件 #####

1.  `rdb.c#rdbSaveBackground()` 函数负责处理必须传输 rdb 文件的场景，其主要处理步骤如下：
    
    > 1.  `redisFork()` 创建子进程，与上文类似，如果现在在子进程中，则调用 `rdbSave()`函数保存 rdb 数据，完成后通过 `sendChildCOWInfo()` 函数将生成 rdb 数据期间 `CopyOnWrite` 使用的内存相关数据发送给父进程
    > 2.  如果现在在父进程中，则只需要更新 server 相关变量即可，比较重要的是 `server.rdb_child_pid` 变量
    
        int rdbSaveBackground(char *filename, rdbSaveInfo *rsi) {
              
         pid_t childpid;
        
         if (hasActiveChildProcess()) return C_ERR;
        
         server.dirty_before_bgsave = server.dirty;
         server.lastbgsave_try = time(NULL);
         openChildInfoPipe();
        
         if ((childpid = redisFork()) == 0) {
              
             int retval;
        
             /* Child */
             redisSetProcTitle("redis-rdb-bgsave");
             redisSetCpuAffinity(server.bgsave_cpulist);
             retval = rdbSave(filename,rsi);
             if (retval == C_OK) {
              
                 sendChildCOWInfo(CHILD_INFO_TYPE_RDB, "RDB");
             }
             exitFromChild((retval == C_OK) ? 0 : 1);
         } else {
              
             /* Parent */
             if (childpid == -1) {
              
                 closeChildInfoPipe();
                 server.lastbgsave_status = C_ERR;
                 serverLog(LL_WARNING,"Can't save in background: fork: %s",
                     strerror(errno));
                 return C_ERR;
             }
             serverLog(LL_NOTICE,"Background saving started by pid %d",childpid);
             server.rdb_save_time_start = time(NULL);
             server.rdb_child_pid = childpid;
             server.rdb_child_type = RDB_CHILD_TYPE_DISK;
             return C_OK;
         }
         return C_OK; /* unreached */
        }
2.  子进程保存 rdb 数据的函数为 `rdb.c#rdbSave()` ，其核心为调用 `rdbSaveRio()` 函数生成 rdb 数据，之后调用 `fflush()` 函数将其写入临时文件，写入完成后将临时文件重命名替换旧的 RDB 文件，此处不作具体分析
    
        int rdbSave(char *filename, rdbSaveInfo *rsi) {
              
         char tmpfile[256];
         char cwd[MAXPATHLEN]; /* Current working dir path for error messages. */
         FILE *fp;
         rio rdb;
         int error = 0;
        
         snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
         fp = fopen(tmpfile,"w");
         if (!fp) {
              
             char *cwdp = getcwd(cwd,MAXPATHLEN);
             serverLog(LL_WARNING,
                 "Failed opening the RDB file %s (in server root dir %s) "
                 "for saving: %s",
                 filename,
                 cwdp ? cwdp : "unknown",
                 strerror(errno));
             return C_ERR;
         }
        
         rioInitWithFile(&rdb,fp);
         startSaving(RDBFLAGS_NONE);
        
         if (server.rdb_save_incremental_fsync)
             rioSetAutoSync(&rdb,REDIS_AUTOSYNC_BYTES);
        
         if (rdbSaveRio(&rdb,&error,RDBFLAGS_NONE,rsi) == C_ERR) {
              
             errno = error;
             goto werr;
         }
        
         /* Make sure data will not remain on the OS's output buffers */
         if (fflush(fp) == EOF) goto werr;
         if (fsync(fileno(fp)) == -1) goto werr;
         if (fclose(fp) == EOF) goto werr;
        
         /* Use RENAME to make sure the DB file is changed atomically only
          * if the generate DB file is ok. */
         if (rename(tmpfile,filename) == -1) {
              
             char *cwdp = getcwd(cwd,MAXPATHLEN);
             serverLog(LL_WARNING,
                 "Error moving temp DB file %s on the final "
                 "destination %s (in server root dir %s): %s",
                 tmpfile,
                 filename,
                 cwdp ? cwdp : "unknown",
                 strerror(errno));
             unlink(tmpfile);
             stopSaving(0);
             return C_ERR;
         }
        
         serverLog(LL_NOTICE,"DB saved on disk");
         server.dirty = 0;
         server.lastsave = time(NULL);
         server.lastbgsave_status = C_OK;
         stopSaving(1);
         return C_OK;
        
        werr:
         serverLog(LL_WARNING,"Write error saving DB on disk: %s", strerror(errno));
         fclose(fp);
         unlink(tmpfile);
         stopSaving(0);
         return C_ERR;
        }
3.  在子进程保存 rdb 数据时，redis 的定时任务 `server.c#serverCron()` 会周期性地去检查这个动作是否完成，关键函数为 `checkChildrenDone()`
    
        int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
              
         
         ......
         
         /* Check if a background saving or AOF rewrite in progress terminated. */
         if (hasActiveChildProcess() || ldbPendingChildren())
         {
              
             checkChildrenDone();
         } 
         ......
        }
4.  `server.c#checkChildrenDone()`函数的逻辑较为简单，可以归纳为以下几步：
    
    > 1.  `wait3()` 函数接收子进程结束的信号，根据子进程的 pid 进行不同的处理
    > 2.  对于 `server.rdb_child_pid` 子进程，其处理函数为 `backgroundSaveDoneHandler()`
    
        void checkChildrenDone(void) {
              
         int statloc;
         pid_t pid;
        
         /* If we have a diskless rdb child (note that we support only one concurrent
          * child), we want to avoid collecting it's exit status and acting on it
          * as long as we didn't finish to drain the pipe, since then we're at risk
          * of starting a new fork and a new pipe before we're done with the previous
          * one. */
         if (server.rdb_child_pid != -1 && server.rdb_pipe_conns)
             return;
        
         if ((pid = wait3(&statloc,WNOHANG,NULL)) != 0) {
              
             int exitcode = WEXITSTATUS(statloc);
             int bysignal = 0;
        
             if (WIFSIGNALED(statloc)) bysignal = WTERMSIG(statloc);
        
             /* sigKillChildHandler catches the signal and calls exit(), but we
              * must make sure not to flag lastbgsave_status, etc incorrectly.
              * We could directly terminate the child process via SIGUSR1
              * without handling it, but in this case Valgrind will log an
              * annoying error. */
             if (exitcode == SERVER_CHILD_NOERROR_RETVAL) {
              
                 bysignal = SIGUSR1;
                 exitcode = 1;
             }
        
             if (pid == -1) {
              
                 serverLog(LL_WARNING,"wait3() returned an error: %s. "
                     "rdb_child_pid = %d, aof_child_pid = %d, module_child_pid = %d",
                     strerror(errno),
                     (int) server.rdb_child_pid,
                     (int) server.aof_child_pid,
                     (int) server.module_child_pid);
             } else if (pid == server.rdb_child_pid) {
              
                 backgroundSaveDoneHandler(exitcode,bysignal);
                 if (!bysignal && exitcode == 0) receiveChildInfo();
             } else if (pid == server.aof_child_pid) {
              
                 backgroundRewriteDoneHandler(exitcode,bysignal);
                 if (!bysignal && exitcode == 0) receiveChildInfo();
             } else if (pid == server.module_child_pid) {
              
                 ModuleForkDoneHandler(exitcode,bysignal);
                 if (!bysignal && exitcode == 0) receiveChildInfo();
             } else {
              
                 if (!ldbRemoveChild(pid)) {
              
                     serverLog(LL_WARNING,
                         "Warning, detected child with unmatched pid: %ld",
                         (long)pid);
                 }
             }
             updateDictResizePolicy();
             closeChildInfoPipe();
         }
        }
5.  `rdb.c#backgroundSaveDoneHandler()` 函数其实只是个转发，其核心是根据 rdb 数据的传输形式调用不同的处理函数，此处为调用 `backgroundSaveDoneHandlerDisk()` 函数
    
    > `backgroundSaveDoneHandlerDisk()` 函数主要更新了一些变量，最重要的是调用 `updateSlavesWaitingBgsave()` 函数启动 rdb 文件的传输
    
        void backgroundSaveDoneHandler(int exitcode, int bysignal) {
              
         switch(server.rdb_child_type) {
              
         case RDB_CHILD_TYPE_DISK:
             backgroundSaveDoneHandlerDisk(exitcode,bysignal);
             break;
         case RDB_CHILD_TYPE_SOCKET:
             backgroundSaveDoneHandlerSocket(exitcode,bysignal);
             break;
         default:
             serverPanic("Unknown RDB child type.");
             break;
         }
        }
        
        void backgroundSaveDoneHandlerDisk(int exitcode, int bysignal) {
              
         if (!bysignal && exitcode == 0) {
              
             serverLog(LL_NOTICE,
                 "Background saving terminated with success");
             server.dirty = server.dirty - server.dirty_before_bgsave;
             server.lastsave = time(NULL);
             server.lastbgsave_status = C_OK;
         } else if (!bysignal && exitcode != 0) {
              
             serverLog(LL_WARNING, "Background saving error");
             server.lastbgsave_status = C_ERR;
         } else {
              
             mstime_t latency;
        
             serverLog(LL_WARNING,
                 "Background saving terminated by signal %d", bysignal);
             latencyStartMonitor(latency);
             rdbRemoveTempFile(server.rdb_child_pid);
             latencyEndMonitor(latency);
             latencyAddSampleIfNeeded("rdb-unlink-temp-file",latency);
             /* SIGUSR1 is whitelisted, so we have a way to kill a child without
              * tirggering an error condition. */
             if (bysignal != SIGUSR1)
                 server.lastbgsave_status = C_ERR;
         }
         server.rdb_child_pid = -1;
         server.rdb_child_type = RDB_CHILD_TYPE_NONE;
         server.rdb_save_time_last = time(NULL)-server.rdb_save_time_start;
         server.rdb_save_time_start = -1;
         /* Possibly there are slaves waiting for a BGSAVE in order to be served
          * (the first stage of SYNC is a bulk transfer of dump.rdb) */
         updateSlavesWaitingBgsave((!bysignal && exitcode == 0) ? C_OK : C_ERR, RDB_CHILD_TYPE_DISK);
        }
6.  `replication.c#updateSlavesWaitingBgsave()` 函数比较简单，其核心在于将 rdb文件读取出来，存入所有等待 rdb 文件的从节点客户端`slave->repldbfd`变量，并为其连接设置写处理函数 `sendBulkToSlave()`，从而完成 rdb 文件传输
    
        void updateSlavesWaitingBgsave(int bgsaveerr, int type) {
              
         listNode *ln;
         int startbgsave = 0;
         int mincapa = -1;
         listIter li;
        
         listRewind(server.slaves,&li);
         while((ln = listNext(&li))) {
              
             client *slave = ln->value;
        
             if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_START) {
              
                 startbgsave = 1;
                 mincapa = (mincapa == -1) ? slave->slave_capa :
                                             (mincapa & slave->slave_capa);
             } else if (slave->replstate == SLAVE_STATE_WAIT_BGSAVE_END) {
              
                 struct redis_stat buf;
        
                 if (bgsaveerr != C_OK) {
              
                     freeClient(slave);
                     serverLog(LL_WARNING,"SYNC failed. BGSAVE child returned an error");
                     continue;
                 }
        
                 /* If this was an RDB on disk save, we have to prepare to send
                  * the RDB from disk to the slave socket. Otherwise if this was
                  * already an RDB -> Slaves socket transfer, used in the case of
                  * diskless replication, our work is trivial, we can just put
                  * the slave online. */
                 if (type == RDB_CHILD_TYPE_SOCKET) {
              
                     serverLog(LL_NOTICE,
                         "Streamed RDB transfer with replica %s succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming",
                             replicationGetSlaveName(slave));
                     /* Note: we wait for a REPLCONF ACK message from the replica in
                      * order to really put it online (install the write handler
                      * so that the accumulated data can be transferred). However
                      * we change the replication state ASAP, since our slave
                      * is technically online now.
                      *
                      * So things work like that:
                      *
                      * 1. We end trasnferring the RDB file via socket.
                      * 2. The replica is put ONLINE but the write handler
                      *    is not installed.
                      * 3. The replica however goes really online, and pings us
                      *    back via REPLCONF ACK commands.
                      * 4. Now we finally install the write handler, and send
                      *    the buffers accumulated so far to the replica.
                      *
                      * But why we do that? Because the replica, when we stream
                      * the RDB directly via the socket, must detect the RDB
                      * EOF (end of file), that is a special random string at the
                      * end of the RDB (for streamed RDBs we don't know the length
                      * in advance). Detecting such final EOF string is much
                      * simpler and less CPU intensive if no more data is sent
                      * after such final EOF. So we don't want to glue the end of
                      * the RDB trasfer with the start of the other replication
                      * data. */
                     slave->replstate = SLAVE_STATE_ONLINE;
                     slave->repl_put_online_on_ack = 1;
                     slave->repl_ack_time = server.unixtime; /* Timeout otherwise. */
                 } else {
              
                     if ((slave->repldbfd = open(server.rdb_filename,O_RDONLY)) == -1 ||
                         redis_fstat(slave->repldbfd,&buf) == -1) {
              
                         freeClient(slave);
                         serverLog(LL_WARNING,"SYNC failed. Can't open/stat DB after BGSAVE: %s", strerror(errno));
                         continue;
                     }
                     slave->repldboff = 0;
                     slave->repldbsize = buf.st_size;
                     slave->replstate = SLAVE_STATE_SEND_BULK;
                     slave->replpreamble = sdscatprintf(sdsempty(),"$%lld\r\n",
                         (unsigned long long) slave->repldbsize);
        
                     connSetWriteHandler(slave->conn,NULL);
                     if (connSetWriteHandler(slave->conn,sendBulkToSlave) == C_ERR) {
              
                         freeClient(slave);
                         continue;
                     }
                 }
             }
         }
         if (startbgsave) startBgsaveForReplication(mincapa);
        }
7.  `replication.c#sendBulkToSlave()` 函数的逻辑很简单，可以总结为以下几步：
    
    > 1.  将从节点客户端 `slave->repldbfd` 存储的 rdb 文件数据写给从节点
    > 2.  调用 `putSlaveOnline()` 函数设置从节点状态为 `SLAVE_STATE_ONLINE`
    
        void sendBulkToSlave(connection *conn) {
              
         client *slave = connGetPrivateData(conn);
         char buf[PROTO_IOBUF_LEN];
         ssize_t nwritten, buflen;
        
         /* Before sending the RDB file, we send the preamble as configured by the
          * replication process. Currently the preamble is just the bulk count of
          * the file in the form "$<length>\r\n". */
         if (slave->replpreamble) {
              
             nwritten = connWrite(conn,slave->replpreamble,sdslen(slave->replpreamble));
             if (nwritten == -1) {
              
                 serverLog(LL_VERBOSE,
                     "Write error sending RDB preamble to replica: %s",
                     connGetLastError(conn));
                 freeClient(slave);
                 return;
             }
             server.stat_net_output_bytes += nwritten;
             sdsrange(slave->replpreamble,nwritten,-1);
             if (sdslen(slave->replpreamble) == 0) {
              
                 sdsfree(slave->replpreamble);
                 slave->replpreamble = NULL;
                 /* fall through sending data. */
             } else {
              
                 return;
             }
         }
        
         /* If the preamble was already transferred, send the RDB bulk data. */
         lseek(slave->repldbfd,slave->repldboff,SEEK_SET);
         buflen = read(slave->repldbfd,buf,PROTO_IOBUF_LEN);
         if (buflen <= 0) {
              
             serverLog(LL_WARNING,"Read error sending DB to replica: %s",
                 (buflen == 0) ? "premature EOF" : strerror(errno));
             freeClient(slave);
             return;
         }
         if ((nwritten = connWrite(conn,buf,buflen)) == -1) {
              
             if (connGetState(conn) != CONN_STATE_CONNECTED) {
              
                 serverLog(LL_WARNING,"Write error sending DB to replica: %s",
                     connGetLastError(conn));
                 freeClient(slave);
             }
             return;
         }
         slave->repldboff += nwritten;
         server.stat_net_output_bytes += nwritten;
         if (slave->repldboff == slave->repldbsize) {
              
             close(slave->repldbfd);
             slave->repldbfd = -1;
             connSetWriteHandler(slave->conn,NULL);
             putSlaveOnline(slave);
         }
        }
8.  `replication.c#putSlaveOnline()` 函数逻辑简单明了，主要做了以下两件事，至此 Master 节点主从同步结束
    
    > 1.  设置从节点同步状态 `slave->replstate` 为 `SLAVE_STATE_ONLINE`
    > 2.  调用 `connSetWriteHandler()` 函数为从节点连接设置写函数 `sendReplyToClient()`，将主节点保存 rdb 期间执行过的操作命令同步到从节点
    
        void putSlaveOnline(client *slave) {
              
         slave->replstate = SLAVE_STATE_ONLINE;
         slave->repl_put_online_on_ack = 0;
         slave->repl_ack_time = server.unixtime; /* Prevent false timeout. */
         if (connSetWriteHandler(slave->conn, sendReplyToClient) == C_ERR) {
              
             serverLog(LL_WARNING,"Unable to register writable event for replica bulk transfer: %s", strerror(errno));
             freeClient(slave);
             return;
         }
         refreshGoodSlavesCount();
         /* Fire the replica change modules event. */
         moduleFireServerEvent(REDISMODULE_EVENT_REPLICA_CHANGE,
                               REDISMODULE_SUBEVENT_REPLICA_CHANGE_ONLINE,
                               NULL);
         serverLog(LL_NOTICE,"Synchronization with replica %s succeeded",
             replicationGetSlaveName(slave));
        }

[Redis]: https://blog.csdn.net/weixin_45505313/article/details/99714605#3_Redis__68
[20201106113036907.jpg_pic_center]: https://img-blog.csdnimg.cn/20201106113036907.jpg#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTUwNTMxMw_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20201105180811590.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTUwNTMxMw==,size_16,color_FFFFFF,t_70#pic_center