etcd网络通讯层(一)——rafthttp中重要的接口

不念不忘少年蓝@ 2022-04-24 20:14 210阅读 0赞

前面我们提到etcd-raft模块，etcd-raft模块并未提供网络层的相关实现，而是将待发送的消息封装进Ready实例返回给上层模块，然后由上层模块决定如何将这些消息发送到集群中的其他节点。etcd将网络层的相关实现单独封装成一个模块，也就是etcd-rafthttp模块。从而降低了etcd-raft和etcd-rafthttp之间的耦合，提高程序的可扩展性。

etcd-rafthttp的模块主要在[https://github.com/etcd-io/etcd/tree/master/etcdserver/api/rafthttp][https_github.com_etcd-io_etcd_tree_master_etcdserver_api_rafthttp] 目录下。

在etcd集群中，每个节点启动时都会与集群中的其他节点建立连接(每个节点即是服务端,也是客户端)，这里以三个节点为例，最终形成如图的网络结构。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NraDIwMTVqYXZh_size_16_color_FFFFFF_t_70][]

在etcd-rafthttp模块中，主要使用两个消息传输通道，Stream和Pipeline消息通道。

Stream消息通道维护的HTTP长连接，主要负责传输数据量较小、发送比较频繁的消息，例如，MsgApp消息、MsgHeartbeat消息、MsgVote消息等。

而Pipeline消息通道在传输数据完成后立即关闭连接，主要负责传输数据量较大、发送频率较低的消息，例如，MsgSnap消息等。

### 一、rafthttp.Transporter接口 ###

Transporter接口是rafthttp包的核心接口，它定义了etcd网络层的核心功能，其具体定义如下：

type Transporter interface {
    	// Start starts the given Transporter.
    	// Start MUST be called before calling other functions in the interface.
    	Start() error  //初始化操作
    	// Handler returns the HTTP handler of the transporter.
    	// A transporter HTTP handler handles the HTTP requests
    	// from remote peers.
    	// The handler MUST be used to handle RaftPrefix(/raft)
    	// endpoint.
    	Handler() http.Handler		//创建Handler实例，并关联到指定URL上
    	// Send sends out the given messages to the remote peers.
    	// Each message has a To field, which is an id that maps
    	// to an existing peer in the transport.
    	// If the id cannot be found in the transport, the message
    	// will be ignored.
    	Send(m []raftpb.Message)  //发送消息
    	// SendSnapshot sends out the given snapshot message to a remote peer.
    	// The behavior of SendSnapshot is similar to Send.
    	SendSnapshot(m snap.Message)   //发送快照数据
    	// AddRemote adds a remote with given peer urls into the transport.
    	// A remote helps newly joined member to catch up the progress of cluster,
    	// and will not be used after that.
    	// It is the caller's responsibility to ensure the urls are all valid,
    	// or it panics.
    	AddRemote(id types.ID, urls []string)   //在集群中添加一个节点时，其他节点会通过该方法添加该新加入节点的信息
    	// AddPeer adds a peer with given peer urls into the transport.
    	// It is the caller's responsibility to ensure the urls are all valid,
    	// or it panics.
    	// Peer urls are used to connect to the remote peer.
    	//Peer接口是当前节点对集群中其他节点的抽象表示，而结构体Peer则是Peer接口的一个具体实现
    	//下面几个方法是对Peer的操作
    	AddPeer(id types.ID, urls []string)
    	// RemovePeer removes the peer with given id.
    	RemovePeer(id types.ID)
    	// RemoveAllPeers removes all the existing peers in the transport.
    	RemoveAllPeers()
    	// UpdatePeer updates the peer urls of the peer with the given id.
    	// It is the caller's responsibility to ensure the urls are all valid,
    	// or it panics.
    	UpdatePeer(id types.ID, urls []string)
    	// ActiveSince returns the time that the connection with the peer
    	// of the given id becomes active.
    	// If the connection is active since peer was added, it returns the adding time.
    	// If the connection is currently inactive, it returns zero time.
    	ActiveSince(id types.ID) time.Time
    	// ActivePeers returns the number of active peers.
    	ActivePeers() int
    	// Stop closes the connections and stops the transporter.
    	Stop()  //关闭操作
    }

Start()方法完成初始化操作

Handler() 创建Handler实例，并关联到指定的URL上，创建HTTP服务时使用

Send()方法用于给对端发送消息

SendSnapshot()用于向对端发送快照数据

Peer接口是当前节点对集群中其他节点(对端)的抽象表示,AddPeer、RemovePeer、RemoveAllPeer、UpdatePeer分别用于添加节点、删除节点、删除所有节点和更新节点的操作。

### 二、Raft接口  ###

Raft接口也是rafthttp包中非常重要的接口，用于集群中当前节点的网络层(etcd-rafthttp)向etcd-raft层发送消息。

type Raft interface {
    	Process(ctx context.Context, m raftpb.Message) error  //将指定消息传递到etcd-raft模块进行处理
    	IsIDRemoved(id uint64) bool     //检测当前节点是否从当前集群中被移除
    	ReportUnreachable(id uint64)  //通知底层的etcd-raft模块，当前节点与指定的节点无法连通
    	ReportSnapshot(id uint64, status raft.SnapshotStatus)  //通知底层的etcd-raft模块，快照数据是否发送成功
    }

Raft接口提供了当前节点的网络层向底层的etcd-raft模块发送消息的功能

### 三、Peer接口 ###

type Peer interface {
    	// send sends the message to the remote peer. The function is non-blocking
    	// and has no promise that the message will be received by the remote.
    	// When it fails to send message out, it will report the status to underlying
    	// raft.
    	//发送单个消息给对端，该方法时非阻塞的，如果出现发送失败，则会将失败信息报告给底层的Raft接口
    	send(m raftpb.Message)
    
    	// sendSnap sends the merged snapshot message to the remote peer. Its behavior
    	// is similar to send.
    	//发送snap.Message，其他行为与上面的send()方法类似
    	sendSnap(m snap.Message)
    
    	// update updates the urls of remote peer.
    	//更新对应节点暴露的URL地址
    	update(urls types.URLs)
    
    	// attachOutgoingConn attaches the outgoing connection to the peer for
    	// stream usage. After the call, the ownership of the outgoing
    	// connection hands over to the peer. The peer will close the connection
    	// when it is no longer used.
    	/*
    	将指定连接与当前的Peer绑定，Peer会将该连接作为Stream消息通道使用
    	当Peer不再使用该连接时，会将该连接关闭
    	*/
    	attachOutgoingConn(conn *outgoingConn)
    	// activeSince returns the time that the connection with the
    	// peer becomes active.
    	/*
    	activeSince返回与其他端(peer)的连接变为active状态的时间。
    	*/
    	activeSince() time.Time
    	// stop performs any necessary finalization and terminates the peer
    	// elegantly.
    	/*
    	关闭当前Peer实例，会关闭底层的网络连接
    	*/
    	stop()
    }

Peer接口是当前节点对集群中其他节点(对端)的抽象表示，该接口主要作用是向对端(集群中的其他节点)发送消息的功能。

后面会对这些接口的实现进行详细分析。

[https_github.com_etcd-io_etcd_tree_master_etcdserver_api_rafthttp]: https://github.com/etcd-io/etcd/tree/master/etcdserver/api/rafthttp
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NraDIwMTVqYXZh_size_16_color_FFFFFF_t_70]: /images/20220212/b63f6ef7ebcc4dc1b232d8e67c2da6d5.png