[Hadoop] Zookeeper伪分布式集群部署

爱被打了一巴掌 2022-05-15 08:47 350阅读 0赞

# 1. Zookeeper集群概念 #

ZK集群：3个节点，一个leader，2个follower

# 2. 伪分布式集群部署 #

伪分布式：1台机器上面搭建多个ZK节点

### 2.1 解压安装zk ###

解压zookeeper安装包并拷贝成三份，也就是三个zk节点

[hadoop@hadoop000 app]$ cp -r zookeeper-3.4.5-cdh5.7.0 zk001
    [hadoop@hadoop000 app]$ cp -r zookeeper-3.4.5-cdh5.7.0 zk002
    [hadoop@hadoop000 app]$ cp -r zookeeper-3.4.5-cdh5.7.0 zk003

## 2.2 修改zk配置文件 ##

分别修改三个zk节点的配置文件

# zk001
    [hadoop@hadoop000 app]$ vi zk001/conf/zoo.cfg
    dataDir=/home/hadoop/tmp/zookeeper001
    server.1=localhost:2888:3888
    server.2=localhost:2889:3889
    server.3=localhost:2890:3890
    clientPort=2181
    
    # zk002
    [hadoop@hadoop000 app]$ vi zk002/conf/zoo.cfg
    dataDir=/home/hadoop/tmp/zookeeper002
    server.1=localhost:2888:3888
    server.2=localhost:2889:3889
    server.3=localhost:2890:3890
    clientPort=2182
    
    # zk003
    [hadoop@hadoop000 app]$ vi zk003/conf/zoo.cfg
    dataDir=/home/hadoop/tmp/zookeeper003
    server.1=localhost:2888:3888
    server.2=localhost:2889:3889
    server.3=localhost:2890:3890
    clientPort=2183
    
    # 其中，2888是服务器与zk leader(master)通信的端口，3888为leader挂掉时，重新选举leader的端口。伪分布式部署时3个zk节点都在同一台机器上，为了端口不发生冲突，所以各节点的端口号依次+1

为每个zk节点创建dataDir文件夹

[hadoop@hadoop000 app]$ mkdir ~/tmp/zookeeper001
    [hadoop@hadoop000 app]$ mkdir ~/tmp/zookeeper002
    [hadoop@hadoop000 app]$ mkdir ~/tmp/zookeeper003

为每个zk节点创建myid

[hadoop@hadoop000 app]$ echo 1 > ~/tmp/zookeeper001/myid
    [hadoop@hadoop000 app]$ echo 2 > ~/tmp/zookeeper002/myid
    [hadoop@hadoop000 app]$ echo 3 > ~/tmp/zookeeper003/myid

## 2.3 启动3个zk节点 ##

启动3个zk节点，每个节点会产生一个QuorumPeerMain进程

# 启动zk001
    [hadoop@hadoop000 app]$ zk001/bin/zkServer.sh start
    [hadoop@hadoop000 app]$ jps
    13919 QuorumPeerMain
    
    # 启动zk002
    [hadoop@hadoop000 app]$ zk002/bin/zkServer.sh start
    [hadoop@hadoop000 app]$ jps
    13969 QuorumPeerMain
    13919 QuorumPeerMain
    
    # 启动zk003
    [hadoop@hadoop000 app]$ zk003/bin/zkServer.sh start
    [hadoop@hadoop000 app]$ jps
    13969 QuorumPeerMain
    14036 QuorumPeerMain
    13919 QuorumPeerMain
    [hadoop@hadoop000 app]$

## 2.4 查看节点状态 ##

查看每个节点状态，可以看到本次启动后，zk002是leader，zk001和zk003是follower

[hadoop@hadoop000 app]$ zk001/bin/zkServer.sh status
    JMX enabled by default
    Using config: /home/hadoop/app/zk001/bin/../conf/zoo.cfg
    Mode: follower
    [hadoop@hadoop000 app]$ zk002/bin/zkServer.sh status
    JMX enabled by default
    Using config: /home/hadoop/app/zk002/bin/../conf/zoo.cfg
    Mode: leader
    [hadoop@hadoop000 app]$ zk003/bin/zkServer.sh status
    JMX enabled by default
    Using config: /home/hadoop/app/zk003/bin/../conf/zoo.cfg
    Mode: follower
    [hadoop@hadoop000 app]$

## 2.5 测试集群数据 ##

从节点zk001启动一个zk client，连接到zk server 2182端口（zk002)。并且创建一个节点，写入数据。

[hadoop@hadoop000 app]$ zk001/bin/zkCli.sh -server localhost:2182
    [zk: localhost:2182(CONNECTED) 0] ls /
    [zookeeper]
    [zk: localhost:2182(CONNECTED) 1] create /test hellozk002
    Created /test
    [zk: localhost:2182(CONNECTED) 2] get /test
    hellozk002
    cZxid = 0x100000002
    ctime = Thu Aug 16 19:53:03 CST 2018
    mZxid = 0x100000002
    mtime = Thu Aug 16 19:53:03 CST 2018
    pZxid = 0x100000002
    cversion = 0
    dataVersion = 0
    aclVersion = 0
    ephemeralOwner = 0x0
    dataLength = 10
    numChildren = 0
    [zk: localhost:2182(CONNECTED) 3]

去节点zk002启动一个zk client，连接zk server 2183端口（zk003），可以查看到刚才创建的test节点和数据。

[hadoop@hadoop000 app]$ zk002/bin/zkCli.sh -server localhost:2183
    [zk: localhost:2183(CONNECTED) 0] ls /
    [zookeeper, test]
    [zk: localhost:2183(CONNECTED) 1] get /test
    hellozk002
    cZxid = 0x100000002
    ctime = Thu Aug 16 19:53:03 CST 2018
    mZxid = 0x100000002
    mtime = Thu Aug 16 19:53:03 CST 2018
    pZxid = 0x100000002
    cversion = 0
    dataVersion = 0
    aclVersion = 0
    ephemeralOwner = 0x0
    dataLength = 10
    numChildren = 0
    [zk: localhost:2183(CONNECTED) 2]

## 2.6 测试节点挂掉情况 ##

将leader节点zk002进程停止，查看集群中各节点选举情况

[hadoop@hadoop000 app]$ kill -9 13969
    
    # 此时zk003变成了leader
    [hadoop@hadoop000 app]$ zk002/bin/zkServer.sh status
    JMX enabled by default
    Using config: /home/hadoop/app/zk002/bin/../conf/zoo.cfg
    Error contacting service. It is probably not running.
    [hadoop@hadoop000 app]$ zk003/bin/zkServer.sh status
    JMX enabled by default
    Using config: /home/hadoop/app/zk003/bin/../conf/zoo.cfg
    Mode: leader
    [hadoop@hadoop000 app]$

集群挂掉后数据还是可以查到，集群内部容错性是可以保证的。

# 3. 分布式集群部署 #

真正的分布式集群部署，过程基本一致，只是不同节点上的ZK路径完全一致，配置文件要使用同样的端口号。

[hadoop@hadoop000 app]$ vi zookeeper-3.4.5-cdh5.7.0/conf/zoo.cfg
    dataDir=/home/hadoop/tmp/zookeeper
    server.1=hadoop000:2888:3888
    server.2=hadoop001:2888:3888
    server.3=hadoop002:2888:3888
    clientPort=2181

每个节点上的myid还是按照上面伪分布式部署的方式，myid号每个节点不同。其他的步骤与伪分布式一模一样。