hadoop HA分布式集群搭建

ゝ一纸荒年。 2022-05-28 08:12 329阅读 0赞

本文所用软件版本及下载地址：  
zookeeper : zookeeper-3.3.6  
hadoop : hadoop-2.7.4  
链接：[https://pan.baidu.com/s/14InWWh2MVWw-0CIDMcypSA][https_pan.baidu.com_s_14InWWh2MVWw-0CIDMcypSA] 密码：1l98  
如果有不对的地方，请大家指正。谢谢 。  
集群搭建步骤：  
**1.集群规划**  
![这里写图片描述][70]  
**2.配置hosts文件**  
vi /etc/hosts  
我的配置是：

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    192.168.0.11 store01
    192.168.0.12 store02
    192.168.0.13 store03

**3.修改hostname**  
每台机器都需要修改，和hosts中IP相对应的名称  
**4.关闭防火墙**  
我是centos7 64位。  
service firewalld stop  
设置开机不启动  
chkconfig firewalld off  
**5.配置SSH免密码登录**  
在三台机器上执行：  
ssh-keygen -t rsa  
三台机器都需要使用相同的用户执行，我是直接用的root用户生成的文件，生成的文件在~/.ssh下面。  
然后在三台机器上执行：  
ssh-copy-id store01;  
ssh-copy-id store02;  
ssh-copy-id store03  
**6.Zookeeper安装**  
(1)进入Zookeeper配置文件，拷贝一份配置文件命名为zoo.cfg并配置

cd /home/zookeeper-3.3.6/conf/
    cp zoo_sample.cfg zoo.cfg
    vi zoo.cfg

我的zoo.cfg配置为：

# The number of milliseconds of each tick
    tickTime=2000
    # The number of ticks that the initial 
    # synchronization phase can take
    initLimit=10
    # The number of ticks that can pass between 
    # sending a request and getting an acknowledgement
    syncLimit=5
    # the directory where the snapshot is stored.
    dataDir=/home/softdata/zookeeper
    # the port at which the clients will connect
    clientPort=2181
    server.1=store01:2888:3888
    server.2=store02:2888:3888
    server.3=store03:2888:3888

(2)在dataDir目录下新建一个文件，文件名为myid，里面的值与配置文件中的值相对应。例如我的store01的机器，配置文件中对应的为1，在myid中写上1。其余的机器也按照这中规则。  
(3)将配置好的zookeeper拷贝到其他机器上，包括dataDir对应的目录，使用scp,以拷贝到store02节点为例

scp -r /home/zookeeper-3.3.6/ root@store02:/home/
    scp -r /home/softdata/zookeeper/ root@store02:/home/softdata/zookeeper/

将store02下，Zookeeper对应的myid文件内容修改为2。store03也这样操作。  
**7.hadoop 中配置JAVA\_HOME**  
在hadoop-env.sh、mapred-env.sh、yarn-env.sh中配置JAVA\_HOME的路径，有些是使用\#号键注释了的，记得去掉\#号。  
**8.hdfs-site.xml**  
这里配置的集群名称为mycluster

<configuration>
    <property>
      <name>dfs.nameservices</name>
      <value>mycluster</value>
    </property>
    <property>
      <name>dfs.ha.namenodes.mycluster</name>
      <value>nn1,nn2</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn1</name>
      <value>store01:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn2</name>
      <value>store02:8020</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>store01:50070</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>store01:50070</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>store02:50070</value>
    </property>
    <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://store01:8485;store02:8485;store03:8485/mycluster</value>
    </property>
    <property>
      <name>dfs.client.failover.proxy.provider.mycluster</name>
     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>
    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/root/.ssh/id_rsa</value>
    </property>
    <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/home/softdata/hadoop/data</value>
    </property>
    <property>
       <name>dfs.ha.automatic-failover.enabled</name>
       <value>true</value>
    </property>
    </configuration>

**9.配置core-site.xml**

<configuration>
    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://mycluster</value>
    </property>
    <property>
       <name>ha.zookeeper.quorum</name>
       <value>store01:2181,store02:2181,store03:2181</value>
    </property>
    <property>
       <name>hadoop.tmp.dir</name>
       <value>/home/softdata/hadoop/tmp</value>
    </property>
    </configuration>

**10.配置mapred-site.xml**

<configuration>
    <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>store01:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>store01:19888</value>
    </property>
    </configuration>

**11.配置yarn-site.xml**

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>store03</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>106800</value>
    </property>
    </configuration>

**12.配置slaves**  
在slaves文件中写上DataNode节点名称，我的配置如下：

store01
    store02
    store03

**13.将Hadoop拷贝到其他机器上去**  
使用scp命令，将配置好的Hadoop拷贝到其他机器上去。  
**14.启动Hadoop**  
(1)启动zookeeper  
为了方便，我建立了一个软链  
ln -s /home/zookeeper-3.3.6/bin/zkServer.sh /usr/local/bin/zkServer  
启动(三台机器都需要启动，启动完成后可用jps命令查看是否启动成功，会有QuorumPeerMain启动)：  
zkServer start  
(2)启动三个JournalNode  
sbin/hadoop-deamon.sh start journalnode  
(3)格式化namenode  
在第一台机器上格式化：  
bin/hdfs namenode -format  
启动刚格式化好的namenode:  
sbin/hadoop-deamon.sh start namenode  
在第二台机器上同步namenode数据：  
bin/hdfs namenode -bootstrapStandby  
启动第二台的namenode:  
sbin/hadoop-deamon.sh start namenode  
(4)格式化Zookeeper  
在其中一个namenode上执行zkfc初始化：  
hdfs zkfc -formatZK  
(5)停止上面的节点：  
sbin/stop-dfs.sh  
(6)全部启动  
sbin/start-all.sh  
(7)web访问  
浏览器访问地址：  
[http://192.168.0.11:50070/dfshealth.html\#tab-overview][http_192.168.0.11_50070_dfshealth.html_tab-overview]  
[http://192.168.0.12:50070/dfshealth.html\#tab-overview][http_192.168.0.12_50070_dfshealth.html_tab-overview]  
有一个节点应为active,另一个节点状态为standby  
**问题：**  
**1. store03上ResourceManager无法正常启动的问题**  
**描述：**从Master上启动集群时，store03上的ResourceManager没有启动  
\*\*原因：\*\*Namenode和ResourceManger如果不是同一台机器，不能在NameNode上启动 yarn，应该在ResouceManager所在的机器上启动yarn。  
启动命令： **sbin/start-yarn.sh**  
2.浏览器访问看到两个状态都为standby  
(1)可能是zookeeper没有成功启动，使用jps查看是否有QuorumPeerMain存在，如果不存在，启动ZK  
(2)jps查看是否有DFSZKFailoverController，如果没有，尝试启动或重新格式化一下ZK  
启动ZK选举制度：  
sbin/hadoop-daemon.sh start zkfc  
(3)检查namenode是否格式化  
(4)手动切换  
bin/hdfs haadmin -transitionToActive nn1 \#\#切换成active  
bin/hdfs haadmin -transitionToStandby nn1 \#\#切换成standby  
注： 如果不让你切换的时候，bin/hdfs haadmin -transitionToActive nn2 –forceactive  
也可以直接通过命令行查看namenode状态， bin/hdfs haadmin -getServiceState nn1  
参考文章：  
1.[https://blog.csdn.net/lsr40/article/details/77165453][https_blog.csdn.net_lsr40_article_details_77165453]  
2.[https://blog.csdn.net/s646575997/article/details/51235583][https_blog.csdn.net_s646575997_article_details_51235583]  
3.[https://blog.csdn.net/aspnetxuexiren/article/details/52226039][https_blog.csdn.net_aspnetxuexiren_article_details_52226039]  
4.[https://www.oschina.net/question/912919\_150256?sort=time][https_www.oschina.net_question_912919_150256_sort_time]  
5.[http://hadoop.apache.org/docs/r2.7.5/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html][http_hadoop.apache.org_docs_r2.7.5_hadoop-project-dist_hadoop-hdfs_HDFSHighAvailabilityWithQJM.html]  
ResourceManager做HA参考参考文章中第一篇

[https_pan.baidu.com_s_14InWWh2MVWw-0CIDMcypSA]: https://pan.baidu.com/s/14InWWh2MVWw-0CIDMcypSA
[70]: /images/20220528/bc2164e655a4464e9ab277e2c0613ee4.png
[http_192.168.0.11_50070_dfshealth.html_tab-overview]: http://192.168.0.11:50070/dfshealth.html#tab-overview
[http_192.168.0.12_50070_dfshealth.html_tab-overview]: http://192.168.0.12:50070/dfshealth.html#tab-overview
[https_blog.csdn.net_lsr40_article_details_77165453]: https://blog.csdn.net/lsr40/article/details/77165453
[https_blog.csdn.net_s646575997_article_details_51235583]: https://blog.csdn.net/s646575997/article/details/51235583
[https_blog.csdn.net_aspnetxuexiren_article_details_52226039]: https://blog.csdn.net/aspnetxuexiren/article/details/52226039
[https_www.oschina.net_question_912919_150256_sort_time]: https://www.oschina.net/question/912919_150256?sort=time
[http_hadoop.apache.org_docs_r2.7.5_hadoop-project-dist_hadoop-hdfs_HDFSHighAvailabilityWithQJM.html]: http://hadoop.apache.org/docs/r2.7.5/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html