hadoop2高可用集群配置安装

小咪咪 2022-08-09 00:38 262阅读 0赞

上一节演示了[ hadoop2集群安装和测试][hadoop2] 其中namenode节点是单点的，一旦namenode宕机，服务将不可使用。本节使用zookeeper配置hadoop namenode的可用集群，并使用yarn对hadoop提供统一的资源管理和调度。

关于操作系统环境的配置以及hadoop、zookeeper的安装这里就不介绍，请参考如下文档：

1.[linux上Zookeeper集群安装及监控][linux_Zookeeper]

2.[hadoop2集群安装和测试之环境配置][hadoop2]  
3.[hadoop2集群安装和测试之软件安装配置][hadoop2 1]

机器分配如下：

study-90      NameNode、DFSZKFailoverController(zkfc)、ResourceManager、QuorumPeerMain  
study-91      NameNode、DFSZKFailoverController(zkfc)、ResourceManager、QuorumPeerMain  
study-92      DataNode、NodeManager、JournalNode、QuorumPeerMain  
study-93      DataNode、NodeManager、JournalNode、JobHistoryServer

QuorumPeerMain为zookeeper进程，JournalNode和zookeeper保持奇数点，最少不少于3个节点，由于机器配置有限这里JournalNode我只配置了2各节点。JobHistory Server这是一个独立的服务，可通过web UI展示历史作业日志，之所以将其独立出来，是为了减轻ResourceManager负担。

为了防止ssh需要密码，我们将这4台机器的的/home/hadoop/.ssh/id\_rsa.pub公钥内容追加到同一个authorized\_keys（文件权限644），并将其复制到所有机器的/home/hadoop/.ssh下。

**一、配置hadoop**

**1.core-site.xml **

<configuration>
                    
            <property>
                    <name>fs.defaultFS</name>
                    <value>hdfs://mycluster</value>
            </property>
                    
            <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/home/hadoop/hadoop-2.6.0/temp</value>
            </property>
                    
            <property>
                    <name>io.file.buffer.size</name>
                    <value>4096</value>
            </property>
                    
            <property>
                    <name>ha.zookeeper.quorum</name>
                    <value>study-90:2181,study-91:2181,study-92:2181</value>
            </property>
    </configuration>

**2.hdfs-site.xml**

<configuration>
            
            <property>
                            <name>dfs.nameservices</name>
                            <value>mycluster</value>
            </property>
            
            <property>
                            <name>dfs.ha.namenodes.mycluster</name>
                            <value>study-90,study-91</value>
            </property>
            
            <property>
                            <name>dfs.namenode.rpc-address.mycluster.study-90</name>
                            <value>study-90:9000</value>
            </property>
            
            <property>
                            <name>dfs.namenode.http-address.mycluster.study-90</name>
                            <value>study-90:50070</value>
            </property>
            
            <property>
                            <name>dfs.namenode.rpc-address.mycluster.study-91</name>
                            <value>study-91:9000</value>
            </property>
            
            <property>
                            <name>dfs.namenode.http-address.mycluster.study-91</name>
                            <value>study-91:50070</value>
            </property>
             
            <property>
                            <name>dfs.namenode.shared.edits.dir</name>
                            <value>qjournal://study-92:8485;study-93:8485/mycluster</value>
            </property>
            
            <property>
                            <name>dfs.journalnode.edits.dir</name>
                            <value>/home/hadoop/hadoop-2.6.0/journal</value>
            </property>
            
            <property>
                            <name>dfs.ha.automatic-failover.enabled</name>
                            <value>true</value>
            </property>
            
            <property>
                            <name>dfs.client.failover.proxy.provider.mycluster</name>
                            <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
            </property>
            
            <property>
                            <name>dfs.ha.fencing.methods</name>
                            <value>
                                    sshfence
                                    shell(/bin/true)
                            </value>
            </property>
            
            <property>
                            <name>dfs.ha.fencing.ssh.private-key-files</name>
                            <value>/home/hadoop/.ssh/id_rsa</value>
            </property>
            
            <property>
                            <name>dfs.ha.fencing.ssh.connect-timeout</name>
                            <value>30000</value>
            </property>
            
            <property>
                            <name>dfs.namenode.name.dir</name>
                            <value>file:/home/hadoop/hadoop-2.6.0/dfs/namenode</value>
                            <final>true</final>
            </property>
            
            <property>
                            <name>dfs.datanode.data.dir</name>
                            <value>file:/home/hadoop/hadoop-2.6.0/dfs/datanode</value>
                            <final>true</final>
            </property>
            
            <property>
                            <name>dfs.replication</name>
                            <value>2</value>
            </property>
            
            <property>
                            <name>dfs.webhdfs.enabled</name>
                            <value>true</value>
            </property>
            
            <property>
                            <name>dfs.permissions</name>
                            <value>false</value>
            </property>
            <property>
                            <name>dfs.permissions.enabled</name>
                            <value>false</value>
            </property>
    </configuration>

**3.mapred-site.xml**

<configuration>
                    
                    <property>
                                    <name>mapreduce.framework.name</name>
                                    <value>yarn</value>
                                    <final>true</final>
                    </property>
                    
                    <property>
                                    <name>mapred.system.dir</name>
                                    <value>file:/home/hadoop/mapred/system</value>
                                    <final>true</final>
                    </property>
                    
                    <property>
                                     <name>mapred.local.dir</name>
                                     <value>file:/home/hadoop/mapred/local</value>
                                     <final>true</final>
                    </property>
    		
                    <property>
                            <name>mapreduce.jobhistory.address</name>
                            <value>study-93:10020</value>
                    </property>
    </configuration>

**4.yarn-site.xml**

<configuration>
                    
            <property>
                    <name>yarn.resourcemanager.ha.enabled</name>
                    <value>true</value>
            </property>
                    
            <property>
                    <name>yarn.resourcemanager.cluster-id</name>
                    <value>yarn-cluster</value>
            </property>
                     
            <property>
                    <name>yarn.resourcemanager.ha.rm-ids</name>
                    <value>rm1,rm2</value>
            </property>
                      
            <property>
                    <name>yarn.resourcemanager.hostname.rm1</name>
                    <value>study-90</value>
            </property>
            <property>
                    <name>yarn.resourcemanager.hostname.rm2</name>
                    <value>study-91</value>
            </property>
            <property>
                    <name>yarn.resourcemanager.recovery.enabled</name>
                    <value>true</value>
            </property>
                    
            <property>
                    <name>yarn.resourcemanager.connect.retry-interval.ms</name>
                    <value>2000</value>
            </property>
            <property>
                    <name>yarn.resourcemanager.zk-address</name>
                    <value>study-90:2181,study-91:2181,study-92:2181</value>
            </property>
            <property>
                    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
            </property>
            <property>
                    <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
                    <value>/yarn-leader-election</value>
            </property>
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    </configuration>

**5.slaves**

study-92
    study-93

配置完成，将hadoop安装目录复制到其他机器对应的目录  
scp -r hadoop-2.6.0  hadoop@study-91:/home/hadoop/hadoop-2.6.0

**二、启动hadoop**

**1.格式化ZK创建命名空间**

./bin/hdfs zkfc -formatZK

**2.启动journalnode**  
在study-92、study-93上启动JournalNode进程

sbin/hadoop-daemon.sh start journalnode

**3.格式化HDFS**

在study-90上格式化HDFS

./bin/hdfs namenode -format mycluster

并将根据dfs.namenode.name.dir配置格式化生产的dfs目录复制到study-91对应的目录上

scp -r dfs/ study-91:/home/hadoop/hadoop-2.6.0/

**4.启动hadoop节点**

在study-90节点：

./sbin/start-dfs.sh

./sbin/start-yarn.sh

在study-91节点：

./sbin/yarn-daemon.sh start resourcemanager

在study-93节点：

./sbin/mr-jobhistory-daemon.sh start historyserver

可以通过jps查看启动进程：

[hadoop@study-90 sbin]$ jps
    1566 QuorumPeerMain
    24184 DFSZKFailoverController
    24650 Jps
    23921 NameNode
    24286 ResourceManager

**三、测试**

**1.环境是否可用**

vi ~/test/hello.txt

hello world
    hello tom
    good by
    good nice

将文件提交到dfs中

./bin/hadoop fs -put ~/test/  /input

运行单词统计例子：

./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output

15/09/25 03:35:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    15/09/25 03:35:13 INFO input.FileInputFormat: Total input paths to process : 1
    15/09/25 03:35:13 INFO mapreduce.JobSubmitter: number of splits:1
    15/09/25 03:35:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443164125300_0001
    15/09/25 03:35:15 INFO impl.YarnClientImpl: Submitted application application_1443164125300_0001
    15/09/25 03:35:15 INFO mapreduce.Job: The url to track the job: http://study-92:8088/proxy/application_1443164125300_0001/
    15/09/25 03:35:15 INFO mapreduce.Job: Running job: job_1443164125300_0001
    15/09/25 03:35:55 INFO mapreduce.Job: Job job_1443164125300_0001 running in uber mode : false
    15/09/25 03:35:55 INFO mapreduce.Job:  map 0% reduce 0%
    15/09/25 03:37:05 INFO mapreduce.Job:  map 67% reduce 0%
    15/09/25 03:37:09 INFO mapreduce.Job:  map 100% reduce 0%
    15/09/25 03:39:45 INFO mapreduce.Job:  map 100% reduce 67%
    15/09/25 03:39:48 INFO mapreduce.Job:  map 100% reduce 100%
    15/09/25 03:40:39 INFO mapreduce.Job: Job job_1443164125300_0001 completed successfully
    15/09/25 03:40:39 INFO mapreduce.Job: Counters: 48
            File System Counters
                    FILE: Number of bytes read=71
                    FILE: Number of bytes written=216098
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=137
                    HDFS: Number of bytes written=41
                    HDFS: Number of read operations=6
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=2
            Job Counters 
                    Launched map tasks=1
                    Launched reduce tasks=1
                    Total time spent by all maps in occupied slots (ms)=92911
                    Total time spent by all reduces in occupied slots (ms)=101550
                    Total time spent by all map tasks (ms)=92911
                    Total time spent by all reduce tasks (ms)=101550
                    Total vcore-seconds taken by all map tasks=92911
                    Total vcore-seconds taken by all reduce tasks=101550
                    Total megabyte-seconds taken by all map tasks=95140864
                    Total megabyte-seconds taken by all reduce tasks=103987200
            Map-Reduce Framework
                    Map input records=4
                    Map output records=8
                    Map output bytes=72
                    Map output materialized bytes=71
                    Input split bytes=97
                    Combine input records=8
                    Combine output records=6
                    Reduce input groups=6
                    Reduce shuffle bytes=71
                    Reduce input records=6
                    Reduce output records=6
                    Spilled Records=12
                    Shuffled Maps =1
                    Failed Shuffles=0
                    Merged Map outputs=1
                    GC time elapsed (ms)=1266
                    CPU time spent (ms)=3600
                    Physical memory (bytes) snapshot=267403264
                    Virtual memory (bytes) snapshot=1689710592
                    Total committed heap usage (bytes)=130592768
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters 
                    Bytes Read=40
            File Output Format Counters 
                    Bytes Written=41

运行过程中可以通过http://study-92:8088/proxy/application\_1443164125300\_0001/查看job运行状态，结束后查看 /output目录文件信息结果：

![Center][]

**2.可用性检查**  
服务启动后可以通过http://192.168.1.90:50070/查看namenode状态，90为active，91为standby

![Center 1][]

![Center 2][]

将study-90上的NameNode的进程kill掉，可以看见namenode active 切换到study-91。

参考文章：

1.[Hadoop-2.4.0分布式安装手册][Hadoop-2.4.0]

[hadoop2]: http://blog.csdn.net/zhu_tianwei/article/details/48764987
[linux_Zookeeper]: http://blog.csdn.net/zhu_tianwei/article/details/40685251
[hadoop2 1]: http://blog.csdn.net/zhu_tianwei/article/details/48765017
[Center]: /images/20220731/67d32823a4f1401389f4cf4356f6bb27.png
[Center 1]: /images/20220731/3e0da4b5e0a149e08c288e6ccaae6790.png
[Center 2]: /images/20220731/082d08f25d094e0a8aff2b2893acc9b6.png
[Hadoop-2.4.0]: http://blog.chinaunix.net/uid-20682147-id-4229024.html?bsh_bid=398174369#_Toc28975