Hadoop 3.2.0 完全分布式集群搭建与 WordCount 运行

矫情吗；* 2021-11-10 16:56 349阅读 0赞

# 一.集群环境搭建 #

首先准备4台服务器(虚拟机)

设置静态ip与地址映射: [centos7 修改为静态ip并设置地址映射][centos7 _ip]

地址映射为如下

![20190803163853388.png][]

然后设置集群SSH免密登录与分发脚本: [centos7配置集群SSH免密登录(包含群发文件脚本)][centos7_SSH]

如果有防火墙记得关闭防火墙

systemctl stop firewalld.service
    systemctl disable firewalld.service

下载 hadoop3.1.2 分别上传到4台服务器的 /root/hadoop 目录

地址: [http://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz][http_archive.apache.org_dist_hadoop_common_hadoop-3.2.0_hadoop-3.2.0.tar.gz]

上传后 将4台设备 分别执行解压 重命名

cd /root/hadoop
    tar -zxvf /root/hadoop/hadoop-3.2.0.tar.gz

安装JDK1.8 将jdk下载并解压到 /root/jdk 重命名为 jdk8

下载地址: [https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html][https_www.oracle.com_technetwork_java_javase_downloads_jdk8-downloads-2133151.html]

![20190803163341187.png][]

编辑 /etc/profile 设置环境变量

vim /etc/profile

在 export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL 下添加

export JAVA_HOME=/root/jdk/jdk8/
    export JRE_HOME=/root/jdk/jdk8/jre
    export HADOOP_HOME=/root/hadoop/hadoop-3.2.0/
    PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:/root/bin
    export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

更新 profile 文件

source /etc/profile

测试环境变量是否生效

java -version   hadoop version

![20190806164530768.png][]

# 二.Hadoop配置修改 #

进入到 hadoop 配置文件目录下

cd /root/hadoop/hadoop3.2.0/etc/hadoop

### 修改 hadoop-env.sh 配置 jdk 路径，定义集群操作用户 ###

在最下面增加

export JAVA_HOME=/root/jdk/jdk8
    
    export HDFS_NAMENODE_USER=root
    export HDFS_DATANODE_USER=root
    export HDFS_SECONDARYNAMENODE_USER=root
    export YARN_RESOURCEMANAGER_USER=root
    export YARN_NODEMANAGER_USER=root
    
    export HADOOP_PID_DIR=/root/hadoop/data/pids
    export HADOOP_LOG_DIR=/root/hadoop/data/logs

### 修改 core-site.xml hadoop核心配置 ###

<configuration>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop-1:8020</value>
      </property>
      <property>
        <name>hadoop.tmp.dir</name>
        <value>/root/hadoop/data/tmp</value>
      </property>
    </configuration>

*  fs.defaultFS为NameNode的地址，hadoop.tmp.dir为hadoop临时目录的地址

### 修改 hdfs-site.xml hadoop 从节点配置 ###

<configuration>
       <property>
         <name>dfs.namenode.http-address</name>
         <value>hadoop-1:9870</value>
       </property>
       <property>
         <name>dfs.namenode.secondary.http-address</name>
         <value>hadoop-2:50090</value>
       </property>
       <property>
         <name>dfs.replication</name>
         <value>2</value>
       </property>
       <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/root/hadoop/data/hdfs/name</value>
       </property>
       <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/root/hadoop/data/hdfs/data</value>
       </property>
    </configuration>

*  dfs.replication 为副本数
 *  dfs.namenode.secondary.http-address是指定secondaryNameNode的http访问地址和端口号
 *  这里将 hadoop-2 设置为 SecondaryNameNode服务器

### 修改 workers 告知 hadoop hdfs上有哪些DataNode节点 ###

hadoop-2
    hadoop-3
    hadoop-4

### 修改 yarn-site.xml 配置yarn服务 ###

<configuration>
        <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
       </property>
       <property>
          <name>yarn.nodemanager.localizer.address</name>
          <value>0.0.0.0:8140</value>
       </property>
       <property>
           <name>yarn.resourcemanager.hostname</name>
           <value>hadoop-1</value>
       </property>
       <property>
           <name>yarn.resourcemanager.webapp.address</name>
           <value>hadoop-1:8088</value>
       </property>
       <property>
           <name>yarn.log-aggregation-enable</name>
           <value>true</value>
       </property>
       <property>
           <name>yarn.log-aggregation.retain-seconds</name>
           <value>604800</value>
       </property>
       <property>
           <name>yarn.log.server.url</name>
           <value>http://hadoop-4:19888/jobhistory/logs</value>
       </property>
    </configuration>

*  yarn.resourcemanager.webapp.address  配置 resourcemanager  服务器的地址与端口
 *  yarn.resourcemanager.hostname  指定  resourcemanager  服务器
 *  yarn.log-aggregation-enable  是配置是否启用日志聚集功能
 *  yarn.log-aggregation.retain-seconds  是配置聚集的日志在HDFS上最多保存多长时间
 *  yarn.log.server.url  配置yarn日志服务器地址

### 修改mapred-site.xml 文件 ###

<configuration>
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
       <property>
           <name>yarn.app.mapreduce.am.env</name>
           <value>HADOOP_MAPRED_HOME=/root/hadoop/hadoop-3.2.0</value>
       </property>
       <property>
           <name>mapreduce.map.env</name>
           <value>HADOOP_MAPRED_HOME=/root/hadoop/hadoop-3.2.0</value>
       </property>
       <property>
           <name>mapreduce.reduce.env</name>
           <value>HADOOP_MAPRED_HOME=/root/hadoop/hadoop-3.2.0</value>
       </property>
       <property>
           <name>mapreduce.jobhistory.address</name>
           <value>hadoop-4:10020</value>
       </property>
       <property>
           <name>mapreduce.jobhistory.webapp.address</name>
           <value>hadoop-4:19888</value>
       </property>
    </configuration>

*  yarn.app.mapreduce.am.env   mapreduce.map.env   mapreduce.reduce.env 
 *  这三个是为mapreduce指定hadoop目录 如果不配置会出现  运行mapreduce找不到main方法等错误
 *  mapreduce.jobhistory.address  配置任务历史服务器地址
 *  mapreduce.jobhistory.webapp.address 配置历史服务器web访问地址

### 将修改好的配置文件 分发到 从节点的三台服务器上 ###

在当前目录下 执行xsync分发脚本

xsync hadoop-env.sh
    xsync core-site.xml
    xsync hdfs-site.xml
    xsync workers
    xsync yarn-site.xml
    xsync mapred-site.xml

配置完成

# 三.Hadoop服务启动 #

在hadoop-1节点执行namenode初始格式化命令 (仅第一次启动需要执行)

hdfs namenode -format

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70][]

执行成功 并生成 目录

cd /root/hadoop/data/dfs/name

![20190803170901538.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 1][]

生成集群唯一id,说明执行成功了

在hadoop-1执行命令

start-dfs.sh
    start-yarn.sh

或者可以执行

start-all.sh

在hadoop-4执行命令 启动job历史服务

mapred --daemon start historyserver

执行完成后分别对4台设备使用jps查看进程

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 2][]

启动成功

*  hadoop-1 为 NameNode ResourceManager 节点
 *  hadoop-2 为 SecondaryNameNode DataNode NodeManager 节点
 *  hadoop-3 为 DataNode NodeManager 节点
 *  hadoop-4 为 DataNode NodeManager JobHistoryServer 节点

查看HDFS web界面 http://hadoop-1:9870

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 3][]

查看 YARN web界面  http://hadoop-1:8088

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 4][]

hadoop搭建完成

# 四.运行WordCount #

首先在 /root 目录下创建一个txt文件

vim /root/test.txt

内容如下

hadoop 1
    hadoop 2
    hadoop 3
    hadoop 4
    hadoop 5
    hadoop 6
    hadoop 7
    hadoop 8
    hadoop 9
    hadoop 10

将test.txt文件上传到 hdfs 执行命令

hdfs dfs -mkdir /user/root
    hdfs dfs -put /root/test.txt /user/root/

找到官方自带的案例jar

cd /root/hadoop/hadoop-3.2.0/share/hadoop/mapreduce/

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 5][]

运行jar执行MapReduce WordCount案例

hadoop jar hadoop-mapreduce-examples-3.2.0.jar wordcount /user/root/test.txt /root/output

*   wordcount 后第一个路径为 文件路径
 *  第二个路径为 结果输出路径 (必须是不存在的)

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 6][]

执行成功 查看执行结果

hdfs dfs -lsr /root/output

![2019080617323014.png][]

success为执行成功

part-r-00000  m 为 mapper 输出 , r 为 reduce 输出 , 00000 为 job 任务编号 整个文件为结果输出文件

hdfs dfs -cat /root/output/part-r-00000

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 7][]

文件中 hadoop 词出现了 10次 结果正确

[centos7 _ip]: https://blog.csdn.net/weixin_40461281/article/details/98035020
[20190803163853388.png]: /images/20211109/0a2566565ecb400982d4972f29749c6a.png
[centos7_SSH]: https://blog.csdn.net/weixin_40461281/article/details/98053498
[http_archive.apache.org_dist_hadoop_common_hadoop-3.2.0_hadoop-3.2.0.tar.gz]: http://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
[https_www.oracle.com_technetwork_java_javase_downloads_jdk8-downloads-2133151.html]: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
[20190803163341187.png]: /images/20211109/9c8116027e92462ea704c47f87a7f2c6.png
[20190806164530768.png]: /images/20211109/26cff2ea6d044d79b1d9d35b7c42666e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70]: /images/20211109/bef362dfd6ff479f8789ccfd892ceacf.png
[20190803170901538.png]: /images/20211109/ef71dfa8ea544dc2bb27fa6d19a342e1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 1]: /images/20211109/8260bf5e590045a7ad0d51d37f47a3fa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 2]: /images/20211109/e5780543c15c4a9ab49026b98e3f115d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 3]: /images/20211109/7787d2a0b8674582b45946eb6cd6aee3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 4]: /images/20211109/311046ed37b1487691d1fac3055e3893.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 5]: /images/20211109/f87b935b66c543b79ccd0f879641e34e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 6]: /images/20211109/35aa63f5953d4b6c8db0819f6cfc2a0c.png
[2019080617323014.png]: /images/20211109/54cd88c195474a46bb716e8045ca39f4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDQ2MTI4MQ_size_16_color_FFFFFF_t_70 7]: /images/20211109/b797042fb5144cffbe04784040f94d27.png