记一次 Centos7.x Hadoop 3.x 部署 YARN 集群及基本操作

港控/mmm° 2022-12-29 06:42 185阅读 0赞

# 一、操作目的： #

1、如何搭建 YARN 分布式集群  
2、使用 YARN 集群提交一些简单的任务

# 二、集群搭建： #

[集群搭建过程][Link 1]

# 三、客户端环境变量配置： #

参考：[Hadoop 3.x HDFS 写入文件 中 客户端环境变量配置][Hadoop 3.x HDFS _ _]

# 四、编辑文件 yarn-site.xml ： #

1、编辑文件

[root@master ~]# vim /usr/bigdata/hadoop-3.3.0/etc/hadoop/yarn-site.xml

2、编辑内容

<configuration>
            <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>master</value>
            </property>
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    </configuration>

3、把编辑后的文件发送给各个节点

[root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/yarn-site.xml root@slave1:/usr/bigdata/hadoop-3.3.0/etc/hadoop/
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/yarn-site.xml root@slave2:/usr/bigdata/hadoop-3.3.0/etc/hadoop/
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/yarn-site.xml root@slave3:/usr/bigdata/hadoop-3.3.0/etc/hadoop/

4、修改 YARN 的启动和关闭脚本

[root@master ~]# vim /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh
    [root@master ~]# vim /usr/bigdata/hadoop-3.3.0/sbin/stop-yarn.sh
    [root@master ~]# vim /usr/bigdata/hadoop-3.3.0/bin/yarn

5、在前面加上以下内容

YARN_RESOURCEMANAGER_USER=root
    HADOOP_SECURE_DN_USER=root
    YARN_NODEMANAGER_USER=root

6、把编辑后的两个文件分别发送给各个节点

[root@master ~]# scp /usr/bigdata/hadoop-3.3.0/bin/yarn root@slave1:/usr/bigdata/hadoop-3.3.0/bin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/bin/yarn root@slave2:/usr/bigdata/hadoop-3.3.0/bin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/bin/yarn root@slave3:/usr/bigdata/hadoop-3.3.0/bin

[root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh root@slave1:/usr/bigdata/hadoop-3.3.0/sbin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh root@slave2:/usr/bigdata/hadoop-3.3.0/sbin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh root@slave3:/usr/bigdata/hadoop-3.3.0/sbin

[root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/stop-yarn.sh root@slave1:/usr/bigdata/hadoop-3.3.0/sbin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/stop-yarn.sh root@slave2:/usr/bigdata/hadoop-3.3.0/sbin
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/sbin/stop-yarn.sh root@slave3:/usr/bigdata/hadoop-3.3.0/sbin

7、启动服务

[root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh

![在这里插入图片描述][20201220234507427.png]  
8、查看各个节点的启动状态  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70]

# 五、浏览器查看 YARN 服务的 WEB 页面 #

浏览器地址栏录入 http://192.168.11.21:8088/ 说明：192.168.11.21 是 Resourcemanager 所在节点的 IP  
效果入下图![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 4]

# 六、在客户端节点上提交 Distributedshell 任务 #

1、提交 Distributedshell 任务

[root@slave2 ~]# /usr/bigdata/hadoop-3.3.0/bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar /usr/bigdata/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar -shell_command uptime

2、运行效果如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 5]  
3、在 YARN 服务的 WEB 页面查看运行结果：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 6]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 7]

# 七、在客户端节点上提交 Mapreduce 类型的任务 #

1、查看 Hadoop 的 classpath 信息

[root@master ~]# hadoop classpath

2、打印信息如下

/usr/bigdata/hadoop-3.3.0/etc/hadoop:/usr/bigdata/hadoop-3.3.0/share/hadoop/common/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/common/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/mapreduce/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn/*

3、编辑配置文件 mapred-site.xml

[root@master ~]# vim /usr/bigdata/hadoop-3.3.0/etc/hadoop/mapred-site.xml

4、编辑如下内容

<configuration>
            <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
            </property>
            <property>
    	        <name>yarn.application.classpath</name>
    	        <value>/usr/bigdata/hadoop-3.3.0/etc/hadoop:/usr/bigdata/hadoop-3.3.0/share/hadoop/common/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/common/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/hdfs/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/mapreduce/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn/lib/*:/usr/bigdata/hadoop-3.3.0/share/hadoop/yarn/*</value>
    		</property>
    </configuration>

5、把编辑后的 mapred-site.xml 文件发送给各个节点

[root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/mapred-site.xml root@slave1:/usr/bigdata/hadoop-3.3.0/etc/hadoop/
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/mapred-site.xml root@slave2:/usr/bigdata/hadoop-3.3.0/etc/hadoop/
    [root@master ~]# scp /usr/bigdata/hadoop-3.3.0/etc/hadoop/mapred-site.xml root@slave3:/usr/bigdata/hadoop-3.3.0/etc/hadoop/

6、重新启动 YARN 的服务

[root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/stop-yarn.sh
    [root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/stop-dfs.sh
    [root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/start-dfs.sh
    [root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh

7、提交 PI Estimator 任务

[root@slave1 ~]# /usr/bigdata/hadoop-3.3.0/bin/hadoop jar /usr/bigdata/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar pi 2 10

命令最后两个参数的含义：2 运行两次；10 每个 map 的任务；两个数相乘就是取样数  
Mapreduce 运行过程及结果：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 9]  
8、在 YARN 服务的 WEB 页面查看运行结果：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 10]

# 八、扩展： #

重启 YARN 服务也可以这样执行

[root@master ~]# yarn --daemon stop resourcemanager
    [root@master ~]# yarn --daemon stop nodemanager
    [root@master ~]# yarn --daemon start resourcemanager
    [root@master ~]# yarn --daemon start nodemanager

至此 Centos7.x Hadoop 3.x 部署 YARN 集群及基本操作完毕，希望能够对您有所帮助！

[Link 1]: https://blog.csdn.net/llwy1428/article/details/111144524
[Hadoop 3.x HDFS _ _]: https://blog.csdn.net/llwy1428/article/details/111445668
[20201220234507427.png]: /images/20221120/9b2dc2dba04846b7bf6556a8438ddd99.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70]: /images/20221120/54076fabf50947e28213acaf30ab64ae.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20201220234752769.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20201220234850272.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201220234907300.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20201220234935705.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20201220235551251.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20201220235414443.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20201220235431970.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20201221001324705.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20201221001428115.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20201221001540797.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4,size_16,color_FFFFFF,t_70