Yarn分布式集群环境部署

冷不防 2021-09-20 03:02 461阅读 0赞

# YARN配置 #

在Hadoop-HA高可用分布式环境中改造

node01:

1）**mapred-site.xml**

<property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
      </property>

2）**yarn-site.xml**

配置RS

<property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
     </property>
    <property>
       <name>yarn.resourcemanager.ha.enabled</name>
       <value>true</value>
     </property>
     <property>
       <name>yarn.resourcemanager.cluster-id</name>
       <value>cluster1</value>
     </property>
     <property>
       <name>yarn.resourcemanager.ha.rm-ids</name>
       <value>rm1,rm2</value>
     </property>
     <property>
       <name>yarn.resourcemanager.hostname.rm1</name>
       <value>bd003</value>
     </property>
     <property>
       <name>yarn.resourcemanager.hostname.rm2</name>
       <value>bd004</value>
     </property>
     <property>
       <name>yarn.resourcemanager.zk-address</name>
       <value>bd002:2181,bd003:2181,bd004:2181</value>
     </property>

**分发至2/3/4**

#分发
    scp yarn-site.xml  bd002:`pwd`

3）启动

node1:
    start-yarn.sh
    node3、node4
    yarn-daemon.sh start resourcemanager

4）访问测试

bd003:8088
    bd004:8088

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0FybmVf_size_16_color_FFFFFF_t_70]

5）最后所有服务的停止和启动

1）启动
    节点2,3,4,启动zk
    #zkServer.sh start
    节点1：
    # start-dfs.sh 
    #start-yarn.sh
    
    node3、node4
    #yarn-daemon.sh start resourcemanager
    
    2）停止
    节点1：
    # stop-dfs.sh 
    #stop-yarn.sh
    node3、node4
    #yarn-daemon.sh stop resourcemanager
    节点2,3,4,停止zk
    #zkServer.sh stop

写了几个脚本，提高效率

#管理resourcemanager
    
    #!/bin/bash
    for i in  bd003 bd004
    do
    echo "---------- $1 ing $i ----------"
    ssh $i "source ~/.bash_profile;yarn-daemon.sh $1 resourcemanager"
    done
    
    
    #查看进程
    #!/bin/bash
    for i in bd001 bd002 bd003 bd004
    do
    echo "-------------------------------------"
    echo "------------ JPS $i -----------------"
    ssh $i "source ~/.bash_profile;jps"
    echo "-------------------------------------"
    sleep 1
    done
    
    
    #批量重启
    #!/bin/bash
    for i in  bd002 bd003 bd004
    do
    echo "-------------------------------------"
    echo "------------ init $1 $i -----------------"
    ssh $i "init $1"
    sleep 2
    done

问题整理：启动报错

bd002: datanode running as process 1501. Stop it first.
    bd004: datanode running as process 1432. Stop it first.
    
    hadoop出现namenode running as process 18472. Stop it first.等等，
    类似的出现了好几个。
    
    解决方法：
    应该是没有正常停止服务导致的
    在启动之前你需要在重新启动hadoop之前要先stop掉所有的hadoop服务。 
    然后恢复正常启动。
       #stop-all.sh
       #start-all.sh

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0FybmVf_size_16_color_FFFFFF_t_70]: /images/20210827/14b42bfd4247430ea4df62ca66deceb0.png