spark on yarn 配置

╰+攻爆jí腚メ 2022-06-03 09:44 253阅读 0赞

### 系统：centos7  软件： ###

1.   hadoop-2.7.2.tar.gz，请勿选择src版本，否则需要自行编译
2.   jdk-7u79-linux-x64.tar.gz
3.    scala-2.10.3.tgz
4.    spark-1.6.1-bin-hadoop2.6.tgz，选择hadooppre-built版本，否则需要安装后编译
5.    zookeeper-3.3.6.tar.gz

**2、详细配置流程说明**

**1，JDK配置**

> \# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local 
> 
> \# cd /usr/local 
> 
> \# ln -s jdk1.7.0\_45 jdk 
> 
> * *
> 
> \# vim /etc/profile 加入以下内容

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  export JAVA\_HOME=/usr/local/jdk   
> > 2.  export CLASS\_PATH=$JAVA\_HOME/lib:$JAVA\_HOME/jre/lib   
> > 3.  export PATH=$PATH:$JAVA\_HOME/bin

> \# source /etc/profile

**2. Scala安装**

> \# tar xvzf scala-2.10.3.tgz -C/usr/local 
> 
> \# cd /usr/local 
> 
> \# ln -s scala-2.10.3 scala 
> 
> *  *
> 
> \# vim /etc/profile 加入以下内容

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  export SCALA\_HOME=/usr/local/scala   
> > 2.  export PATH=$PATH:$SCALA\_HOME/bin

**3. SSH免登录配置**

> \#ssh-keygen
> 
> 在node1下生成的密钥对：id\_rsa和id\_rsa.pub，默认存储在"~/.ssh"目录下，包括两个文件，id\_rsa和id\_rsa.pub，分别为私钥和公钥
> 
>  将公钥写入信任文件中
> 
> \#cat ~/.ssh/id\_rsa.pub >> ~/.ssh/authorized\_keys
> 
>        然后修改authorized\_keys文件的权限
> 
> \#chmod 644 ~/.ssh/authorized\_keys
> 
>        node1中的authorized\_keys拷贝至其余节点的~/.ssh目录下，修改SSH配置文件"/etc/ssh/sshd\_config"的下列内容（这步可以省略）
> 
> RSAAuthentication yes \# 启用 RSA 认证 
> 
> PubkeyAuthentication yes \# 启用公钥私钥配对认证方式 
> 
> AuthorizedKeysFile .ssh/authorized\_keys \# 公钥文件路径（和上面生成的文件同）

> \#service sshd restart

> 重启SSH服务

4.**主机名设置**

> \# vim /etc/hosts

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  10.25.248.1    node1  
> > 2.  10.25.248.29   node2  
> > 3.  10.25.248.17   node3  
> > 
> >   
> >

> \#vi /etc/sysconfig/network
> 
>       HOSTNAME=node1/node2/node3

**5. Zookeeper安装**

> \# mkdir –p /root/data/hadoop/zookeeper/\{data,logs\}
> 
> 两个文件夹都需要预先建立好，否则会运行时会报错
> 
>  
> 
> \# vim /usr/local/zookeeper/conf/zoo.cfg

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  tickTime=2000  
> > 2.  initLimit=10  
> > 3.  syncLimit=5  
> > 4.     
> > 5.  dataDir=/root/data/hadoop/zookeeper/data  
> > 6.  clientPort=2181  
> > 7.     
> > 8.  server.1=10.25.248.1:2888:3888  
> > 9.  server.2=10.25.248.29:2888:3888  
> > 10. server.3=10.25.248.17:2888:3888  
> > 
> >   
> >   
> >

> 接着在/root/data/hadoop/zookeeper/data下分别建立名为myid文件，文件内容为上述zoo.cfg中IP地址对应server.\[number\]中的number
> 
> node1 : echo 1 > /root/data/hadoop/zookeeper/data/myid
> 
> node2 : echo 2 > /root/data/hadoop/zookeeper/data/myid
> 
> node3 : echo 3 > /root/data/hadoop/zookeeper/data/myid
> 
>  
> 
> 执行zkServer.sh start启动zookeeper，相应zkServer.shstop退出，zkServer.sh status查看状态
> 
> 分别在node1，node2，node3执行zkServer.sh start，然后通过zkServer.sh status查看状态，如果发现每个node当前状态标记为follower或者leader，那么测试通过

6. **Hadoop（HDFS HA）集群部署**

> \# tar xvzf hadoop-2.2.0.tgz -C/usr/local
> 
> \# cd /usr/local
> 
> \# ln -s hadoop-2.2.0 hadoop
> 
> * *
> 
> \#vim /etc/profile

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  export HADOOP\_HOME=/usr/local/hadoop  
> > 2.  export HADOOP\_PID\_DIR=/root/data/hadoop/pids  
> > 3.  export HADOOP\_COMMON\_LIB\_NATIVE\_DIR=$HADOOP\_HOME/lib/native  
> > 4.  export HADOOP\_OPTS="$HADOOP\_OPTS-Djava.library.path=$HADOOP\_HOME/lib/native"  
> > 5.  export HADOOP\_MAPRED\_HOME=$HADOOP\_HOME  
> > 6.  export HADOOP\_COMMON\_HOME=$HADOOP\_HOME  
> > 7.  export HADOOP\_HDFS\_HOME=$HADOOP\_HOME  
> > 8.  export YARN\_HOME=$HADOOP\_HOME  
> > 9.  export HADOOP\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop  
> > 10. export HDFS\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop  
> > 11. export YARN\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop  
> > 12. export JAVA\_LIBRARY\_PATH=$HADOOP\_HOME/lib/native  
> > 13. export PATH=$PATH:$HADOOP\_HOME/bin:$HADOOP\_HOME/sbin  
> > 
> >   
> >

> 接下来有8个配置文件需要修改，配置文件均在$HADOOP\_HOME/etc/hadoop/目录下

1.  hadoop-env.sh, mapred-env.sh, yarn-env.sh中加入以下内容

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  export JAVA\_HOME=/usr/local/jdk   
> > 2.  export CLASS\_PATH=$JAVA\_HOME/lib:$JAVA\_HOME/jre/lib   
> > 3.      
> > 4.  export HADOOP\_HOME=/usr/local/hadoop   
> > 5.  export HADOOP\_PID\_DIR=/root/data/hadoop/pids   
> > 6.  export HADOOP\_COMMON\_LIB\_NATIVE\_DIR=$HADOOP\_HOME/lib/native   
> > 7.  export HADOOP\_OPTS="$HADOOP\_OPTS-Djava.library.path=$HADOOP\_HOME/lib/native"   
> > 8.      
> > 9.  export HADOOP\_PREFIX=$HADOOP\_HOME   
> > 10.     
> > 11. export HADOOP\_MAPRED\_HOME=$HADOOP\_HOME   
> > 12. export HADOOP\_COMMON\_HOME=$HADOOP\_HOME   
> > 13. export HADOOP\_HDFS\_HOME=$HADOOP\_HOME   
> > 14. export YARN\_HOME=$HADOOP\_HOME   
> > 15.     
> > 16. export HADOOP\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop   
> > 17. export HDFS\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop   
> > 18. export YARN\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop   
> > 19.     
> > 20. export JAVA\_LIBRARY\_PATH=$HADOOP\_HOME/lib/native   
> > 21.     
> > 22. export PATH=$PATH:$JAVA\_HOME/bin:$HADOOP\_HOME/bin:$HADOOP\_HOME/sbin  
> > 
> >   
> > 
> 
> 2，core-site.xml

> 
> 
> 
> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1. **<****configuration****>**  
> > 2.     **<****property****>**   
> > 3.         **<****name****>**fs.defaultFS**</****name****>**   
> > 4.         **<****value****>**hdfs://lab369**</****value****>**   
> > 5.     **</****property****>**   
> > 6.                                                                                                                                                                 
> > 7.     **<****property****>**   
> > 8.         **<****name****>**io.file.buffer.size**</****name****>**   
> > 9.         **<****value****>**131072**</****value****>**   
> > 10.     **</****property****>**   
> > 11.                                                                                                                                                                 
> > 12.     **<****property****>**   
> > 13.         **<****name****>**hadoop.tmp.dir**</****name****>**   
> > 14.         **<****value****>**file:/root/data/hadoop/storage/tmp**</****value****>**   
> > 15.     **</****property****>**   
> > 16.                                                                                                                                                                 
> > 17.     **<****property****>**   
> > 18.         **<****name****>**ha.zookeeper.quorum**</****name****>**   
> > 19.         **<****value****>**node1:2181,node2:2181,node3:2181**</****value****>**   
> > 20.     **</****property****>**   
> > 21.                                                                                                                                                                 
> > 22.     **<****property****>**   
> > 23.         **<****name****>**ha.zookeeper.session-timeout.ms**</****name****>**   
> > 24.         **<****value****>**2000**</****value****>**   
> > 25.     **</****property****>**   
> > 26.                                                                                                                                                                 
> > 27.     **<****property****>**   
> > 28.         **<****name****>**fs.trash.interval**</****name****>**   
> > 29.         **<****value****>**4320**</****value****>**   
> > 30.     **</****property****>**   
> > 31.                                                                                                                                                                 
> > 32.     **<****property****>**   
> > 33.          **<****name****>**hadoop.http.staticuser.use**</****name****>**   
> > 34.          **<****value****>**root**</****value****>**   
> > 35.     **</****property****>**   
> > 36.                                                                                                                                                                 
> > 37.     **<****property****>**   
> > 38.         **<****name****>**hadoop.proxyuser.hadoop.hosts**</****name****>**   
> > 39.         **<****value****>**\***</****value****>**   
> > 40.     **</****property****>**   
> > 41.                                                                                                                                                                 
> > 42.     **<****property****>**   
> > 43.         **<****name****>**hadoop.proxyuser.hadoop.groups**</****name****>**   
> > 44.         **<****value****>**\***</****value****>**   
> > 45.     **</****property****>**   
> > 46.                                                                                                                                                                 
> > 47.     **<****property****>**   
> > 48.         **<****name****>**hadoop.native.lib**</****name****>**   
> > 49.         **<****value****>**true**</****value****>**   
> > 50.     **</****property****>**   
> > 51. **</****configuration****>**  
> > 
> > 
> > 
> 
> 3， hdfs-site.xml

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1. **<?****xml** version="1.0" encoding="UTF-8"**?>**  
> > 2. **<?****xml-stylesheet** type="text/xsl" href="configuration.xsl"**?>**  
> > 3.   
> > 18.   
> > 19. **<****configuration****>**  
> > 20.     **<****property****>**   
> > 21.         **<****name****>**dfs.namenode.name.dir**</****name****>**   
> > 22.         **<****value****>**file:/root/data/hadoop/storage/hdfs/name**</****value****>**   
> > 23.     **</****property****>**   
> > 24.                                                                                                                                                     
> > 25.     **<****property****>**   
> > 26.         **<****name****>**dfs.datanode.data.dir**</****name****>**   
> > 27.         **<****value****>**file:/root/data/hadoop/storage/hdfs/data**</****value****>**   
> > 28.     **</****property****>**   
> > 29.                                                                                                                                                     
> > 30.     **<****property****>**   
> > 31.         **<****name****>**dfs.replication**</****name****>**   
> > 32.         **<****value****>**2**</****value****>**   
> > 33.     **</****property****>**   
> > 34.                                                                                                                                                     
> > 35.     **<****property****>**   
> > 36.         **<****name****>**dfs.blocksize**</****name****>**   
> > 37.         **<****value****>**67108864**</****value****>**   
> > 38.     **</****property****>**   
> > 39.                                                                                                                                                     
> > 40.     **<****property****>**   
> > 41.         **<****name****>**dfs.datanode.du.reserved**</****name****>**   
> > 42.         **<****value****>**10737418240**</****value****>**   
> > 43.     **</****property****>**   
> > 44.                                                                                                                                                     
> > 45.     **<****property****>**   
> > 46.         **<****name****>**dfs.webhdfs.enabled**</****name****>**   
> > 47.         **<****value****>**true**</****value****>**   
> > 48.     **</****property****>**   
> > 49.                                                                                                                                                     
> > 50.     **<****property****>**   
> > 51.         **<****name****>**dfs.permissions**</****name****>**   
> > 52.         **<****value****>**true**</****value****>**   
> > 53.     **</****property****>**   
> > 54.                                                                                                                                                     
> > 55.     **<****property****>**   
> > 56.         **<****name****>**dfs.permissions.enabled**</****name****>**   
> > 57.         **<****value****>**true**</****value****>**   
> > 58.     **</****property****>**   
> > 59.                                                                                                                                                     
> > 60.     **<****property****>**   
> > 61.         **<****name****>**dfs.nameservices**</****name****>**   
> > 62.         **<****value****>**lab369**</****value****>**   
> > 63.     **</****property****>**   
> > 64.                                                                                                                                                     
> > 65.     **<****property****>**   
> > 66.         **<****name****>**dfs.ha.namenodes.lab369**</****name****>**   
> > 67.         **<****value****>**nn1,nn2**</****value****>**   
> > 68.     **</****property****>**   
> > 69.                                                                                                                                                     
> > 70.     **<****property****>**   
> > 71.         **<****name****>**dfs.namenode.rpc-address.lab369.nn1**</****name****>**   
> > 72.         **<****value****>**node1:8020**</****value****>**   
> > 73.     **</****property****>**   
> > 74.                                                                                                                                                     
> > 75.     **<****property****>**   
> > 76.         **<****name****>**dfs.namenode.rpc-address.lab369.nn2**</****name****>**   
> > 77.         **<****value****>**node2:8020**</****value****>**   
> > 78.     **</****property****>**   
> > 79.                                                                                                                                                     
> > 80.     **<****property****>**   
> > 81.         **<****name****>**dfs.namenode.servicerpc-address.lab369.nn1**</****name****>**   
> > 82.         **<****value****>**node1:53310**</****value****>**   
> > 83.     **</****property****>**   
> > 84.                                                                                                                                                     
> > 85.     **<****property****>**   
> > 86.         **<****name****>**dfs.namenode.servicerpc-address.lab369.nn2**</****name****>**   
> > 87.         **<****value****>**node2:53310**</****value****>**   
> > 88.     **</****property****>**   
> > 89.                                                                                                                                                     
> > 90.     **<****property****>**   
> > 91.         **<****name****>**dfs.namenode.http-address.lab369.nn1**</****name****>**   
> > 92.         **<****value****>****<****span** style="color:\#ff0000;"**>**node1:**<****span** style="font-family:Arial, Helvetica, sans-serif;"**>**50070**</****span****>****</****span****>****</****value****>**   
> > 93.     **</****property****>**   
> > 94.                                                                                                                                                     
> > 95.     **<****property****>**   
> > 96.         **<****name****>**dfs.namenode.http-address.lab369.nn2**</****name****>**   
> > 97.         **<****value****>****<****span** style="color:\#ff0000;"**>**node2:50070**</****span****>****</****value****>**   
> > 98.     **</****property****>**   
> > 99.                                                                                                                                                     
> > 100.     **<****property****>**   
> > 101.         **<****name****>**dfs.datanode.http.address**</****name****>**   
> > 102.         **<****value****>****<****span** style="color:\#ff0000;"**>**0.0.0.0:50070**</****span****>****</****value****>**   
> > 103.     **</****property****>**   
> > 104.                                                                                                                                                     
> > 105.     **<****property****>**   
> > 106.         **<****name****>**dfs.namenode.shared.edits.dir**</****name****>**   
> > 107.         **<****value****>**qjournal://node1:8485;node2:8485;node3:8485/lab369**</****value****>**  
> > 108.     **</****property****>**   
> > 109.                                                                                                                                                     
> > 110.     **<****property****>**   
> > 111.         **<****name****>**dfs.client.failover.proxy.provider.lab369**</****name****>**   
> > 112.         **<****value****>**org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider**</****value****>**   
> > 113.     **</****property****>**   
> > 114.                                                                                                                                                     
> > 115.     **<****property****>**   
> > 116.         **<****name****>**dfs.ha.fencing.methods**</****name****>**   
> > 117.         **<****value****>**sshfence**</****value****>**   
> > 118.     **</****property****>**   
> > 119.                                                                                                                                                     
> > 120.     **<****property****>**   
> > 121.         **<****name****>**dfs.ha.fencing.ssh.private-key-files**</****name****>**   
> > 122.         **<****value****>**/root/.ssh/id\_rsa**</****value****>**   
> > 123.     **</****property****>**   
> > 124.                                                                                                                                                     
> > 125.     **<****property****>**   
> > 126.         **<****name****>**dfs.ha.fencing.ssh.connect-timeout**</****name****>**   
> > 127.         **<****value****>**30000**</****value****>**   
> > 128.     **</****property****>**   
> > 129.                                                                                                                                                     
> > 130.     **<****property****>**   
> > 131.         **<****name****>**dfs.journalnode.edits.dir**</****name****>**   
> > 132.         **<****value****>**/root/data/hadoop/storage/hdfs/journal**</****value****>**   
> > 133.     **</****property****>**   
> > 134.                                                                                                                                                     
> > 135.     **<****property****>**   
> > 136.         **<****name****>**dfs.ha.automatic-failover.enabled**</****name****>**   
> > 137.         **<****value****>**true**</****value****>**   
> > 138.     **</****property****>**   
> > 139.                                                                                                                                                     
> > 140.     **<****property****>**   
> > 141.         **<****name****>**ha.failover-controller.cli-check.rpc-timeout.ms**</****name****>**   
> > 142.         **<****value****>**60000**</****value****>**   
> > 143.     **</****property****>**   
> > 144.                                                                                                                                                     
> > 145.     **<****property****>**   
> > 146.         **<****name****>**ipc.client.connect.timeout**</****name****>**   
> > 147.         **<****value****>**60000**</****value****>**   
> > 148.     **</****property****>**   
> > 149.                                                                                                                                                     
> > 150.     **<****property****>**   
> > 151.         **<****name****>**dfs.image.transfer.bandwidthPerSec**</****name****>**   
> > 152.         **<****value****>**41943040**</****value****>**   
> > 153.     **</****property****>**   
> > 154.                                                                                                                                                     
> > 155.     **<****property****>**   
> > 156.         **<****name****>**dfs.namenode.accesstime.precision**</****name****>**   
> > 157.         **<****value****>**3600000**</****value****>**   
> > 158.     **</****property****>**   
> > 159.                                                                                                                                                     
> > 160.     **<****property****>**   
> > 161.         **<****name****>**dfs.datanode.max.transfer.threads**</****name****>**   
> > 162.         **<****value****>**4096**</****value****>**   
> > 163.     **</****property****>**   
> > 164. **</****configuration****>**  
> > 
> > 
> > 
> >

> 4，mapred-site.xml

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1. **<****configuration****>**   
> > 2.     **<****property****>**   
> > 3.         **<****name****>**mapreduce.framework.name**</****name****>**   
> > 4.         **<****value****>**yarn**</****value****>**   
> > 5.     **</****property****>**   
> > 6.                                                                                                                                          
> > 7.     **<****property****>**   
> > 8.         **<****name****>**mapreduce.jobhistory.address**</****name****>**   
> > 9.         **<****value****>**node1:10020**</****value****>**   
> > 10.     **</****property****>**   
> > 11.                                                                                                                                          
> > 12.     **<****property****>**   
> > 13.         **<****name****>**mapreduce.jobhistory.webapp.address**</****name****>**   
> > 14.         **<****value****>**node1:19888**</****value****>**   
> > 15.     **</****property****>**  
> > 16.       
> > 17.     **<****property****>**  
> > 18.         **<****name****>**mapreduce.application.classpath**</****name****>**  
> > 19.         **<****value****>**  
> > 20.         /usr/local/hadoop/etc/hadoop,  
> > 21.         /usr/local/hadoop/share/hadoop/common/\*,  
> > 22.         /usr/local/hadoop/share/hadoop/common/lib/\*,  
> > 23.         /usr/local/hadoop/share/hadoop/hdfs/\*,  
> > 24.         /usr/local/hadoop/share/hadoop/hdfs/lib/\*,  
> > 25.         /usr/local/hadoop/share/hadoop/mapreduce/\*,  
> > 26.         /usr/local/hadoop/share/hadoop/mapreduce/lib/\*,  
> > 27.         /usr/local/hadoop/share/hadoop/yarn/\*,  
> > 28.         /usr/local/hadoop/share/hadoop/yarn/lib/\*  
> > 29.         **</****value****>**  
> > 30.     **</****property****>**  
> > 31. **</****configuration****>**  
> > 
> > 
> > 
> >

> 5，yarn-site.xml

> 
> 
> 
> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1. **<****configuration****>**  
> > 2.   
> > 3.   
> > 4.     **<****property****>**   
> > 5.         **<****name****>**yarn.nodemanager.aux-services**</****name****>**   
> > 6.         **<****value****>**mapreduce\_shuffle**</****value****>**   
> > 7.     **</****property****>**   
> > 8.                                                                                                            
> > 9.     **<****property****>**   
> > 10.         **<****name****>**yarn.nodemanager.aux-services.mapreduce.shuffle.class**</****name****>**   
> > 11.         **<****value****>**org.apache.hadoop.mapred.ShuffleHandler**</****value****>**   
> > 12.     **</****property****>**   
> > 13.                                                                   
> > 14.     **<****property****>**   
> > 15.         **<****name****>**yarn.resourcemanager.scheduler.address**</****name****>**   
> > 16.         **<****value****>**node1:8030**</****value****>**   
> > 17.     **</****property****>**   
> > 18.                                                                                                            
> > 19.     **<****property****>**   
> > 20.         **<****name****>**yarn.resourcemanager.resource-tracker.address**</****name****>**   
> > 21.         **<****value****>**node1:8031**</****value****>**  
> > 22.     **</****property****>**   
> > 23.                                                                                                            
> > 24.     **<****property****>**   
> > 25.         **<****name****>**yarn.resourcemanager.address**</****name****>**   
> > 26.         **<****value****>**node1:8032**</****value****>**   
> > 27.     **</****property****>**   
> > 28.                                                                                                            
> > 29.     **<****property****>**   
> > 30.         **<****name****>**yarn.resourcemanager.admin.address**</****name****>**   
> > 31.         **<****value****>**node1:8033**</****value****>**   
> > 32.     **</****property****>**                                                                                                                                                                                                                 
> > 33.                                                                                                            
> > 34.     **<****property****>**   
> > 35.         **<****name****>**yarn.resourcemanager.webapp.address**</****name****>**   
> > 36.         **<****value****>**node1:80**</****value****>**   
> > 37.     **</****property****>**   
> > 38.   
> > 39.     **<****property****>**  
> > 40.         **<****name****>**yarn.nodemanager.hostname**</****name****>**  
> > 41.         **<****value****>**node3**</****value****>**   
> > 42.         **<****description****>**the nodemanagers bind to this port**</****description****>**  
> > 43.     **</****property****>**  
> > 44.       
> > 45.     **<****property****>**   
> > 46.         **<****name****>**yarn.nodemanager.webapp.address**</****name****>**   
> > 47.         **<****value****>**$\{yarn.nodemanager.hostname\}:80**</****value****>**   
> > 48.     **</****property****>**   
> > 49.       
> > 50.     **<****property****>**  
> > 51.         **<****name****>**yarn.nodemanager.address**</****name****>**  
> > 52.         **<****value****>**$\{yarn.nodemanager.hostname\}:8034**</****value****>**  
> > 53.         **<****description****>**the nodemanagers bind to this port**</****description****>**  
> > 54.     **</****property****>**  
> > 55.                                                                                                            
> > 56.     **<****property****>**   
> > 57.         **<****name****>**yarn.nodemanager.local-dirs**</****name****>**   
> > 58.         **<****value****>**$\{hadoop.tmp.dir\}/nodemanager/local**</****value****>**   
> > 59.     **</****property****>**   
> > 60.                                                                                                            
> > 61.     **<****property****>**   
> > 62.         **<****name****>**yarn.nodemanager.remote-app-log-dir**</****name****>**   
> > 63.         **<****value****>**$\{hadoop.tmp.dir\}/nodemanager/remote**</****value****>**   
> > 64.     **</****property****>**   
> > 65.                                                                                                            
> > 66.     **<****property****>**   
> > 67.         **<****name****>**yarn.nodemanager.log-dirs**</****name****>**   
> > 68.         **<****value****>**$\{hadoop.tmp.dir\}/nodemanager/logs**</****value****>**   
> > 69.     **</****property****>**   
> > 70.                                                                                                            
> > 71.     **<****property****>**   
> > 72.         **<****name****>**yarn.nodemanager.log.retain-seconds**</****name****>**   
> > 73.         **<****value****>**604800**</****value****>**   
> > 74.     **</****property****>**   
> > 75.                                                                                                            
> > 76.     **<****property****>**   
> > 77.         **<****name****>**yarn.nodemanager.resource.cpu-vcores**</****name****>**   
> > 78.         **<****value****>**2**</****value****>**   
> > 79.     **</****property****>**   
> > 80.                                                                                                            
> > 81.     **<****property****>**   
> > 82.         **<****name****>**yarn.nodemanager.resource.memory-mb**</****name****>**   
> > 83.         **<****value****>**10240**</****value****>**   
> > 84.     **</****property****>**   
> > 85.                                                                                                            
> > 86.     **<****property****>**   
> > 87.         **<****name****>**yarn.scheduler.minimum-allocation-mb**</****name****>**   
> > 88.         **<****value****>**256**</****value****>**   
> > 89.     **</****property****>**   
> > 90.                                                                                                            
> > 91.     **<****property****>**   
> > 92.         **<****name****>**yarn.scheduler.maximum-allocation-mb**</****name****>**   
> > 93.         **<****value****>**40960**</****value****>**   
> > 94.     **</****property****>**   
> > 95.                                                                                                            
> > 96.     **<****property****>**   
> > 97.         **<****name****>**yarn.scheduler.minimum-allocation-vcores**</****name****>**   
> > 98.         **<****value****>**1**</****value****>**   
> > 99.     **</****property****>**   
> > 100.                                                                                                            
> > 101.     **<****property****>**   
> > 102.         **<****name****>**yarn.scheduler.maximum-allocation-vcores**</****name****>**   
> > 103.         **<****value****>**8**</****value****>**   
> > 104.     **</****property****>**   
> > 105. **</****configuration****>**  
> > 
> > 
> > 
> 
> 6，slaves**（此处如果超过一个节点，不要填主机名，要填IP）**

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  10.25.248.17  
> > 
> >   
> >   
> >

接下来依次执行以下命令：

a)        在namenode1上执行，创建命名空间

\# hdfs zkfc -formatZK

b)       在对应的节点上启动日志程序journalnode

\# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode

c)        格式化主NameNode节点（node1）

\# hdfs namenode -format

d)       启动主NameNode节点

\# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

e)        格式备NameNode节点（node2）

\# hdfs namenode -bootstrapStandby

f)         启动备NameNode节点（node2）

\# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

g)       在两个NameNode节点（node1、node2）上执行

\# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start zkfc

h)       启动所有的DataNode节点（node3）

\# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode

i)         启动Yarn（node1）

\# cd /usr/local/hadoop &&sbin/start-yarn.sh

**7，Spark安装与配置**

> \# tar xvzf spark-0.9.0-incubating.tgz-C/usr/local
> 
> \# cd /usr/local
> 
> \# ln -s spark-0.9.0-incubating spark
> 
>  
> 
> \# vim /etc/profile
> 
> export SPARK\_HOME=/usr/local/spark
> 
> export PATH=$PATH:$SPARK\_HOME/bin
> 
>  
> 
> \# source /etc/profile
> 
>  
> 
> \# cd /usr/local/spark/conf
> 
> \# mkdir -p /data/spark/tmp
> 
> 接下来，配置两个配置文件，文件在SPARK\_HOME/conf/目录下

1，spark-env.sh

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1.  export JAVA\_HOME=/usr/local/jdk  
> > 2.  export SCALA\_HOME=/usr/local/scala  
> > 3.  export HADOOP\_HOME=/usr/local/hadoop  
> > 4.  export SPARK\_MASTER\_PORT=7077  
> > 5.  export SPARK\_MASTER\_WEBUI\_PORT=7070  
> > 6.  export SPARK\_WORKER\_CORES=2  
> > 7.  export SPARK\_WORKER\_MEMORY=1024m  
> > 8.  export SPARK\_WORKER\_INSTANCES=2  
> > 9.    
> > 10. export SPARK\_LOCAL\_DIR="/root/data/spark/tmp"  
> > 11. export SPARK\_JAVA\_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK\_LOCAL\_DIR -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:$SPARK\_HOME/logs/gc.log -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=60"  
> > 
> >   
> >

2，slaves（多个节点不能是主机名）

> > **\[html\]**  [view plain][]   [copy][view plain]
> > 
> > 1. **<****span** style="font-size: 13.3333px;"**>**10.25.248.29**</****span****>**  
> > 2. **<****span** style="font-size: 13.3333px;"**>**10.25.248.17**</****span****>**  
> > 
> > 
> > 
> 
> 分别进入每个节点的/usr/local/spark/sbin目录下，主节点执行\#./start-master.sh，子节点执行\#./start-slaves.sh，或者在主节点直接执行sbin/start-all.sh
> 
> 
>

**8，测试流程**

**1，进程运行情况测试**

> 在每个节点执行\#jps指令，若输出结果为以下内容，则测试通过，否则进入/usr/local/hadoop/logs或者/usr/local/spark/logs目录下查看log文件进行检查

主namenode

>   
>

从namenode

>   
>

datanode节点

>   
> 
> 
>   
> 
> 
>   
>

其中Master和Worker是Spark相应进程  
Hmaster和HregionServer为Hbase相应进程(这里不需要)  
DFSZKFailoverController为Zookeeper相应进程  
ResourceManager和NodeManager为Yarn相应进程

其余为Hadoop相应进程

**2，HDFS测试**

在任意节点下执行

\#hadoopfs –mkdir /test

\#hadoopfs –ls /

\#hadoopfs –put /test/test.txt

若不报错，则说明测试通过

**3，Spark测试**

**1)       Spark本地模式测试（Spark Standalone）**

> \#run-exampleorg.apache.spark.examples.SparkPi 100
> 
> \#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar ./test.txt
> 
> 在[http://node1:4040][http_node1_4040]中观察输出结果
> 
>  
> 
> \#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://\[hdfsnamespace\]/test/test.txt
> 
> 在[http://node1:4040][http_node1_4040]中观察输出结果
> 
>   
>

> > **2)       ** **Spark集群模式测试（Spark on Yarn）**

> \#spark-submit --class org.apache.spark.examples.JavaWordCount--master yarn --deploy-mode client/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://\[hdfsnamespace\]/test/test.txt
> 
>  
> 
> \#spark-submit --classorg.apache.spark.examples.JavaWordCount --master yarn --deploy-mode cluster/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://\[hdfsnamespace\]/test/test.txt
> 
> 可通过[http://node1:80][http_node1_80]进入UI界面查看
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Spark 2.0.0发布已经有一段时间了，目前公司生产环境还是使用1.6系列版本。为了测试Spark 2.0.0各方面的稳定性和计算性能，我基于CDH集群环境，搭建了Spark On Yarn集群环境。
> 
>         
> 
> 环境信息：
> 
>          CDH版本：CDH-5.7.0
> 
>                    其中，Hadoop版本：2.6.0
> 
>          Java版本：1.7.0\_80
> 
>          Scala版本：2.11.7
> 
>          Spark版本：spark-2.0.0-bin-hadoop2.6
> 
>          Spark集群规划：
> 
>         
> 
> <table style="border-collapse:collapse; border-spacing:0px; max-width:100%; color:rgb(69,69,69); font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-size:16px"> 
> <tbody style=""> 
> <tr style=""> 
> <td style=""> IP地址 </td> 
> <td style=""> 主机名 </td> 
> <td style=""> 角色 </td> 
> </tr> 
> <tr style=""> 
> <td style=""> 10.20.24.199 </td> 
> <td style=""> SZB-L0029554 </td> 
> <td style=""> Master/Worker </td> 
> </tr> 
> <tr style=""> 
> <td style=""> 10.20.24.196 </td> 
> <td style=""> SZB-L0029556 </td> 
> <td style=""> Worker </td> 
> </tr> 
> <tr style=""> 
> <td style=""> 10.20.24.198 </td> 
> <td style=""> SZB-L0029557 </td> 
> <td style=""> Worker </td> 
> </tr> 
> <tr style=""> 
> <td style=""> 10.20.24.200 </td> 
> <td style=""> SZB-L0029558 </td> 
> <td style=""> Worker </td> 
> </tr> 
> <tr style=""> 
> <td style=""> 10.20.24.201 </td> 
> <td style=""> SZB-L0029559 </td> 
> <td style=""> Worker </td> 
> </tr> 
> </tbody> 
> </table>
> 
>  
> 
> 安装步骤如下：
> 
> 1.      Spark集群的每个节点的/etc/hosts添加所有的集群主机信息
> 
> 127.0.0.1        localhost
> 
> 10.20.24.199     SZB-L0029554
> 
> 10.20.24.196     SZB-L0029556
> 
> 10.20.24.198     SZB-L0029557
> 
> 10.20.24.200     SZB-L0029558
> 
> 10.20.24.201     SZB-L0029559
> 
>  
> 
> 2.      给Spark单独规划用户和用户组
> 
> groupadd -g 8888myspark
> 
> useradd -m -d /var/lib/myspark -g spark -u 5888 myspark
> 
> passwd myspark
> 
>  
> 
> 3.      Spark集群所有节点部署Java
> 
> Java部署好之后要记得配置环境变量（/etc/profile）并source生效，如下：
> 
> export JAVA\_HOME=/usr/java/latest
> 
> export JRE\_HOME=/usr/java/latest/jre
> 
> export PATH=$PATH:$JAVA\_HOME/bin
> 
> exportCLASSPATH=./:$JAVA\_HOME/lib:$JAVA\_HOME/jre/lib
> 
>  
> 
> 验证：
> 
> \[root@SZB-L0029554 ~\]\# echo$JAVA\_HOME
> 
> /usr/java/latest
> 
> \[root@SZB-L0029554 ~\]\# java -version
> 
> java version "1.7.0\_80"
> 
> Java(TM) SE Runtime Environment(build 1.7.0\_80-b15)
> 
> Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixedmode)
> 
>  
> 
>  
> 
> 4.      Spark集群所有节点部署Scala
> 
> Scala部署好之后，配置环境变量并source生效，如下：
> 
> export SCALA\_HOME=/usr/local/scala
> 
> export PATH=.:$SCALA\_HOME/bin:$PATH
> 
>  
> 
> \[root@SZB-L0029554 ~\]\# echo$SCALA\_HOME
> 
> /usr/local/scala
> 
> \[root@SZB-L0029554 ~\]\# scala-version
> 
> Scala code runner version 2.11.7 -- Copyright 2002-2013,LAMP/EPFL
> 
>  
> 
> 5.      Spark集群的节点间创建信任关系
> 
>  
> 
> 6.      Spark集群的每个节点关闭防火墙等服务
> 
>  
> 
> 7.      Spark的Master节点安装部署
> 
> 解压缩并设置软链接，安装后的目录如下：
> 
> \[myspark@SZB-L0029554 ~\]$ ll
> 
> total 4
> 
> lrwxrwxrwx  1 myspark spark   25 Aug 8 14:20 spark -> spark-2.0.0-bin-hadoop2.6
> 
> drwxr-xr-x 13 myspark spark 4096 Aug  8 15:03 spark-2.0.0-bin-hadoop2.6
> 
>  
> 
> 配置环境变量(myspark用户家目录的.bashrc)：
> 
> exportSPARK\_HOME=/var/lib/myspark/spark
> 
> export PATH=.:$SPARK\_HOME/bin:$SPARK\_HOME/sbin:$PATH
> 
>  
> 
> 使环境变量生效：
> 
> \[myspark@SZB-L0029554 ~\]$ source .bashrc
> 
>  
> 
> 8.      Spark的Master节点配置
> 
> 8.1   创建目录并修改属组和属主
> 
> mkdir -p /hadoopdata1/sparkdata/local
> 
> mkdir -p /var/lib/myspark/spark/logs
> 
> mkdir -p /hadoopdata1/sparkdata/work
> 
> chown -R myspark:myspark/hadoopdata1/sparkdata
> 
>  
> 
> 8.2   配置conf/spark-env.sh
> 
> export SPARK\_HOME=/var/lib/myspark/spark
> 
> export JAVA\_HOME=/usr/java/jdk1.7.0\_80
> 
> export HADOOP\_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
> 
> export PATH=$PATH:$HADOOP\_HOME/bin:$HADOOP\_HOME/sbin
> 
> export HADOOP\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop
> 
> export YARN\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop
> 
> export SPARK\_LIBARY\_PATH=.:$JAVA\_HOME/lib:$JAVA\_HOME/jre/lib:$HADOOP\_HOME/lib/native
> 
> SPARK\_MASTER\_HOST=10.20.24.199
> 
> \#web页面端口
> 
> SPARK\_MASTER\_WEBUI\_PORT=28686
> 
> \#Spark的local目录
> 
> SPARK\_LOCAL\_DIRS=/hadoopdata1/sparkdata/local
> 
> \#worker目录
> 
> SPARK\_WORKER\_DIR=/hadoopdata1/sparkdata/work
> 
> \#Driver内存大小
> 
> SPARK\_DRIVER\_MEMORY=4G
> 
> \#Worker的cpu核数
> 
> SPARK\_WORKER\_CORES=16
> 
> \#worker内存大小
> 
> SPARK\_WORKER\_MEMORY=64g
> 
> \#Spark的log日志目录
> 
> SPARK\_LOG\_DIR=/var/lib/myspark/spark/logs
> 
>  
> 
> 8.3   配置slaves
> 
> SZB-L0029554
> 
> SZB-L0029556
> 
> SZB-L0029557
> 
> SZB-L0029558
> 
> SZB-L0029559
> 
>  
> 
> 8.4   配置spark-defaults.conf
> 
> \#建议开启，可以对已完成的任务记录其详细日志
> 
> spark.eventLog.enabled  true
> 
> \#eventLog的文件存放位置
> 
> spark.eventLog.dir      hdfs://SZB-L0023776:8020/myspark/eventLog/applicationHistory
> 
>  
> 
> 其他值保持默认即可。
> 
>  
> 
> 9.      将Master的Spark整个目录拷贝到Worker节点
> 
> scp -r spark-2.0.0-bin-hadoop2.6 myspark@SZB-L0029556:/var/lib/myspark/
> 
> scp -r spark-2.0.0-bin-hadoop2.6 myspark@SZB-L0029557:/var/lib/myspark/
> 
> scp -r spark-2.0.0-bin-hadoop2.6 myspark@SZB-L0029558:/var/lib/myspark/
> 
> scp -r spark-2.0.0-bin-hadoop2.6 myspark@SZB-L0029559:/var/lib/myspark/
> 
> 另外将Master节点的.bashrc环境变量也拷贝到所有的Worker节点并执行source命令使环境变量生效。
> 
>  
> 
> 10.  启动Spark集群环境
> 
> 在Master节点上启动Spark集群：
> 
> \[myspark@SZB-L0029554 ~\]$ start-all.sh
> 
>  
> 
> 查看Master节点的进程：
> 
> \[myspark@SZB-L0029554 ~\]$ jps
> 
> 29593 Master
> 
> 29709 Worker
> 
> 29970 Jps
> 
>  
> 
> 再查看其他Worker节点的进程：
> 
> \[myspark@SZB-L0029556 ~\]$ jps
> 
> 25276 Jps
> 
> 24754 Worker
> 
>  
> 
> 11.  通过Web UI查看
> 
> 我们访问[http://szb-l0029554:28686/][http_szb-l0029554_28686]页面，查看内容：
> 
> ![Image 1][]
> 
> 可以看到有5个Workers。
> 
>  
> 
> 12.  Spark环境验证
> 
> 我们先执行官网自带的计算Pi的例子：
> 
> \[myspark@SZB-L0029554 ~\]$ spark-submit --masterspark://SZB-L0029554:7077 --class org.apache.spark.examples.SparkPi --nameSparkPI /var/lib/myspark/spark/examples/jars/spark-examples\_2.11-2.0.0.jar
> 
>  
> 
> 输出结果为：
> 
> Pi is roughly 3.1383756918784593
> 
>  
> 
> 下面我们通过Yarn资源调度平台来执行相关操作：
> 
> \[myspark@SZB-L0029554 ~\]$ spark-shell --master yarn
> 
> Spark context Web UI available athttp://10.20.24.199:4040
> 
> Spark context available as 'sc'(master = yarn, app id = application\_1469373167347\_0106).
> 
> Spark session available as 'spark'.
> 
> Welcome to
> 
>      \_\_\_\_              \_\_
> 
>     / \_\_/\_\_  \_\_\_ \_\_\_\_\_/ /\_\_
> 
>    \_\\ \\/ \_ \\/ \_ \`/ \_\_/  '\_/
> 
>   /\_\_\_/ .\_\_/\\\_,\_/\_/ /\_/\\\_\\   version2.0.0
> 
>      /\_/
> 
>         
> 
> Using Scala version 2.11.8 (JavaHotSpot(TM) 64-Bit Server VM, Java 1.7.0\_80)
> 
> Type in expressions to have themevaluated.
> 
> Type :help for more information.
> 
>  
> 
> scala> val textFile =sc.textFile("/myspark/testdata/README.md")
> 
> textFile:org.apache.spark.rdd.RDD\[String\] = /myspark/testdata/README.mdMapPartitionsRDD\[1\] at textFile at <console>:24
> 
>  
> 
> scala> textFile.filter(line =>line.contains("Spark")).count()
> 
> res0: Long = 19
> 
>  
> 
> 我们上面执行的是统计HDFS文件系统上的/myspark/testdata/README.md文件里面包含Spark的行数。
> 
>  
> 
> 可以根据上面日志提示的网址[http://10.20.24.199:4040][http_10.20.24.199_4040]来查看作业执行情况，当我们使用浏览器访问[http://10.20.24.199:4040][http_10.20.24.199_4040]时会自动调整到Yarn的管理页面，比如我这里是
> 
> ![Image 1][]
> 
> 
>

[view plain]: http://blog.csdn.net/qq_23330633/article/details/52216155?locationNum=4#
[http_node1_4040]: http://node1:4040/
[http_node1_80]: http://node1/
[http_szb-l0029554_28686]: http://szb-l0029554:28686/
[Image 1]: 
[http_10.20.24.199_4040]: http://10.20.24.199:4040/