hadoop2集群安装和测试之软件安装配置

桃扇骨 2022-08-08 20:58 193阅读 0赞

**二、安装JDK**  
使用hadoop用户，下载jdk-7u79-linux-x64.gz到用户根目录  
tar -zxvf   jdk-7u79-linux-x64.gz  
配置java环境变量  
vi ~/.bash\_profile \#添加  
JAVA\_HOME=/home/hadoop/jdk1.7.0\_79  
JRE\_HOME=/home/hadoop/jdk1.7.0\_79/jre  
CLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar  
PATH=$JAVA\_HOME/bin:$PATH  
export JAVA\_HOME JRE\_HOME CLASSPATH  PATH  
使配置生效  
source  ~/.bash\_profile

配置其他机器java

**三、安装hadoop**  
下载hadoop-2.6.0.tar.gz安装文件到hadoop根目录，解压：tar -zxvf  hadoop-2.6.0.tar.gz  
**1、配置hadoop**  
cd /home/hadoop/hadoop-2.6.0/etc/hadoop  
1）指定JAVA\_HOME  
vi etc/hadoop/hadoop-env.sh   
export JAVA\_HOME=/home/hadoop/jdk1.7.0\_79  
vi etc/hadoop/yarn-env.sh  
export JAVA\_HOME=/home/hadoop/jdk1.7.0\_79  
2）配置core-site.xml  
configuration增加配置

<property>
    		<name>hadoop.tmp.dir</name>
    		<value>/home/hadoop/hadoop-2.6.0/temp</value>
    </property>
    <property>
    		<name>fs.defaultFS</name>
    		<value>hdfs://study-90:9000</value>
    </property>
    <property>
    		<name>io.file.buffer.size</name>
    		<value>4096</value>
    </property>

3）配置hdfs-site.xml   
configuration增加配置

<property>
    	<name>dfs.namenode.name.dir</name>
    	<value>file:/home/hadoop/hadoop-2.6.0/dfs/namenode</value>
    	<final>true</final>
    </property>
    <property>
    	<name>dfs.datanode.data.dir</name>
    	<value>file:/home/hadoop/hadoop-2.6.0/dfs/datanode</value>
    	<final>true</final>
    </property>
    <property>
    	<name>dfs.replication</name>
    	<value>2</value>
    </property>
    <property>
    	<name>dfs.nameservices</name>
    	<value>hadoop-cluster</value>
    </property>
    <property>
    	<name>dfs.namenode.secondary.http-address</name>
    	<value>study-90:50090</value>
    </property>
    <property>
    	<name>dfs.webhdfs.enabled</name>
    	<value>true</value>
    </property>
    <property>
    	<name>dfs.permissions</name>
    	<value>false</value>
    </property>

4）配置mapred-site.xml  
configuration增加配置

<property>
    	<name>mapreduce.framework.name</name>
    	<value>yarn</value>
    	<final>true</final>
    </property>
    <property>
    	<name>mapreduce.jobtracker.http.address</name>
    	<value>study-90:50030</value>
    </property>
    <property>
      <name>mapreduce.jobhistory.address</name>
      <value>study-90:10020</value>
    </property>
    <property>
      <name>mapreduce.jobhistory.webapp.address</name>
      <value>study-90:19888</value>
    </property>
    <property>
    	<name>mapred.system.dir</name>
    	<value>file:/home/hadoop/mapred/system</value>
    	<final>true</final>
    </property>
    <property>
    	 <name>mapred.local.dir</name>
    	 <value>file:/home/hadoop/mapred/local</value>
    	 <final>true</final>
    </property>
    <property>
    	<name>mapred.job.tracker</name>
    	<value>http://study-90:9001</value>
    </property>

5）配置yarn-site.xml  
configuration增加配置

<property>
    	<name>yarn.nodemanager.aux-services</name>
    	<value>mapreduce_shuffle</value>
    </property>
     <property>
    	<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
    <description>The hostname of the RM.</description>
    <name>yarn.resourcemanager.hostname</name>
    <value>study-90</value>
    </property>  
    <property>
    	<name>yarn.resourcemanager.scheduler.address</name>
    	<value>study-90:8030</value>
    </property>
    <property>
    	<name>yarn.resourcemanager.resource-tracker.address</name>
    	<value>study-90:8031</value>
    </property>
    <property>
    	<name>yarn.resourcemanager.address</name>
    	<value>study-90:8032</value>
    </property>
    <property>
    	<name>yarn.resourcemanager.admin.address</name>
    	<value>study-90:8033</value>
    </property>
    <property>
    	<name>yarn.resourcemanager.webapp.address</name>
    	<value>study-90:8088</value>
    </property>

6）格式化hdfs  
将hadoop安装目录复制到其他机器对应的目录  
scp -r hadoop-2.6.0  hadoop@study-91:/home/hadoop/hadoop-2.6.0  
格式化hdfs  
bin/hadoop namenode -format  
如果新增机器，删除/home/hadoop/hadoop-2.6.0/temp目录重新格式化  
**3、启动hadoop**  
启动：sbin/start-all.sh  
关闭：sbin/stop-all.sh  
可以通过jps，查看进程是否启动  
$ jps  \#namenode节点  
5620 ResourceManager  
5389 NameNode  
$ jps  \#datanode节点  
3161 NodeManager  
3063 DataNode  
启动完毕，通过访问http://192.168.1.90:8088/cluster可以查看集群节点状态  
![Center][]  
访问http://192.168.1.90:50070/dfshealth.html\#tab-datanode查看数据节点

![Center 1][]  
访问查看hdfs文件：http://192.168.1.90:50070/explorer.html\#/

**常用命令**：  
查看文件列表  bin/hadoop fs -ls  /  
创建文件目录  bin/hadoop fs -mkdir /user  
删除文件   bin/hadoop fs -rm /user  
上传文件   bin/hadoop fs –put ./newFile /user/  
下载文件   bin/hadoop fs –get /user/newFile newFile1  
查看文件   bin/hadoop fs –cat /user/newFile  
提交MAPREDUCE JOB   bin/hadoop jar /home/admin/hadoop/job.jar \[jobMainClass\] \[jobArgs\]  
杀死某个正在运行的JOB  bin/hadoop job -kill job\_201005310937\_0053

[Center]: /images/20220731/b2c0f6d3b2994ed18ca15ee7b718ef99.png
[Center 1]: /images/20220731/5f3046f56a294513b3ea3d8c14f920e3.png