Hadoop集群配置安装

蔚落 2024-04-18 07:13 102阅读 0赞

安装包资料链接: https://pan.baidu.com/s/1YFbBJNesfhl0eBZ9B3pwKw 提取码: rmba

### 安装准备 ###

\[root@tandelin hadoop\]\# cd /usr/local/  
\[root@tandelin hadoop\]\# ls  
\[root@tandelin hadoop\]\# tar -zxvf hadoop-2.5.0-cdh5.3.6.tar.gz  
\[root@tandelin hadoop\]\# ls

##### 重命名文件 #####

\[root@tandelin hadoop\]\# mv hadoop-2.5.0-cdh5.3.6 hadoop  
\[root@tandelin hadoop\]\# ls  
\[root@tandelin hadoop\]\# cd hadoop/  
\[root@tandelin hadoop\]\# ls

### 修改配置文件 ###

\#配置环境变量  
vi ~/.bashrc  
\#添加hadoop环境变量  
\[root@tandelin hadoop\]\# vi ~/.bashrc  
export HADOOP\_HOME=/usr/local/hadoop  
export PATH= P A T H : PATH: PATH:HADOOP\_HOME/bin:$HADOOP\_HOME/sbin  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70]  
\[root@tandelin hadoop\]\# source ~/.bashrc

### 创建namedata ###

\[root@tandelin hadoop\]\# mkdir /usr/local/data

### 查看hadoop是否安装配置正确 ###

\[root@tandelin hadoop\]\# hadoop version  
\[root@tandelin hadoop\]\# java -version  
\[root@tandelin hadoop\]\# vim hadoop-env.sh  
\[root@tandelin hadoop\]\# ls  
\[root@tandelin hadoop\]\# vim /etc/hadoop/hadoop-env.sh  
\[root@tandelin hadoop\]\# find / -name hadoop-env.sh  
\[root@tandelin hadoop\]\# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh  
\[root@tandelin hadoop\]\# source /etc/profile  
\[root@tandelin hadoop\]\# hadoop version

#### hadoop启动过程中出现Error: JAVA\_HOME is not set and could not be found.解决办法 ####

###### 1.在linux下,如何找java的安装路径 ######

\[root@tandelin hadoop\]\# whereis java  
java: /usr/bin/java /usr/share/java /usr/lib/jvm/java-8-openjdk-amd64/bin/java

###### 2.修改hadoop目录下的/etc/hadoop/hadoop-env.sh文件 ######

安装好hadoop后，启动终端和启动hadoop时出现Error: JAVA\_HOME is not set and could not be found.错误,需要修改hadoop目录下的/etc/hadoop/hadoop-env.sh文件。用vim hadoop-env.sh打开文件

\[root@tandelin hadoop\]\# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

export JAVA\_HOME=$\{JAVA\_HOME\}

export JAVA\_HOME=/usr/  
export PATH= J A V A H O M E / b i n : JAVA\_HOME/bin: JAVAHOME/bin:PATH  
export CLASSPATH=.: J A V A H O M E / l i b / d t . j a r : JAVA\_HOME/lib/dt.jar: JAVAHOME/lib/dt.jar:JAVA\_HOME/lib/tools.jar

![在这里插入图片描述][20190903174302645.png]

### 环境配置 ###

修改/usr/local/hadoop/etc/hadoop/路径下的配置文件：core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、slaves

\[root@tandelin hadoop\]\# cd /usr/local/hadoop/etc/hadoop/

##### **修改core-site.xml** #####

[root@tandelin hadoop]#  vim core-site.xml
    
    
    <property>
      <name>fs.default.name</name>
      <value>hdfs://tandelin:9000</value>
    </property>

##### **修改hdfs-site.xml** #####

[root@tandelin hadoop]#  vim hdfs-site.xml
    <property>
      <name>dfs.name.dir</name>
      <value>/usr/local/data/namenode</value>
    </property>
    <property>
      <name>dfs.data.dir</name>
      <value>/usr/local/data/datanode</value>
    </property>
    <property>
      <name>dfs.tmp.dir</name>
      <value>/usr/local/data/tmp</value>
    </property>
    <property>
      <name>dfs.replication</name>
      <value>3</value>
    </property>

##### 修改mapred-site.xml #####

先重命名mapred-site.xml.template为mapred-site.xml

[root@tandelin hadoop]# vim mapred-site.xml 
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>

##### **修改yarn-site.xml** #####

[root@tandelin hadoop]# vim yarn-site.xml
    
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>tandelin</value>
    </property>
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>

##### **修改slaves文件** #####

[root@tandelin hadoop]#  vim slaves
    
    tandelin

### 在另外两台机器上搭建tandelin2、tandelin3 ###

##### 在tandelin机器上执行 #####

搭建第2台机器：

[root@tandelin hadoop]#    cd /usr/local
      [root@tandelin hadoop]#  scp -r hadoop root@tandelin2:/usr/local
      [root@tandelin hadoop]#   scp ~/.bashrc root@tandelint2:~/

##### 再在tandelin2上执行 #####

对.bashrc文件进行source，以让它生效。

[root@tandelin2 hadoop]#   source ~/.bashrc

##### 创建data目录。 #####

[root@tandelin2 hadoop]#  mkdir /usr/local/data

使用以下命令进行查看

[root@tandelin2 hadoop]# hadoop version
    [root@tandelin2 hadoop]# yarn version

##### 重新启动hdfs集群 #####

格式化namenode  
在tandelin上执行以下命令

[root@tandelin hadoop]# hdfs namenode -format

##### 启动hdfs集群 #####

\[root@tandelin2 hadoop\]\# start-dfs.sh

##### 验证启动是否成功 #####

\[root@tandelin2 hadoop\]\# jps

hdfs管理界面：  
http://tandelin:50070  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70 2]

#### 启动yarn集群 ####

\[root@tandelin2 hadoop\]\# start-yarn.sh

参考资料：https://www.jianshu.com/p/92720b8ae7d2

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/f0e26c1fe5144ab5b9b26b98e27311eb.png
[20190903174302645.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/073e57804cfc401ba4684ff8426e5f5d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/08709cb53d58439ba4b1225e05507bdb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RhbmRlbGlu_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/312fd2f7b3f34844a212ba4bb737afd8.png