Ubuntu下 Hadoop 1.2.1 配置安装

谁践踏了优雅 2021-11-09 10:06 355阅读 0赞

尝试进入大数据领域开发，从目前最火的Hadoop入手是必须的，本文一步步记录了 Ubuntu 环境安装Hadoop 1.2.1 ,搭建单机模式以及伪分布式模式，希望对各位和我一样的菜鸟有所帮助，共同学习进步，欢迎交流。尝试2.1安装失败，由于文件结构不太一样缺乏相关资料，也非做运维的，遂放弃改用成熟的hadoop 1.2.1版本。

\----------------------------------------单机模式-----------------------------------------

下载：hadoop-1.2.1.tar.gz

解压：

**\[plain\]** [view plain][] [copy][view plain]

1.  tar -xzvf hadoop-1.2.1.tar.gz

转移：

**\[plain\]** [view plain][] [copy][view plain]

1.  mv hadoop-1.2.1/ /opt

进入conf文件夹hadoop-env.sh配置环境变量

![SouthEast][]

让环境变量配置生效source，是source,不是sh,补充下sh和.的区别，[参考][Link 1]

**\[plain\]** [view plain][] [copy][view plain]

1.  source hadoop-env.sh

检测是否安装成功

![SouthEast 1][]

**\[html\]** [view plain][] [copy][view plain]

单机版hadoop安装成功，测试一下

**\[html\]** [view plain][] [copy][view plain]

1.  mkdir input

**\[html\]** [view plain][] [copy][view plain]

1.  cp -R conf input/

运行wordcount程序并将结果保存到output

hadoop hadoop-examples-1.2.1.jar

错误: 找不到或无法加载主类 hadoop-examples-1.2.1.jar

hadoop **jar** hadoop-examples-1.2.1.jar wordcount input output 发现错误，粗心害死猫....修改后运行,OK!

![SouthEast 2][]

单机模式成功， 接下来搭建伪分布式

\-------------------------------------------华丽的分割线 伪分布式-----------------------------------------

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1$ mkdir tmp
2.  dat@dat-HP:/opt/hadoop-1.2.1$ mkdir hdfs
3.  dat@dat-HP:/opt/hadoop-1.2.1$ mkdir hdfs/name
4.  dat@dat-HP:/opt/hadoop-1.2.1$ mkdir hdfs/data

编辑core-site.xml

**\[plain\]** [view plain][] [copy][view plain]

1.  <configuration>
2.  <property>
3.  <name>fs.default.name</name>
4.  <value>hdfs://localhost:9000</value>
5.  </property>
6.  <property>
7.  <name>hadoop.tmp.dir</name>
8.  <value>/opt/hadoop-1.2.1/tmp</value>
9.  </property>
10. </configuration>

hdfs-site.xml:

**\[plain\]** [view plain][] [copy][view plain]

1.  <configuration>
2.  <property>
3.  <name>dfs.replication</name>
4.  <value>1</value>
5.  </property>
6.  <property>
7.  <name>dfs.name.dir</name>
8.  <value>/opt/hadoop-1.2.1/hdfs/name</value>
9.  </property>
10. <property>
11. <name>dfs.data.dir</name>
12. <value>/opt/hadoop-1.2.1/hdfs/data</value>
13. </property>
14. </configuration>

mapred-site.xml:

**\[plain\]** [view plain][] [copy][view plain]

1.  <configuration>
2.  <property>
3.  <name>mapred.job.tracker</name>
4.  <value>localhost:9001</value>
5.  </property>
6.  </configuration>

格式化HDFS(建议只格式化一次，多次格式化据传后边会引起问题，本人没有尝试...):

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/conf$ source hadoop-env.sh
2.  dat@dat-HP:/opt/hadoop-1.2.1/conf$ hadoop namenode -format

![SouthEast 3][]

启动Hadoop:

执行执行bin/start-all.sh来启动所有服务，包括namenode,datanode，start-all.sh脚本用来装载守护进程

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/bin$ start-all.sh

![SouthEast 4][]

用java的jps命令验证是否安装成功：

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1$ jps

![SouthEast 5][]

少一个DataNode进程没有启动，通过查看上边现实的路径下的日志发现：

![SouthEast 6][]

很明显，data文件夹的权限不对，修改权限：

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/hdfs$ chmod 755 data

关闭hadoop,重新启动：

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/bin$ stop-all.sh

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/bin$ start-all.sh

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/bin$ jps

![SouthEast 7][]  
  
6个进程全部启动了，此时安装正常，可以进行Hadoop的相关操作了。

检查运行状态：可以通过下面的操作来查看服务是否正常，在Hadoop中用于监控集群健康状态的Web界面

http://localhost:50030/jobtracker.jsp - JobTracker状态  
http://localhost:50060/tasktracker.jsp - Task Tracker 状态  
http://localhost:50070/dfshealth.jsp - NameNode 状态

hadoop的伪分布模式已经安装成功，再次在伪分布模式下运行一下WordCount来感受MapReduce过程：

这时注意程序是在文件系统dfs运行的，创建的文件也都基于文件系统：

首先在dfs中创建input目录

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/conf$ hadoop dfs -mkdir input
2.  Warning: $HADOOP\_HOME is deprecated.

将conf中的文件拷贝到dfs中的input

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1$ hadoop dfs -copyFromLocal conf/\* input

在伪分布式模式下运行WordCount

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1$ hadoop jar hadoop-examples-1.2.1.jar wordcount input output

![SouthEast 8][]

INFO mapred.JobClient: map 100% reduce 0% 到这里卡死不动了。

解决办法：查询鸟文论坛帖子（[http://stackoverflow.com/questions/10165549/hadoop-wordcount-example-stuck-at-map-100-reduce-0][http_stackoverflow.com_questions_10165549_hadoop-wordcount-example-stuck-at-map-100-reduce-0]）发现，修改/etc/hosts文件，加入主机名和IP地址的映射。

检查主机名：

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1/conf$ hostname

![SouthEast 9][]

修改/etc/hosts文件：

![SouthEast 10][]

重新来过：

**\[plain\]** [view plain][] [copy][view plain]

1.  hadoop dfs -rmr output/
2.  hadoop jar hadoop-examples-1.2.1.jar wordcount input output

依旧不起作用...........重启，再重新来过.................ok！！！

![SouthEast 11][]

oK!!看见没有，你成功了！！！！！！

检测一下输出信息：

**\[plain\]** [view plain][] [copy][view plain]

1.  dat@dat-HP:/opt/hadoop-1.2.1$ hadoop dfs -cat output/\*

![SouthEast 12][]

ok!!此时伪分布式系统环境已可以正常使用~~~！

YOU GET YOUR FOOT IN THE DOOR！

转载于:https://www.cnblogs.com/devtao/p/3389079.html

[view plain]: http://blog.csdn.net/wyswlp/article/details/10564847
[SouthEast]: /images/20211109/429ba4849d0041ee87c512787c932923.png
[Link 1]: http://blog.csdn.net/wyswlp/article/details/10564285
[SouthEast 1]: /images/20211109/1283e7fdd0234ffa8869fd1fab1d5457.png
[SouthEast 2]: /images/20211109/2c5db235af084e11a456b9ce9ca2248d.png
[SouthEast 3]: /images/20211109/401d2dc8d6a74744a482993b1bada9ab.png
[SouthEast 4]: /images/20211109/a025509d00964366bb345615677dc900.png
[SouthEast 5]: /images/20211109/ba2d7273090b4e97b555b71babd05103.png
[SouthEast 6]: /images/20211109/0946bb8a0eb9437b83780c7310456a03.png
[SouthEast 7]: /images/20211109/b99cb4539b854bf1a11c0d590a5b8ea2.png
[SouthEast 8]: /images/20211109/f0db5c8f574c4a369c93bbd2395987bd.png
[http_stackoverflow.com_questions_10165549_hadoop-wordcount-example-stuck-at-map-100-reduce-0]: http://stackoverflow.com/questions/10165549/hadoop-wordcount-example-stuck-at-map-100-reduce-0
[SouthEast 9]: /images/20211109/87de577a89924f5e939d3c70863478b2.png
[SouthEast 10]: /images/20211109/8c4251b34cbb4384beb9a99d1886f88a.png
[SouthEast 11]: /images/20211109/ac810be5b63545da95f84936a0688b5c.png
[SouthEast 12]: /images/20211109/4bcdfaeefbaf49af92b9ae585652a1b3.png