window 安装Hadoop3.0.0

ゞ浴缸里的玫瑰 2023-10-18 15:07 20阅读 0赞

第一步：Hadoop 官网下载地址：[https://hadoop.apache.org/releases.html][https_hadoop.apache.org_releases.html]。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70][]

第二步：将下载hadoop-3.0.0.tar.gz 解压至指定文件夹 C:\\hadoop

![20190304143727225.png][]

第三步：配置hadoop 涉及环境变量：

HADOOP\_HOME:

![20190304143832304.png][]

PATH :

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 1][]

第四步：涉及Hadoop配置 ：

1、修改C:/hadoop/hadoop-3.0.0/etc/hadoop/core-site.xml配置:

<configuration>  
<property>  
<name>fs.default.name</name>  
<value>hdfs://localhost:9000</value>  
</property>  
</configuration>

2、修改C:/hadoop/hadoop-3.0.0/etc/hadoop/mapred-site.xml配置:

<configuration>  
<property>  
<name>mapreduce.framework.name</name>  
<value>yarn</value>  
</property>  
</configuration>

3、在C:/hadoop/hadoop-3.0.0目录下创建data目录，作为数据存储路径：

![format_png][]

在C:/hadoop/hadoop-3.0.0/data目录下创建datanode目录；

在C:/hadoop/hadoop-3.0.0/data目录下创建namenode目录；

4、修改C:/hadoop/hadoop-3.0.0/etc/hadoop/hdfs-site.xml配置:

<configuration>  
  
<property>  
<name>dfs.replication</name>  
<value>1</value>  
</property>  
<property>  
<name>dfs.permissions</name>  
<value>false</value>  
</property>  
<property>  
<name>dfs.namenode.name.dir</name>  
<value>/D:/hadoop-3.0.0/data/namenode</value>  
</property>  
<property>  
<name>fs.checkpoint.dir</name>  
<value>/D:/hadoop-3.0.0/data/snn</value>  
</property>  
<property>  
<name>fs.checkpoint.edits.dir</name>  
<value>/D:/hadoop-3.0.0/data/snn</value>  
</property>  
<property>  
<name>dfs.datanode.data.dir</name>  
<value>/D:/hadoop-3.0.0/data/datanode</value>  
</property>  
</configuration>

5、修改C:/hadoop/hadoop-3.0.0/etc/hadoop/yarn-site.xml配置:

<configuration>  
  
<property>  
<name>yarn.nodemanager.aux-services</name>  
<value>mapreduce\_shuffle</value>  
</property>  
<property>  
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
<value>org.apache.hadoop.mapred.ShuffleHandler</value>  
</property>  
</configuration>  
6、修改C:/hadoop/hadoop-3.0.0/etc/hadoop/hadoop-env.cmd配置，找到"set JAVA\_HOME=%JAVA\_HOME%"替换为set JAVA\_HOME=set JAVA\_HOME=C:\\PROGRA~1\\Java\\jdk1.8.0\_162

7、bin目录替换，至https://github.com/steveloughran/winutils下载解压

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 2][]

找到对应的版本后完整替换bin目录即可

至此配置完成.

第五步：启动Hadoop 服务

启动Hadoop 格式化：C:\\Users\\zzg>hadoop namenode -format

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 3][]

进入C:\\hadoop\\hadoop-3.0.0\\sbin 文件夹，通过start-all.cmd启动服务： C:\\Users\\zzg>start-all.cmd

此时可以看到同时启动了如下4个服务：

Hadoop Namenode

Hadoop datanode

YARN Resourc Manager

YARN Node Manager

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 4][]

第六步：Hadoop涉及应用

1、通过http://127.0.0.1:8088/即可查看集群所有节点状态：

![format_png 1][]

2、访问http://localhost:9870/即可查看文件管理页面：

进入文件管理页面：

![format_png 2][]

创建目录：

![format_png 3][]

上传文件

![format_png 4][]

上传成功

![format_png 5][]

注意：在之前的版本中文件管理的端口是50070，在3.0.0中替换为了9870端口，具体变更信息来源如下官方说明 http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html\#Web\_Interface

![format_png 6][]

3、通过hadoop命令行进行文件操作：

mkdir命令创建目录：hadoop fs -mkdir hdfs://localhost:9000/user

![format_png 7][]

如下新增的user目录

![format_png 8][]

put命令上传文件：hadoop fs -put C:\\Users\\songhaifeng\\Desktop\\11.txt hdfs://localhost:9000/user/

![format_png 9][]

如下上传文件

![format_png 10][]

ls命令查看指定目录文件列表：hadoop fs -ls hdfs://localhost:9000/user/

[https_hadoop.apache.org_releases.html]: https://hadoop.apache.org/releases.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20190304143502902.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n,size_16,color_FFFFFF,t_70
[20190304143727225.png]: https://img-blog.csdnimg.cn/20190304143727225.png
[20190304143832304.png]: https://img-blog.csdnimg.cn/20190304143832304.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20190304143922805.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n,size_16,color_FFFFFF,t_70
[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzFlMzRmZmEyZTg5OGFjZTkwMGQxOGVjY2Q0MGI5YTU1MDA5LmpwZw?x-oss-process=image/format,png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20190304144338248.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20190304144551939.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20190304144928286.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3V6aGl3ZW5nYW5n,size_16,color_FFFFFF,t_70
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzQ5MTZlMDM1OGZhODY1NzIwZDk3YWEwOWQ2MTI5ZjI3M2I2LmpwZw?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0L2YwYWRlOTNmNTQyOWQ5NWM1YTgwNjg1MGQ5NmUzNGUwM2RhLmpwZw?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzM2NmVmMGUzNDZhYzY5ZmMxMTMwNWRhMTdhZmMwZWY5ZWUzLmpwZw?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzQzMjdjOTU1OGNlOWJjMWQ3NjRlYTliZTI1MjM5N2IzY2ZjLmpwZw?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzJkZjRlNzM2NzFhMTg2OGQyMTM0MTdjMjJjNzQzM2JjMTU4LmpwZw?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzBjMTBmMjg5YWIzZDIzNjM4YTZlZWZjOWMyZmRkNWM3ZmIwLmpwZw?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0L2U4MjljYTZhZmNiM2IyZjVkMjVmZDAyYzE1MjU3YzU2MmUyLmpwZw?x-oss-process=image/format,png
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzI5ZTc0NmUxZTk4YjQ1ZjljNTFmMWU0YWFmNmI4YzQxYjRlLmpwZw?x-oss-process=image/format,png
[format_png 9]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzUwYzQ1MzU3ZDA1OGNiYjUxY2U4NmQ3ZDRlNjNlMTdjN2E0LmpwZw?x-oss-process=image/format,png
[format_png 10]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9vc2NpbWcub3NjaGluYS5uZXQvb3NjbmV0LzFlNzRhMDgwZTJjMWU1OTk4ZDU4ODc4Zjc5NjcwMGE0NTZhLmpwZw?x-oss-process=image/format,png