大数据：Windows环境下搭建安装Hadoop详解

心已赠人 2024-02-19 21:42 57阅读 0赞

基于Windows10 64位。Hadoop-2.8.3，winutils-2.8.3。

1，下载Hadoop发布的稳定包：https://archive.apache.org/dist/hadoop/common/

下载后把文件解压，如图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70][]

本文基于Hadoop版本hadoop-2.8.3

2，当前电脑要安装有java的jdk，配置java环境和java\_home。我没修改环境变量，我是通过Windows下bat文件写入代码实现，命令行启动的bat文件代码：

set JAVA_HOME=D:\Java
    set HADOOP_HOME=E:\hadoop\hadoop-2.8.3
    
    set path=C:\Windows\System32;%JAVA_HOME%\bin;%HADOOP_HOME%\bin;
    start cls

3，此时启动第2步的bat文件，输入命令：hadoop version，然后输出如图：

![20190808151836939.png][]

表明Hadoop初步的搭建安装成功。但是此时仍然不能查看节点等，还需要完成下面步骤。

4，Hadoop在Windows下完整运行需要Windows组件支持，从

[https://github.com/steveloughran/winutils][https_github.com_steveloughran_winutils]

这里下载一个，下载完成后，解压得到若干版本的winutils，我选用2.8.3版本：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1][]

然后把上面的bin文件一股脑全部复制到第1步中解压后的hadoop-2.8.3\\bin文件夹下面，覆盖原先的bin目录文件下所有文件。

5，创建三个文件夹，namenode，datanode，tmp。如图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 2][]

6，把以下4个配置文件全部复制替换掉hadoop\\hadoop-2.8.3\\etc\\hadoop下的文件：

core-site.xml：

<?xml version="1.0" encoding="UTF-8"?>
    <configuration>
    <property>
    	<name>fs.defaultFS</name>
    	<value>hdfs://localhost:9000</value>
    </property>
    
    <property>
    	<name>hadoop.tmp.dir</name>
    	<value>/E:/hadoop/my/tmp</value>
    </property>
    </configuration>

hdfs-site.xml：

<?xml version="1.0" encoding="UTF-8"?>
    <configuration>
    <property>       
    <name>dfs.replication</name>       
    <value>1</value>   
    </property>  
    <property>
            <name>dfs.namenode.name.dir</name>
            <value>/E:/hadoop/my/namenode</value>
    </property>
    <property>
    	<name>dfs.datanode.data.dir</name>
    	<value>/E:/hadoop/my/datanode</value>
    </property>
    </configuration>

mapred-site.xml：

<?xml version="1.0"?>
    <configuration>
         <property>
          	  <name>mapreduce.framework.name</name>
         	  <value>yarn</value>
        </property>
     </configuration>

yarn-site.xml：

<?xml version="1.0"?>
    <configuration>
    	<property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    		
    		<property>
                    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                    <value>org.apache.hahoop.mapred.ShuffleHandler</value>
            </property>
     </configuration>

7，从[https://github.com/steveloughran/winutils][https_github.com_steveloughran_winutils]

下载winutils包，选择用2.8.3版本的。把\\bin下所有文件复制到hadoop\\hadoop-2.8.3\\bin下面，最后:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 3][]

8，然后在cmd窗口执行命令：

hdfs namenode -format

输出：

![20190808152916928.png][]

表明配置成功。

9，进入sbin目录下，执行命令：start-all，启动Hadoop，如图：

![20190808153255156.png][]

执行命令后，将弹出四个窗口，如图:

![20190808153323925.png][]

至此，Hadoop 2.8.3在Windows 10上的搭建完成。

在控制台输入 jps 命令，得到：

![20190808153457854.png][]

进程id和进程名。

打开网址：[http://localhost:50070][http_localhost_50070]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 4][]

打开网址：[http://localhost:8088][http_localhost_8088]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 5][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/5d7f206499234949a3603dff55abdf49.png
[20190808151836939.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/016819a7b4ba42e8b251b56b1ade7b27.png
[https_github.com_steveloughran_winutils]: https://github.com/steveloughran/winutils
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/1509ea3eea014b34a7a412b193035902.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/e4b0735144fd445e8facca46111d6490.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 3]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/7e2b8fd9f6f9433ba780a8824e1e539f.png
[20190808152916928.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/37d71c9f83484967adf6cece512c0cd1.png
[20190808153255156.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/2fbeaef8835242f8b3e930688b619b5d.png
[20190808153323925.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/fdc25f6c9a63490ebcf1c4afb2749f99.png
[20190808153457854.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/f7910cdad43e460ab42decbc201f1c42.png
[http_localhost_50070]: http://localhost:50070
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 4]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/3352390e6d754b7487c05d73c218e763.png
[http_localhost_8088]: http://localhost:8088
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3BoaWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 5]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/da885b72c44c42919a82d70dc092f7e6.png