Windows本地配置hadoop运行环境

快来打我* 2022-10-02 13:49 239阅读 0赞

windows本地配置hadoop运行环境，可以帮助我们更好进行开发或者hadoop学习。  
本文介绍了如何搭建一个hadooop伪运行环境。

#### 一、下载安装包并配置环境变量 ####

下载hadoop binary安装安装包：  
[https://hadoop.apache.org/releases.html][https_hadoop.apache.org_releases.html]  
下载完后配置环境变量：  
HADOOP\_HOME=D:\\hadoop-2.6.0  
path里面添加：%HADOOP\_HOME%/bin

#### 二、修改hadoop配置文件 ####

进入hadoop安装目录D:\\hadoop-2.6.0\\etc\\hadoop，修改D:\\hadoop-2.6.0\\etc\\hadoop下面的  
core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    
    
    
    <configuration>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000/</value>
      </property>
      <property>
        <name>io.native.lib.available</name>
        <value>false</value>
      </property>
      <property>
        <name>hadoop.native.lib</name>
        <value>false</value>
      </property>
      <property>
        <name>io.compression.codecs</name>
        <value>org.apache.hadoop.io.compress.GzipCodec,
               org.apache.hadoop.io.compress.DefaultCodec,
               com.hadoop.compression.lzo.LzoCodec,
               com.hadoop.compression.lzo.LzopCodec,
               org.apache.hadoop.io.compress.BZip2Codec,
               org.apache.hadoop.io.compress.SnappyCodec
            </value>
    </property>
    <property>
        <name>io.compression.codec.lzo.class</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property>
    </configuration>

hdfs-site.xml

mapred-site.xml

<?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    
    
    
    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        <property>
            <name>mapred.compress.map.output</name>
            <value>true</value>
        </property>
        <property>
            <name>mapred.map.output.compression.codec</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property> 
        <property> 
            <name>mapred.child.env</name> 
            <value>LD_LIBRARY_PATH=</value>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        <property>
            <name>mapred.compress.map.output</name>
            <value>true</value>
        </property>
        <property>
            <name>mapred.map.output.compression.codec</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property> 
        <property> 
            <name>mapred.child.env</name> 
            <value>LD_LIBRARY_PATH=D:\hadoop-2.7.3-win64\lib</value> 
        </property>
    </configuration>

然后到bin目录下执行：  
hdfs namenode -format  
再到sbin目录下执行：  
start-all.cmd  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2EyMDA4MjIxNDYwODU_size_16_color_FFFFFF_t_70]

#### 三、安装完后验证 ####

浏览器输入：http://localhost:8088可以打开管理界面  
命令行输入：

D:\hadoop-2.6.0\sbin>hadoop fs -ls /
    Found 2 items
    drwxr-xr-x   - HP supergroup          0 2019-06-19 10:28 /data
    drwxr-xr-x   - HP supergroup          0 2019-06-19 11:00 /input
    
    D:\hadoop-2.6.0\sbin>hadoop fs -mkdir /data1
    
    D:\hadoop-2.6.0\sbin>hadoop fs -ls /
    Found 3 items
    drwxr-xr-x   - HP supergroup          0 2019-06-19 10:28 /data
    drwxr-xr-x   - HP supergroup          0 2019-06-19 11:20 /data1
    drwxr-xr-x   - HP supergroup          0 2019-06-19 11:00 /input

[https_hadoop.apache.org_releases.html]: https://hadoop.apache.org/releases.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2EyMDA4MjIxNDYwODU_size_16_color_FFFFFF_t_70]: /images/20220112/91e14f490083440182c6351bd8dcc71e.png