Hadoop之Lzo压缩配置

约定不等于承诺〃 2022-12-27 11:04 158阅读 0赞

# Hadoop之Lzo压缩配置 #

## 一、hadoop-lzo编译 ##

> hadoop本身并不支持lzo压缩，故需要使用twitter提供的hadoop-lzo开源组件。hadoop-lzo需依赖hadoop和lzo进行编译，编译步骤如下。

1.  **环境准备**  
    maven（下载安装，配置环境变量，修改sitting.xml加阿里云镜像）  
    gcc-c++  
    zlib-devel  
    autoconf  
    automake  
    libtool  
    通过yum安装即可，yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool
2.  **下载、安装并编译LZO**
    
    wget [http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz][http_www.oberhumer.com_opensource_lzo_download_lzo-2.10.tar.gz]
    
    tar -zxvf lzo-2.10.tar.gz
    
    cd lzo-2.10
    
    ./configure -prefix=/usr/local/hadoop/lzo/
    
    make
    
    make install
3.  **编译hadoop-lzo源码**
    
    2.1 下载hadoop-lzo的源码，下载地址：[https://github.com/twitter/hadoop-lzo/archive/master.zip][https_github.com_twitter_hadoop-lzo_archive_master.zip]
    
    2.2 解压之后，修改pom.xml
    
        <hadoop.current.version>3.1.3</hadoop.current.version>
    
    2.3 声明两个临时环境变量
    
        export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
         export LIBRARY_PATH=/usr/local/hadoop/lzo/lib
    
    2.4 编译  
    进入hadoop-lzo-master，执行maven编译命令
    
        mvn package -Dmaven.test.skip=true
    
    2.5 进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件

## 二、Hadoop相关配置 ##

**1）将编译好后的hadoop-lzo-0.4.20.jar 放入hadoop-3.1.3/share/hadoop/common/**

[hadoop@hadoop102 common]$ pwd
    /opt/module/hadoop-3.1.3/share/hadoop/common
    [hadoop@hadoop102 common]$ ls
    hadoop-lzo-0.4.20.jar

**2）同步hadoop-lzo-0.4.20.jar到hadoop103、hadoop104**

[hadoop@hadoop102 common]$ xsync hadoop-lzo-0.4.20.jar

**3）core-site.xml增加配置支持LZO压缩**

<configuration>
        <property>
            <name>io.compression.codecs</name>
            <value>
                org.apache.hadoop.io.compress.GzipCodec,
                org.apache.hadoop.io.compress.DefaultCodec,
                org.apache.hadoop.io.compress.BZip2Codec,
                org.apache.hadoop.io.compress.SnappyCodec,
                com.hadoop.compression.lzo.LzoCodec,
                com.hadoop.compression.lzo.LzopCodec
            </value>
        </property>
    
        <property>
            <name>io.compression.codec.lzo.class</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property>
    </configuration>

**4）同步core-site.xml到hadoop103、hadoop104**

[hadoop@hadoop102 hadoop]$ xsync core-site.xml

**5）启动及查看集群**

[hadoop@hadoop102 hadoop-3.1.3]$ sbin/start-dfs.sh
    [hadoop@hadoop103 hadoop-3.1.3]$ sbin/start-yarn.sh

## 三、项目经验之LZO创建索引 ##

1.  创建LZO文件的索引，LZO压缩文件的可切片特性依赖于其索引，故我们需要手动为LZO压缩文件创建索引。若无索引，则LZO文件的切片只有一个。
    
        hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer big_file.lzo
2.  测试  
    （1）将bigtable.lzo（200M）上传到集群的根目录
    
        [hadoop@hadoop102 module]$ hadoop fs -mkdir /input
        [hadoop@hadoop102 module]$ hadoop fs -put bigtable.lzo /input
    
    （2）执行wordcount程序
    
        [hadoop@hadoop102 module]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
        examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat /input 	 /output1
    
    ![在这里插入图片描述][20201210140208211.png]
    
    （3）对上传的LZO文件建索引
    
        [hadoop@hadoop102 module]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-lzo-0.4.20.jar 
         com.hadoop.compression.lzo.DistributedLzoIndexer /input/bigtable.lzo
    
    （4）再次执行WordCount程序
    
        [hadoop@hadoop102 module]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
        examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat 
        /input /output2
    
    ![在这里插入图片描述][20201210140230366.png]
3.  注意：如果以上任务，在运行过程中报如下异常  
    Container \[pid=8468,containerID=container\_1594198338753\_0001\_01\_000002\] is running 318740992B beyond the ‘VIRTUAL’

memory limit. Current usage: 111.5 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. 
    Killing container.
    Dump of the process-tree for container_1594198338753_0001_01_000002 :

解决办法：在hadoop102的/opt/module/hadoop-3.1.3/etc/hadoop/yarn-site.xml文件中增加如下配置，然后分发到hadoop103、hadoop104服务器上，并重新启动集群。

<property>
       <name>yarn.nodemanager.pmem-check-enabled</name>
       <value>false</value>
    </property>
    
    
    <property>
       <name>yarn.nodemanager.vmem-check-enabled</name>
       <value>false</value>
    </property>

[http_www.oberhumer.com_opensource_lzo_download_lzo-2.10.tar.gz]: http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
[https_github.com_twitter_hadoop-lzo_archive_master.zip]: https://github.com/twitter/hadoop-lzo/archive/master.zip
[20201210140208211.png]: /images/20221120/409420e4c51640c7adf22ef6857ba086.png
[20201210140230366.png]: https://img-blog.csdnimg.cn/20201210140230366.png