Mac 下编译 Hadoop

Myth丶恋晨 2022-02-02 10:53 300阅读 0赞

# Mac 下编译 Hadoop-2.9.2 #

## 系统环境 ##

系统： Mac OS_10.14.4
    maven： Apache Maven 3.6.0
    jdk: jdk_1.8.0_201
    ProtocolBuffer： portoc-2.5.0
    zlib： 1.2.3-lib
    OpenSSL： 1_0_2r
    cmale： 3.14.3-win64-x64
    Cygwin： 2.897_x86_64
    hadoop： hadoop-2.9.2
    snappy:

## 搭建编译环境 ##

1.java环境在oracle官网下载安装安装好配置好环境变量
    
    2.maven我用的是brew直接安装的`brew install maven`
    
    3.cmake 使用brew安装 `brew install cmake`
    
    4.openssl 使用brew安装 `brew install openssl`
    
    5.snappy 使用brew安装 `brew install snappy`
    
    6.zlib 使用brew 安装 `brew install zlib`
    
    7.protobuf由于必须使用2.5.0版本需要自行[下载](https://github.com/protocolbuffers/protobuf/releases)安装
    
    下载后解压后进入protobuf目录
    
    # 配置
    $ ./configure --prefix=/usr/local/Cellow/protoc/2.5/
    
    # 编译
    $ make
    
    # 安装
    $ make install
    
    # 配置环境变量 `~/.bash_profile` 或 `~/.zshrc`
    export PATH="/usr/local/Cellow/protoc/2.5/bin:$PATH"
    
    # 验证protoc版本
    $ protoc --version

## 过程中可能遇到的问题 ##

*  1.\[ERROR\] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1  
    \[ERROR\] around Ant part …… @ 5:153 in /Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

解决办法： 由于这个xml中的编译指令中需要用到一个环境变量`OPENSSL_ROOT_DIR`由于我们前面没有配置所以他在执行的时候就会报错，需要在`~/.bash_profile` 或 `~/.zshrc`中指定`OPENSSL_ROOT_DIR`的安装目录我的配置是`export OPENSSL_ROOT_DIR="/usr/local/Cellor/openssl/1.0.2r"`

*  2.由于OpenSSL爆出有重大的漏洞Mac OS没有使用OpenSSL而是使用了`LibreSSl`可以在你的Mac下执行`which openssl` 然后找到`/usr/bin/openssl` 执行`/usr/bin/openssl version` 就会出现LibreSSl 2.6.4，但是编译hadoop必须要使用openssl，可以吧openssl的引用改一个名称，但是Mac是不允许你关闭Mac的保护模式但是不建议这样做，我们可以修改系统的shell的环境变量使自己配置的环境变量覆盖系统的环境变量，例如：`export PATH="/usr/local/Cellor/openssl/1.0.2r/bin:$PATH"`

## 编译hadoop ##

# 下载[hadoop源码](https://hadoop.apache.org/releases.html)
    
    # 解压缩后进入hadoop源码路径
    
    # 执行一条maven命令（你的maven需要配置到环境变量中）
    $ mvn package -Pdist,native -DskipTests -Dtar
    
    # 编译过程不会太长，但是下载各种依赖就会浪费很长时间，耐心等待！
    
    # 过程中可能会出现各各样的问题我只是列举了我花费比较久时间解决的问题，如果无法解决可以留言。

## 编译完成 ##

**经过漫长的编译终于出现了令人兴奋的画面**

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
    [INFO]
    [INFO] Apache Hadoop Main ................................. SUCCESS [  1.679 s]
    [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.158 s]
    [INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.071 s]
    [INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.897 s]
    [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.297 s]
    [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.699 s]
    [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.280 s]
    [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.940 s]
    [INFO] Apache Hadoop Auth ................................. SUCCESS [  4.245 s]
    [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.548 s]
    [INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
    [INFO] Apache Hadoop NFS .................................. SUCCESS [  3.707 s]
    [INFO] Apache Hadoop KMS .................................. SUCCESS [  9.596 s]
    [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.046 s]
    [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.270 s]
    [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 46.683 s]
    [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  7.154 s]
    [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.156 s]
    [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  3.215 s]
    [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.857 s]
    [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.849 s]
    [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.100 s]
    [INFO] Apache Hadoop YARN ................................. SUCCESS [  0.073 s]
    [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 10.318 s]
    [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.624 s]
    [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.413 s]
    [INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.091 s]
    [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  8.550 s]
    [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 26.322 s]
    [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.491 s]
    [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  4.873 s]
    [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  3.360 s]
    [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 16.652 s]
    [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.993 s]
    [INFO] Apache Hadoop YARN Client .......................... SUCCESS [  4.276 s]
    [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.371 s]
    [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.464 s]
    [INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.747 s]
    [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  5.310 s]
    [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.894 s]
    [INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.053 s]
    [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.099 s]
    [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.366 s]
    [INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.061 s]
    [INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.098 s]
    [INFO] Apache Hadoop YARN Project ......................... SUCCESS [  7.639 s]
    [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.157 s]
    [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 15.248 s]
    [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  9.780 s]
    [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.561 s]
    [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  8.028 s]
    [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.123 s]
    [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  4.353 s]
    [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.349 s]
    [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.550 s]
    [INFO] Apache Hadoop MapReduce ............................ SUCCESS [  3.609 s]
    [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.987 s]
    [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  3.530 s]
    [INFO] Apache Hadoop Archives ............................. SUCCESS [  1.456 s]
    [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.654 s]
    [INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.363 s]
    [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.833 s]
    [INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.550 s]
    [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.378 s]
    [INFO] Apache Hadoop Extras ............................... SUCCESS [  2.101 s]
    [INFO] Apache Hadoop Pipes ................................ SUCCESS [  6.900 s]
    [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.804 s]
    [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  7.562 s]
    [INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.022 s]
    [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  9.670 s]
    [INFO] Apache Hadoop Client ............................... SUCCESS [  5.619 s]
    [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.638 s]
    [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.075 s]
    [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 10.540 s]
    [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 12.875 s]
    [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.670 s]
    [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.058 s]
    [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:02 min]
    [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.473 s]
    [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.042 s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  09:17 min
    [INFO] Finished at: 2019-05-06T05:03:29-07:00
    [INFO] ------------------------------------------------------------------------
    MacBookPro:hadoop-2.9.2-src leone$

## 大功告成编译后的hadoop放在`hadoop-2.9.2-src\hadoop-dist\target`目录下。 ##