Hadoop1.1 和Hadoop 2.4 集群安装版

我就是我 2022-08-13 19:41 166阅读 0赞

### 1 环境准备 ###

#### 1.1 硬件配置 ####

Dell 960 CPU英特尔 酷睿2 四核 Q8300 @ 2.50GHz

内存：4GB

硬盘：320GB

安装Window7,安装虚拟机

#### 1.2 软件 ####

ü CentOS6.5

ü VMware Workstation 10.0.2

ü Secure CRT 7.0

ü WinSCP 5.5.3

ü JDK 1.6.0\_43

ü Hadoop1.2.1

ü Eclipse3.6

ü Window7

#### 1.3 网络拓扑结构 ####

192.168.1.53 namenode53

192.168.1.54 datanode54

192.168.1.55 datanode55

192.168.1.56 datanode56

#### 1.4 系统分区 ####

/dev/sda6 4.0G 380M 3.4G 10% /

tmpfs 495M 72K 495M 1% /dev/shm

/dev/sda2 7.9G 419M 7.1G 6% /app

/dev/sda3 7.9G 146M 7.4G 2% /applog

/dev/sda1 194M 30M 155M 16% /boot

/dev/sda5 5.8G 140M 5.3G 3% /data

/dev/sda8 2.0G 129M 1.8G 7% /home

/dev/sda9 2.0G 68M 1.9G 4% /opt

/dev/sda12 2.0G 36M 1.9G 2% /tmp

/dev/sda7 4.0G 3.3G 509M 87% /usr

/dev/sda10 2.0G 397M 1.5G 21% /var

#### 1.5 虚拟机配置 ####

4台虚拟机，配置为：1 CPU\*1GBMEM\*40GB硬盘 安装CentOS6.5 64位

图1.5-1

分别创建两个用户：root ,hadoop

root 为管理员用户

hadoop 为Hadoop运行用户

#### 1.6 SSH免密码登录 ####

一、创建在用户的home目录下创建  .ssh文件夹

mkdir .ssh

可以隐藏文件夹或文件内容

ls -a

二、 生成证书

证书分为：dsa和rsa

ssh-keygen -t rsa -P  '' -b 1024

ssh-keygen 生成命令

\-t 表示证书 rsa

\-p 密码提示语 ''

\-b 证书大小 为：1024

执行后 将会生成密钥文件和私钥文件

\-rwx------ 1 apch apache 883 May 20 15:13 id\_rsa  
\-rwx------ 1 apch apache 224 May 20 15:13 id\_rsa.pub

三、 把公钥信息写入 authorized\_keys 文档中

cat  id\_rsa.pub  >>  authorized\_keys

（将生成的公钥文件写入 authorized\_keys 文件）

四、设置文件和目录权限

设置authorized\_keys权限  
$ chmod 600 authorized\_keys   
设置.ssh目录权限  
$ chmod 700 -R .ssh

五 修改/etc/ssh/sshd\_config  (需要使用root用户登录)

vi  /etc/ssh/sshd\_config

Protocol 2 (仅使用SSH2)   
PermitRootLogin yes (允许root用户使用SSH登陆，根据登录账户设置)

ServerKeyBits 1024 (将serverkey的强度改为1024)

PasswordAuthentication no (不允许使用密码方式登陆)

PermitEmptyPasswords no   (禁止空密码进行登陆)

RSAAuthentication yes  （启用 RSA 认证）

PubkeyAuthentication yes （启用公钥认证）

AuthorizedKeysFile   .ssh/authorized\_keys

六、重启sshd 服务 (需要使用root用户登录)

service sshd restart

七、本地验证测试

ssh -v  localhost (开启登录调试模式)

#### 1.7 JDK安装 ####

从Oracle官网下－[jdk-6u43-linux-x64.bin][]

安装到：/usr/java目录

配置JAVA\_HOME:

export JAVA\_HOME=/usr/java/jdk1.6.0\_43

export JAVA\_BIN=/usr/java/jdk1.6.0\_43/bin

export PATH=$PATH:$JAVA\_HOME/bin

export CLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar

export JAVA\_HOME JAVA\_BIN PATH CLASSPATH

生效配置文件

source /etc/profile

验证安装：java -version

### 2 Hadoop1.2.1安装及配置 ###

#### 2.1 下载Hadoop安装介质 ####

下载地址：[http://mirrors.hust.edu.cn/apache/hadoop/common/][http_mirrors.hust.edu.cn_apache_hadoop_common]

并上传到Linux系统 rz（前置条件必须安装rz,sz包）

#### 2.2 解压Hadoop安装介质 ####

将hadoop安装到/app/hadoop/的目录下：

tar -zcvf /home/hadoop/hadoop-1.2.1.tar.gz

/app/hadoop/的权限为：hadoop:hadoop（用户、用户组）

#### 2.3 Hadoop配置文件 ####

需要配置修改配置文件如下：

ü hadoop-env.sh

ü core-site.xml

ü hdfs-site.xml

ü mapred-site.xml

ü masters

ü slaves

vi hadoop-env.sh

export JAVA\_HOME/usr/java/jdk1.6.0\_43

vi core-site.xml

vi hdfs-site.xml

添加以下内容：

<name>dfs.replication</name>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

vi mapred-site.xml

<name>mapred.job.tracker</name>

</property>

</configuration>

vi masters

192.168.1.53

vi slaves

192.168.1.54

192.168.1.55

192.168.1.56

#### 2.4 复制Hadoop安装介质 ####

经过2.3 节配置，将192.168.1.53服务上已配置完成hadoop文件，打包

tar -zcvf hadoop.tar.gz ./hadoop/

再将hadoop.tar.gz 复制到192.168.1.54-56

scp hadoop.tar.gz [ hadoop@192.168.1.54:/app/hadoop/][hadoop_192.168.1.54_app_hadoop]

在分别登录到192.168.1.54~56 上，

cd /app/hadoop/

tar –zxvf hadoop.tar.gz

#### 2.5 配置HADOOP\_HOME ####

192.68.1.53-56 配置HADOOP\_HOME

vi /etc/profile

\# set hadoop path

export HADOOP\_HOME=/app/hadoop/hadoop

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export PATH=$PATH:$HADOOP\_HOME/bin

source /etc/profile

生效配置文件

export HADOOP\_HOME\_WARN\_SUPPRESS=1

解决 启动Hadoop时报了一个警告信息

验证配置：

hadoop version

#### 2.6 格式化 namenode ####

**格式化HDFS文件系统**

hadoop namenode -format

#### 2.7 启动Hadoop ####

使用SecureCRT 7.0登录到192.168.1.53

命令：

cd /app/hadoop/hadoop/bin/

./start-all.sh

查看Hadoop的nameNode进程（192.168.1.53）

说明已启动NameNode,JobTracker,SecondaryNameNode

查看Hadoop的datanode进程：（192.168.1.54）

说明已启动：DataNode,TaskTracker

### 3 Hadoop1.2.1验证 ###

#### 3.1 Hadoop 控制台 ####

[http://192.168.1.53:50070/dfshealth.jsp][http_192.168.1.53_50070_dfshealth.jsp]

[http://192.168.1.53:50030/jobtracker.jsp][http_192.168.1.53_50030_jobtracker.jsp]

#### 3.2 Hadoop wordcount运行 ####

hadoop jar hadoop-examples-1.2.1.jar wordcount /tmp/input /tmp/output

通过jobtracker控制台监控结果如下：

### 4 Hadoop2.4环境准备 ###

#### 4.1 虚拟机准备 ####

从CentOS-53克隆CentOS-57-60

分别虚拟机名称、主机名、IP

CentOS-57 192.168.1.57 namenode57

CentOS-58 192.168.1.58 datanode58

CentOS-59 192.168.1.59 datanode59

CentOS-60 192.168.1.60 datanode60

(会使用VMware[虚拟机][Link 1]的的克隆功能，快速的复制已安装好的[系统][Link 2]。可是克隆完之后，会发现没有eth0网，切换到root用户下，才能修改，否则权限不足)

**配置网卡：**

vi /etc/sysconfig/network-scripts/ifcfg-eth0

修改网络卡配置－MAC地址

vi /etc/sysconfig/network-scripts/ifcfg-eth0

修改主机名：

vi /etc/sysconfig/network

重启系统，reboot

#### 4.2 网络拓扑结构 ####

参见1.3

192.168.1.57-60 修改hosts

添加以下内容：

192.168.1.57 namenode57

192.168.1.58 datanode58

192.168.1.59 datanode59

192.168.1.60 datanode60

su –

切换到root用户

vi /etc/hosts

192.168.1.57 namenode57

192.168.1.58 datanode58

192.168.1.59 datanode59

192.168.1.60 datanode60

#### 4.3 系统分区 ####

参见1.4

#### 4.4 虚拟机配置 ####

参见1.5

#### 4.5 SSH免密码登录配置 ####

参见1.6

#### 4.6 JDK安装 ####

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

jdk-7u55-linux-x64.rpm

**安装JDK（192.168.1.57-60）**

将JDK7安装介质上传到192.168.1.57上“jdk-7u55-linux-x64.rpm”

rpm -i jdk-7u55-linux-x64.rpm

安装成功

cd /usr/java

检查是否安装成功

安装目录：/usr/java/jdk1.7.0\_55

配置JAVA\_HOME

vi /etc/profile

添加以下内容

export JAVA\_HOME=/usr/java/jdk1.7.0\_55

export JAVA\_BIN=/usr/java/jdk1.7.0\_55/bin

export PATH=$PATH:$JAVA\_HOME/bin

export CLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar

export JAVA\_HOME JAVA\_BIN PATH CLASSPATH

生效配置文件

source /etc/profile

检查安装版本：

java -version

表示安装成功

### 5 Hadoop2.4安装及配置 ###

#### 5.1 下载Hadoop安装介质 ####

[http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/][http_mirrors.cnnic.cn_apache_hadoop_common_hadoop-2.4.0]

并上传到Linux系统 rz命令上传（前置条件必须安装rz,sz包）

登录到192.168.1.57

#### 5.2 解压Hadoop安装介质 ####

安装目录为：/app/hadoop/hadoop2.4

分别在192.168.1.57－60服务器，以hadoop用户登录，并创建目录：hadoop2.4

cd /app/hadoop/

tar -zxvf /home/hadoop/hadoop-2.4.0.tar.gz

注：Hadoop-2.4.0 变化太大了，配置目录都变了

cd /app/hadoop/hadoop-2.4.0/lib/native

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 32-bit LSB shared object,

环境变量配置：

su - (切换到root下)

vi /etc/profile

添加以下内容：

export HADOOP\_HOME=/app/hadoop/hadoop-2.4.0

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export PATH=$PATH:$HADOOP\_HOME/bin

export HADOOP\_COMMON\_LIB\_NATIVE\_DIR=$\{HADOOP\_HOME\}/lib/native

export HADOOP\_OPTS="-Djava.library.path=$HADOOP\_HOME/lib"

生效配置文件

source /etc/profile

#### 5.3 编译Hadoop本地库 ####

##### 5.3.1 编译环境准备 #####

基于CentOS6.5 64位操作编译

主要涉及到工具有：hadoop-2.4.0-src.tar.gz、Ant、Maven、JDK、GCC、CMake、openssl

第一步升级系统相关编译所需的软件（升级最新版）：

yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel

wget [ http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz][http_mirrors.cnnic.cn_apache_hadoop_common_hadoop-2.4.0_hadoop-2.4.0-src.tar.gz] （源代版）

tar -zxvf hadoop-2.4.0-src.tar.gz

wget [ http://apache.fayea.com/apache-mirror//ant/binaries/apache-ant-1.9.4-bin.tar.gz][http_apache.fayea.com_apache-mirror_ant_binaries_apache-ant-1.9.4-bin.tar.gz]

tar -xvf apache-ant-1.9.4-bin.tar.gz

wget [ http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz][http_apache.fayea.com_apache-mirror_maven_maven-3_3.0.5_binaries_apache-maven-3.0.5-bin.tar.gz]

tar -xvf apache-maven-3.0.5-bin.tar.gz

vi /etc/profile

export JAVA\_HOME=/usr/java/jdk1.7.0\_55

export JAVA\_BIN=/usr/java/jdk1.7.0\_55/bin

export ANT\_HOME=/home/hadoop/ant

export MVN\_HOME=/home/hadoop/maven

export FINDBUGS\_HOME=/home/hadoop/findbugs-2.0.3

export PATH=$PATH:$JAVA\_HOME/bin:$ANT\_HOME/bin:$MVN\_HOME/bin:$FINDBUGS\_HOME/bin

生产配置文件：

source /etc/profile

验证是否配置成功

ant –version

mvn -version

findbugs –version

验证结果：

**安装protobuf（以root用户登录）**

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar zxf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make

make install

protoc --version

**安装cmake（以root用户登录）**

wget http://www.cmake.org/files/v2.8/cmake-2.8.12.2-Linux-i386.tar.gz

./bootstrap

make

make install

cmake –version

为了加速编译，将maven镜像库指向：开源中国

cd maven/conf

vi settings.xml

添加以下内容：

<id>nexus\-osc</id>

<name>Nexus osc</name>

<url>http://maven.oschina.net/content/groups/public/</url>

</mirror>

<id>nexus\-osc\-thirdparty</id>

<mirrorOf>thirdparty</mirrorOf>

<name>Nexus osc thirdparty</name>

<url>http://maven.oschina.net/content/repositories/thirdparty/</url>

</mirror>

</activation>

<id>nexus</id>

<name>local private nexus</name>

<url>http://maven.oschina.net/content/groups/public/</url>

</releases>

<enabled>false</enabled>

</snapshots>

</repository>

</repositories>

<id>nexus</id>

<name>local private nexus</name>

<url>http://maven.oschina.net/content/groups/public/</url>

</releases>

<enabled>false</enabled>

</snapshots>

</pluginRepository>

</pluginRepositories>

</profile>

详细说明可参见：

[http://maven.oschina.net/help.html][http_maven.oschina.net_help.html]

##### 5.3.2 编译Hadoop #####

mvn package -DskipTests -Pdist,native –Dtar

此时在下载maven依赖所有包及插件

慢慢等待中……（花6个小时，终于看到一编译错误）

编译成功，检查nativelib 是否编译成功

cd hadoop-dist/target/hadoop-2.4.0/lib/native

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

代表编译成功

**错误1**

\[INFO\] ------------------------------------------------------------------------

\[INFO\] BUILD FAILURE

\[INFO\] ------------------------------------------------------------------------

\[INFO\] Total time: 46.796s

\[INFO\] Finished at: Wed Jun 04 13:28:37 CST 2014

\[INFO\] Final Memory: 36M/88M

\[INFO\] ------------------------------------------------------------------------

\[ERROR\] Failed to execute goal on project hadoop-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:2.4.0: Failure to find org.apache.commons:commons-compress:jar:1.4.1 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> \[Help 1\]

\[ERROR\]

\[ERROR\] To see the full stack trace of the errors, re-run Maven with the -e switch.

\[ERROR\] Re-run Maven using the -X switch to enable full debug logging.

\[ERROR\]

\[ERROR\] For more information about the errors and possible solutions, please read the following articles:

\[ERROR\] \[Help 1\] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

\[ERROR\]

\[ERROR\] After correcting the problems, you can resume the build with the command

\[ERROR\] mvn <goals> -rf :hadoop-common

解决方法：

根据上面日志提示说找不到“org.apache.commons:commons-compress:jar:1.4.1”,

直接将本地（Windows）包复制到Linux系统中，解决了。

**错误2**

\[INFO\] ------------------------------------------------------------------------

\[INFO\] BUILD FAILURE

\[INFO\] ------------------------------------------------------------------------

\[INFO\] Total time: 2:16.693s

\[INFO\] Finished at: Wed Jun 04 13:56:31 CST 2014

\[INFO\] Final Memory: 48M/239M

\[INFO\] ------------------------------------------------------------------------

\[ERROR\] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, 没有那个文件或目录

\[ERROR\] around Ant part ...<exec dir="/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failοnerrοr="true">... @ 4:133 in /home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

\[ERROR\] -> \[Help 1\]

\[ERROR\]

\[ERROR\] To see the full stack trace of the errors, re-run Maven with the -e switch.

\[ERROR\] Re-run Maven using the -X switch to enable full debug logging.

\[ERROR\]

\[ERROR\] For more information about the errors and possible solutions, please read the following articles:

\[ERROR\] \[Help 1\] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

\[ERROR\]

\[ERROR\] After correcting the problems, you can resume the build with the command

\[ERROR\] mvn <goals> -rf :hadoop-common

解决方法：

是没有安装cmake导致的，再重新安装cmake；参考《5.3.1编译环境准备》

错误3

错误提示是找不到相应的文件和不能创建目录，在网上没有相关错误（根据自己经验修改目录权限为：775，让目录有创建文件或文件夹的权限，另外最好保证hadoop编译目录有2.5G至4G的空间）

chmod -Rf 775 ./ hadoop-2.4.0-src

main:

\[mkdir\] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/test-dir

\[INFO\] Executed tasks

\[INFO\]

\[INFO\] --- maven-antrun-plugin:1.7:run (make) @ hadoop-pipes ---

\[INFO\] Executing tasks

错误3

main:

\[mkdir\] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native

\[exec\] -- The C compiler identification is GNU 4.4.7

\[exec\] -- The CXX compiler identification is GNU 4.4.7

\[exec\] -- Check for working C compiler: /usr/bin/cc

\[exec\] -- Check for working C compiler: /usr/bin/cc -- works

\[exec\] -- Detecting C compiler ABI info

\[exec\] -- Detecting C compiler ABI info - done

\[exec\] -- Check for working CXX compiler: /usr/bin/c++

\[exec\] -- Check for working CXX compiler: /usr/bin/c++ -- works

\[exec\] -- Detecting CXX compiler ABI info

\[exec\] -- Detecting CXX compiler ABI info - done

\[exec\] CMake Error at /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):

\[exec\] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the

\[exec\] system variable OPENSSL\_ROOT\_DIR (missing: OPENSSL\_LIBRARIES

\[exec\] OPENSSL\_INCLUDE\_DIR)

\[exec\] Call Stack (most recent call first):

\[exec\] /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (\_FPHSA\_FAILURE\_MESSAGE)

\[exec\] /usr/local/share/cmake-2.8/Modules/FindOpenSSL.cmake:313 (find\_package\_handle\_standard\_args)

\[exec\] CMakeLists.txt:20 (find\_package)

\[exec\]

\[exec\] -- Configuring incomplete, errors occurred!

\[exec\] See also "/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".

\[INFO\] ------------------------------------------------------------------------

\[INFO\] Reactor Summary:

\[INFO\]

\[INFO\] Apache Hadoop Main ................................ SUCCESS \[13.745s\]

\[INFO\] Apache Hadoop Project POM ......................... SUCCESS \[5.538s\]

\[INFO\] Apache Hadoop Annotations ......................... SUCCESS \[7.296s\]

\[INFO\] Apache Hadoop Assemblies .......................... SUCCESS \[0.568s\]

\[INFO\] Apache Hadoop Project Dist POM .................... SUCCESS \[5.858s\]

\[INFO\] Apache Hadoop Maven Plugins ....................... SUCCESS \[8.541s\]

\[INFO\] Apache Hadoop MiniKDC ............................. SUCCESS \[8.337s\]

\[INFO\] Apache Hadoop Auth ................................ SUCCESS \[7.348s\]

\[INFO\] Apache Hadoop Auth Examples ....................... SUCCESS \[4.926s\]

\[INFO\] Apache Hadoop Common .............................. SUCCESS \[2:35.956s\]

\[INFO\] Apache Hadoop NFS ................................. SUCCESS \[18.680s\]

\[INFO\] Apache Hadoop Common Project ...................... SUCCESS \[0.059s\]

\[INFO\] Apache Hadoop HDFS ................................ SUCCESS \[5:03.525s\]

\[INFO\] Apache Hadoop HttpFS .............................. SUCCESS \[38.335s\]

\[INFO\] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS \[23.780s\]

\[INFO\] Apache Hadoop HDFS-NFS ............................ SUCCESS \[8.769s\]

\[INFO\] Apache Hadoop HDFS Project ........................ SUCCESS \[0.159s\]

\[INFO\] hadoop-yarn ....................................... SUCCESS \[0.134s\]

\[INFO\] hadoop-yarn-api ................................... SUCCESS \[2:07.657s\]

\[INFO\] hadoop-yarn-common ................................ SUCCESS \[1:10.680s\]

\[INFO\] hadoop-yarn-server ................................ SUCCESS \[0.165s\]

\[INFO\] hadoop-yarn-server-common ......................... SUCCESS \[24.174s\]

\[INFO\] hadoop-yarn-server-nodemanager .................... SUCCESS \[27.293s\]

\[INFO\] hadoop-yarn-server-web-proxy ...................... SUCCESS \[5.177s\]

\[INFO\] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS \[11.399s\]

\[INFO\] hadoop-yarn-server-resourcemanager ................ SUCCESS \[28.384s\]

\[INFO\] hadoop-yarn-server-tests .......................... SUCCESS \[1.346s\]

\[INFO\] hadoop-yarn-client ................................ SUCCESS \[12.937s\]

\[INFO\] hadoop-yarn-applications .......................... SUCCESS \[0.108s\]

\[INFO\] hadoop-yarn-applications-distributedshell ......... SUCCESS \[5.303s\]

\[INFO\] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS \[3.212s\]

\[INFO\] hadoop-yarn-site .................................. SUCCESS \[0.050s\]

\[INFO\] hadoop-yarn-project ............................... SUCCESS \[8.638s\]

\[INFO\] hadoop-mapreduce-client ........................... SUCCESS \[0.135s\]

\[INFO\] hadoop-mapreduce-client-core ...................... SUCCESS \[43.622s\]

\[INFO\] hadoop-mapreduce-client-common .................... SUCCESS \[36.329s\]

\[INFO\] hadoop-mapreduce-client-shuffle ................... SUCCESS \[6.058s\]

\[INFO\] hadoop-mapreduce-client-app ....................... SUCCESS \[20.058s\]

\[INFO\] hadoop-mapreduce-client-hs ........................ SUCCESS \[16.493s\]

\[INFO\] hadoop-mapreduce-client-jobclient ................. SUCCESS \[11.685s\]

\[INFO\] hadoop-mapreduce-client-hs-plugins ................ SUCCESS \[3.222s\]

\[INFO\] Apache Hadoop MapReduce Examples .................. SUCCESS \[12.656s\]

\[INFO\] hadoop-mapreduce .................................. SUCCESS \[8.060s\]

\[INFO\] Apache Hadoop MapReduce Streaming ................. SUCCESS \[8.994s\]

\[INFO\] Apache Hadoop Distributed Copy .................... SUCCESS \[15.886s\]

\[INFO\] Apache Hadoop Archives ............................ SUCCESS \[6.659s\]

\[INFO\] Apache Hadoop Rumen ............................... SUCCESS \[15.722s\]

\[INFO\] Apache Hadoop Gridmix ............................. SUCCESS \[11.778s\]

\[INFO\] Apache Hadoop Data Join ........................... SUCCESS \[5.953s\]

\[INFO\] Apache Hadoop Extras .............................. SUCCESS \[6.414s\]

\[INFO\] Apache Hadoop Pipes ............................... FAILURE \[3.746s\]

\[INFO\] Apache Hadoop OpenStack support ................... SKIPPED

\[INFO\] Apache Hadoop Client .............................. SKIPPED

\[INFO\] Apache Hadoop Mini-Cluster ........................ SKIPPED

\[INFO\] Apache Hadoop Scheduler Load Simulator ............ SKIPPED

\[INFO\] Apache Hadoop Tools Dist .......................... SKIPPED

\[INFO\] Apache Hadoop Tools ............................... SKIPPED

\[INFO\] Apache Hadoop Distribution ........................ SKIPPED

\[INFO\] ------------------------------------------------------------------------

\[INFO\] BUILD FAILURE

\[INFO\] ------------------------------------------------------------------------

\[INFO\] Total time: 19:43.155s

\[INFO\] Finished at: Wed Jun 04 17:40:17 CST 2014

\[INFO\] Final Memory: 79M/239M

\[INFO\] ------------------------------------------------------------------------

\[ERROR\] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

\[ERROR\] around Ant part ...<exec dir="/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failοnerrοr="true">... @ 5:123 in /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

\[ERROR\] -> \[Help 1\]

\[ERROR\]

\[ERROR\] To see the full stack trace of the errors, re-run Maven with the -e switch.

\[ERROR\] Re-run Maven using the -X switch to enable full debug logging.

\[ERROR\]

\[ERROR\] For more information about the errors and possible solutions, please read the following articles:

\[ERROR\] \[Help 1\] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

\[ERROR\]

\[ERROR\] After correcting the problems, you can resume the build with the command

根据网上提示（ 下面需要再安装openssl-devel,安装命令yum install openssl-devel,此步不做的话会报如下错误

\[exec\] CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:66 (MESSAGE):  
\[exec\]   Could NOT find OpenSSL  
\[exec\] Call Stack (most recent call first):  
\[exec\]   CMakeLists.txt:20 (find\_package)  
\[exec\]  
\[exec\]  
\[exec\] -- Configuring incomplete, errors occurred!  
\[ERROR\] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -> \[Help 1\]  
\[ERROR\]  
\[ERROR\] To see the full stack trace of the errors, re-run Maven with the -e switch.  
\[ERROR\] Re-run Maven using the -X switch to enable full debug logging.  
\[ERROR\]  
\[ERROR\] For more information about the errors and possible solutions, please read the following articles:  
\[ERROR\] \[Help 1\] [ http://cwiki.apache.org/confluen ... oExecutionException][http_cwiki.apache.org_confluen ... oExecutionException]  
\[ERROR\]  
\[ERROR\] After correcting the problems, you can resume the build with the command  
\[ERROR\]   mvn <goals> -rf :hadoop-pipes

）

错误连接：[http://f.dataguru.cn/thread-189176-1-1.html][http_f.dataguru.cn_thread-189176-1-1.html]

原因是：在安装openssl-devel,少写一个l,重新安装一下

解决方法：重新安装openssl-devel

yum install openssl-devel

##### 5.3.3 编译总结 #####

1、 必须安装（yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel）

2、 必须安装（**protobuf，CMake**）编译工具

3、 必须配置（ANT、MAVEN、FindBugs）

4、 将maven库指向开源中国，这样就可以加快编译速度，即加快下载依赖jar包速度

5、 编译出错需求详细观察出错日志，根据错误日志分析原因再结束百度和Google解决错误；

#### 5.4 Hadoop 配置文件 ####

cd hadoop-2.4.0

cd etc/hadoop

core-site.xml

yarn-site.xml

hdfs-site.xml

mapred-site.xml

hadoop-env.sh

**vi core-site.xml**

<name>fs.defaultFS</name>

<value>hdfs://namenode57:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/app/hadoop/current/tmp</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/app/hadoop/current/data</value>

</property>

</configuration>

**vi yarn-site.xml**

<name>yarn.resourcemanager.address</name>

<value>namenode57:18040</value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>namenode57:18030</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>namenode57:18088</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>namenode57:18025</value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>namenode57:18141</value>

</property>

<name>mapreduce.framework.name</name>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce\_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

**vi hdfs-site.xml**

<name>dfs.namenode.rpc-address</name>

<value>namenode57:9001</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>/app/hadoop/current/dfs</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/app/hadoop/current/data</value>

</property>

</configuration>

**vi mapred-site.xml**

<name>mapreduce.framework.name</name>

</property>

</configuration>

vi hadoop-env.sh

export JAVA\_HOME=/usr/java/jdk1.7.0\_55

export HADOOP\_COMMON\_LIB\_NATIVE\_DIR=$\{HADOOP\_HOME\}/lib/native

export HADOOP\_OPTS="-Djava.library.path=$HADOOP\_HOME/lib"

#### 5.5 复制Hadoop安装介质 ####

scp hadoop-2.4.0.tar.gz [ hadoop@192.168.1.58:/app/hadoop/][hadoop_192.168.1.58_app_hadoop]

分别复制到：192.168.1.58-60

#### 5.6 格式化NameNode ####

./hdfs namenode -format

#### 5.7 启动Hadoop ####

cd /app/hadoop/hadoop-2.4.0/

./start-all.sh

查看进程hadoop进程：

jps

ps –ef|grep java

14/06/04 07:48:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on

启动时一直报错，准备编译Hadoop64位依赖包（5.3编译Hadoop本地库），再进行替换

替换掉32位的native库

删除原32位的native库

cd /app/hadoop/hadoop-2.4.0/lib

rm -rf native/

将5.3节编译好native 64位的库复制到：/app/hadoop/hadoop-2.4.0/lib

cd /data/hadoop/hadoop-2.4.0-src/hadoop-dist/target/hadoop-2.4.0/lib

cp -r ./native /app/hadoop/hadoop-2.4.0/lib/

错误1：

2014-06-04 18:30:57,450 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager

java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9\_ and can not start with numbers

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:186)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)

2014-06-04 18:30:57,458 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN\_MSG:

解决方法：

vi /app/hadoop/hadoop-2.4.0/etc/hadoop/yarn-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce.shuffle</value>

</property>

改为

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce\_shuffle</value>

</property>

总结：

从2.0以后启动和停止hadoop的命令start-all.sh和stop-all.sh不建议使用，开始摒弃掉取而代之的将使用start-dfs.sh和start-yarn.sh启动hadoop,详细请看官方说明。

#### 5.8 检查Hadoop启动情况 ####

##### 5.8.1 检查namenode是否启动 #####

在192.168.1.57检查

jps

##### 5.8.2 编译Hadoop #####

分别检查192.168.1.58~60

Jps

### 6 Hadoop2.4验证 ###

#### 6.1 Hadoop 控制台 ####

HDFS

[http://192.168.1.57:50070/dfshealth.html\#tab-overview][http_192.168.1.57_50070_dfshealth.html_tab-overview]

[http://namenode57:8088/cluster/][http_namenode57_8088_cluster]

#### 6.2 Hadoop wordcount运行 ####

##### 6.2.1 创建文件夹及将本地文件复制到:hdfs系统中 #####

1、创建/tmp/input文件夹

hadoop fs -mkdir /tmp

hadoop fs -mkdir /tmp/input

2、将本地文件复制到hdfs系统中

hadoop fs -put /usr/hadoop/file\* /tmp/input

3、查看test.txt文件是否成功上传到hdfs上

hadoop fs -ls /tmp/input

##### 6.2.1 运行wordcount程序 #####

hadoop jar hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/input /tmp/output

### 7 Hadoop安装错误 ###

#### 7.1 [ hadoop2.4无法连接ResourceManager问题][hadoop2.4_ResourceManager] ####

错误日志：

ountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:07,154 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:08,156 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:09,159 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:10,161 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:11,164 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:12,166 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:13,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

2014-06-05 07:01:14,171 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

解决方法，在yarn.xml文件里配置，所有机器都修改

<name>yarn.resourcemanager.hostname</name>

<value>namenode57</value>

</property>

[jdk-6u43-linux-x64.bin]: http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html
[http_mirrors.hust.edu.cn_apache_hadoop_common]: http://mirrors.hust.edu.cn/apache/hadoop/common/
[hadoop_192.168.1.54_app_hadoop]: mailto:hadoop@192.168.1.54:/app/hadoop/
[http_192.168.1.53_50070_dfshealth.jsp]: http://192.168.1.53:50070/dfshealth.jsp
[http_192.168.1.53_50030_jobtracker.jsp]: http://192.168.1.53:50030/jobtracker.jsp
[Link 1]: http://www.2cto.com/os/xuniji/
[Link 2]: http://www.2cto.com/os/
[http_mirrors.cnnic.cn_apache_hadoop_common_hadoop-2.4.0]: http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/
[http_mirrors.cnnic.cn_apache_hadoop_common_hadoop-2.4.0_hadoop-2.4.0-src.tar.gz]: http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz
[http_apache.fayea.com_apache-mirror_ant_binaries_apache-ant-1.9.4-bin.tar.gz]: http://apache.fayea.com/apache-mirror/ant/binaries/apache-ant-1.9.4-bin.tar.gz
[http_apache.fayea.com_apache-mirror_maven_maven-3_3.0.5_binaries_apache-maven-3.0.5-bin.tar.gz]: http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
[http_maven.oschina.net_help.html]: http://maven.oschina.net/help.html
[http_cwiki.apache.org_confluen ... oExecutionException]: http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[http_f.dataguru.cn_thread-189176-1-1.html]: http://f.dataguru.cn/thread-189176-1-1.html
[hadoop_192.168.1.58_app_hadoop]: mailto:hadoop@192.168.1.58:/app/hadoop/
[http_192.168.1.57_50070_dfshealth.html_tab-overview]: http://192.168.1.57:50070/dfshealth.html#tab-overview
[http_namenode57_8088_cluster]: http://namenode57:8088/cluster/
[hadoop2.4_ResourceManager]: http://hi.baidu.com/scalong/item/e83207e05b02f2d1ea34c90a