Linux下hadoop的安装(CDH版本)

Bertha 。 2023-06-14 15:53 19阅读 0赞

### 1. 下载 ###

下载地址：[http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz][http_archive.cloudera.com_cdh5_cdh_5_hadoop-2.6.0-cdh5.7.0.tar.gz]

**注意：在安装hadoop之前确保[JDK][]安装成功。**

### 2. 配置免密登录 ###

Hadoop 组件之间需要基于 SSH 进行通讯。

#### 2.1 配置映射 ####

配置 ip 地址和主机名映射：

vim /etc/hosts
    192.168.43.174 hadoop000
    192.168.43.174 localhost

#### 2.2 生成公私钥 ####

执行下面命令行生成公匙和私匙

ssh-keygen -t rsa

#### 2.3 授权 ####

进入 `~/.ssh` 目录下，查看生成的公匙和私匙，并将公匙写入到授权文件：

[hadoop@hadoop000 ~]$ cd .ssh/
    [hadoop@hadoop000 .ssh]$ ll
    total 16
    -rw-r--r-- 1 hadoop hadoop  398 Sep 15  2017 authorized_keys
    -rw------- 1 hadoop hadoop 1675 Sep 15  2017 id_rsa
    -rw-r--r-- 1 hadoop hadoop  398 Sep 15  2017 id_rsa.pub
    -rw-r--r-- 1 hadoop hadoop 1583 Nov 14 11:28 known_host

写入公匙到授权文件

[hadoop@hadoop000 .ssh]cat id_rsa.pub >> authorized_keys
    [hadoop@hadoop000 .ssh] chmod 600 authorized_keys

检测免密登录：

[hadoop@hadoop000 ~]$ ssh hadoop000
    Last login: Wed Nov 20 11:30:01 2019 from hadoop000

### 3 下面开始环境搭建 ###

#### 3.1解压 ####

tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C ~/app/

#### 3.2 配置环境变量 ####

vi ~/.bash_profile
    export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
    export PATH=$HADOOP_HOME/bin:$PATH

刷新配置

source ~/.bash_profile

#### 3.3 配置文件修改 ####

cd /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/

修改配置文件 `vi hadoop-env.sh`

# JDK安装路径
    export JAVA_HOME=/home/hadoop/app/jdk1.8.0_144

修改配置文件 `vi core-site.xml`

<configuration>
    <property>
    
             <name>fs.defaultFS</name>
             <value>hdfs://hadoop000:8020</value>
     </property>
    
      <property>
       
               <name>hadoop.tmp.dir</name>
                 <value>/home/hadoop/app/tmp</value>
      </property>
    </configuration>

修改配置文件 `vi hdfs-site.xml`

<configuration>
        <property>
            
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

修改配置文件 `vi slaves`

hadoop000

#### 4 第一次启动 Hadoop 时需要进行初始化，进入 `${HADOOP_HOME}/bin/` 目录下，执行以下命令： ####

./hdfs namenode -format

#### 进入 `${HADOOP_HOME}/sbin/` 目录下，启动 HDFS： ####

./start-dfs.sh

### 5 验证是否启动成功 ###

方式一：执行 `jps` 查看 `NameNode` 和 `DataNode` 服务是否已经启动：

[hadoop@hadoop000 sbin]$ jps
    4262 SecondaryNameNode
    4057 DataNode
    4382 Jps
    3950 NameNode

方式二：查看 `Web UI` 界面，端口为 `50070`：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4_size_16_color_FFFFFF_t_70]

### 6 Hadoop(YARN)环境搭建 ###

#### 6.1 环境配置 ####

1.  `mapred-site.xml`

cd /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
    #如果没有mapred-site.xml，则拷贝一份样例文件后再修改
    cp mapred-site.xml.template mapred-site.xml

<configuration>
    <property>
    
        <name>mapreduce.framework.name</name>
            <value>yarn</value>
            </property>
    </configuration>

1.  `yarn-site.xml`

<property>
        <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
            </property>
    </configuration>

#### 6.2启动 ####

进入 `${HADOOP_HOME}/sbin/` 目录下，启动 YARN：

[hadoop@hadoop000 sbin]$ ./start-yarn.sh 
    starting yarn daemons
    starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop000.out
    hadoop000: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop000.out
    
    执行 `jps` 命令查看 `NodeManager` 和 `ResourceManager` 服务是否已经启动：
    [hadoop@hadoop000 sbin]$ jps
    5107 NodeManager
    4262 SecondaryNameNode
    5001 ResourceManager
    4057 DataNode
    5261 Jps
    3950 NameNode

或查看WEB UI界面，端口号为 `8088`：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4_size_16_color_FFFFFF_t_70 1]

[http_archive.cloudera.com_cdh5_cdh_5_hadoop-2.6.0-cdh5.7.0.tar.gz]: http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
[JDK]: https://blog.csdn.net/qq_35885488/article/details/103157651
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191120113435184.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191120123123944.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg1NDg4,size_16,color_FFFFFF,t_70