Hadoop Hive Sqoop安装教程

ゞ浴缸里的玫瑰 2023-02-22 08:23 31阅读 0赞

### 文章目录 ###

*  前提条件
 *  1 关闭防火墙&清空规则
 *  安装包说明
 *  3 JDK安装
 *   *  3.1 指定安装目录为/usr/java
     *  3.2配置环境变量
 *  4 安装
 *   *   *  配置个人环境变量
 *  5 部署
 *   *  5.1 添加本地信任
     *  5.2 配置文件
     *   *  5.2.1 core-site
         *  5.2.2 SNN
         *  5.2.3 DN
     *  5.4 配置文件部署
     *   *  5.4.1
         *  5.3.2 mapred
         *  5.3.4 yarn
     *  5.5 启动
     *   *   *  hdfs
             *  yarn
         *  测试pi
 *  hdfs启动bug

# 前提条件 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 2]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 3]

# 1 关闭防火墙&清空规则 #

\#停止  
systemctl stop firewalld

\#禁用  
systemctl disable firewalld

\#清空防火墙的规则  
iptables -F

# 安装包说明 #

下载地址

http://archive.cloudera.com/cdh5/cdh/5/

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 4]

# 3 JDK安装 #

## 3.1 指定安装目录为/usr/java ##

[root@localhost SOFT]# mkdir /usr/java/

tar开

[root@localhost SOFT]# tar -xzvf jdk-8u181-linux-x64.tar.gz -C /usr/java/

## 3.2配置环境变量 ##

*  JAVA\_HOME

echo "export JAVA_HOME=/usr/java/jdk1.8.0_181" >> /etc/profile

*  PATH

echo "export PATH=/usr/java/jdk1.8.0_181/bin:${PATH}" >> /etc/profile

*  source 一下

source /etc/profile

**验证**

java -version

![在这里插入图片描述][20200702175243646.png]

# 4 安装 #

**创建用户**

[root@localhost java]# ll /home/ifengs
    total 0
    [root@localhost java]#

**进入ifengs 并创建文件夹**

[ifengs@localhost ~]$ mkdir app software sourcecode log data tmp lib
    [ifengs@localhost ~]$ ll
    total 0
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 app
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 data
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 lib
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 log
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 software
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 sourcecode
    drwxrwxr-x. 2 ifengs ifengs 6 Jul  2 18:26 tmp
    [ifengs@localhost ~]$

*  app 软件
 *  data 额外的数据
 *  lib 额外的第三方的jdbc…jar
 *  log 日志
 *  sourcecode 源代码
 *  tmp 临时文件夹

**移动安装包到software下面**

mv hadoop... /home/ifengs/software/

**修改安装包所有者**

chown -R ifeng:ifeng /home/ifeng/software/*

**解压安装包**

tar -xzvf hadoop-2.6 -C ~/app/
    
    ...同理

**创建软连接**

ln -s hadoop-2.6.0...  hadoop
    ...

### 配置个人环境变量 ###

配置 ,bashrc

export HADOOP_HOME=/home/zull1/app/hadoop
    export Hive_HOME=/home/zull1/app/hive
    
    export PATH=${HADOOP_HOME}/bin:${Hive_HOME}/bin:$PATH

source .bashrc

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 5]

# 5 部署 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 6]

## 5.1 添加本地信任 ##

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
      $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
      $ chmod 0600 ~/.ssh/authorized_keys

**测试**  
首先在/etc/hosts 中配置内网ip

ssh zuoll1

无需密码直接登录即可

## 5.2 配置文件 ##

### 5.2.1 core-site ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 7]

<configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://zuoll1:9000</value>
        </property>
    </configuration>

### 5.2.2 SNN ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 9]

<configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

<configuration>
        <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>zuoll1:9868</value>
        </property>
    </configuration>

<configuration>
        <property>
            <name>dfs.namenode.secondary.https-address</name>
            <value>zuoll1:9869</value>
        </property>
    </configuration>

### 5.2.3 DN ###

<configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://zuoll1:9000</value>
        </property>
    </configuration>

## 5.4 配置文件部署 ##

cd ~/app/hadoop/etc/hadoop

### 5.4.1 ###

**slave的配置**

vi slaves
    
    # 清空后输入
    zuoll1

**core-site**

vi core-site.xml

<property>
            <name>fs.defaultFS</name>
            <value>hdfs://ifengs:9000</value>
        </property>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 10]  
修改 hadoop.tmp.dir

vi core-site.xml

<property>
            <name>hadoop.tmp.dir</name>
            <value>/home/zuoll1/tmp</value>
        </property>

默认是在系统的tmp下 但是系统tmp 会半个月清理一次  
**hdfs**

<property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>ifengs:9868</value>
        </property>
        <property>
            <name>dfs.namenode.secondary.https-address</name>
            <value>ifengs:9869</value>
        </property>

snn

### 5.3.2 mapred ###

/etc/hadoop/mapred-site.xml

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 11]

<configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>

### 5.3.4 yarn ###

<configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
        <property>

8088容易被挖矿ST

<property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>zuoll1:18088</value>
        </property>

## 5.5 启动 ##

#### hdfs ####

格式化hdfs

hdfs namenode -format

![在这里插入图片描述][20200703214711776.PNG]  
**启动hdfs**

start-dfs.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 12]  
JAVA\_HOME已经配置 启动还是报错

apache hadoop 重大bug！！！

去改下官方的配置文件

vi ~/app/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk!.8....

**再次启动dfs**

start-dfs.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 13]  
**hdfs查看文件**

hdfs dfs -ls /

**hdfs创建文件**

hdfs dfs -mkdir /zuoll1

#### yarn ####

start-yarn.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 14]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 15]

### 测试pi ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 16]

[ifeng@localhost hadoop]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar pi 10 100

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 17]

# hdfs启动bug #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 18]  
[1>&2][1_2]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200702231958910.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020070223200772.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200703113454679.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200703190207356.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200702171722674.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[20200702175243646.png]: https://img-blog.csdnimg.cn/20200702175243646.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200703192336837.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200703193036889.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/2020070319585752.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200703200309294.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/2020070320070130.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200703210728267.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200703211825739.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[20200703214711776.PNG]: https://img-blog.csdnimg.cn/20200703214711776.PNG
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200703215303249.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200703232732379.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200705105453668.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200705105534503.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200705105838125.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200705110129643.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/2020070322245342.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70
[1_2]: https://blog.csdn.net/ithomer/article/details/9288353