Hive笔记2-安装部署

喜欢ヅ旅行 2022-03-02 12:38 223阅读 0赞

## **二、安装部署** ##

环境：macOS High Sierra10.13.6、mysql8.0.15、Hadoop3.1.1、Hive3.1.1。

参考资料：

*  [学习 Hadoop3.0 一、Hadoop3.0的安装与配置][Hadoop3.0 _Hadoop3.0]
 *  [hadoop安装和配置][hadoop]
 *  [hadoop详细安装及配置][hadoop 1]
 *  [Hadoop: Setting up a Single Node Cluster][Hadoop_ Setting up a Single Node Cluster]
 *  [安装hadoop3.0版本踩坑][hadoop3.0]
 *  [Hadoop \_ 疑难杂症 解决1 - WARN util.NativeCodeLoader: Unable to load native-hadoop library for your plat][Hadoop _ _ _1 - WARN util.NativeCodeLoader_ Unable to load native-hadoop library for your plat]
 *  [Unable to load native-hadoop library for your platform(已解决)][Unable to load native-hadoop library for your platform]
 *  [Hive安装与配置详解][Hive]

### 1、安装MySQL ###

1.  在[mysql官网][mysql]下载安装包。我下载的是mysql-8.0.15-macos10.14-x86\_64.dmg。双击文件就进入了安装流程，按照提示操作就行了。
2.  安装完成后，在终端输入`mysql -uroot -ppwd`(pwd是root用户的密码)，能进入mysql的命令行就说明安装成功。

### 2、安装配置Hadoop ###

#### 1. 安装hadoop ####

可以在[Hadoop官网][Hadoop]下载压缩包，解压安装。也可以通过brew进行安装，安装完成后文件在/usr/local/Cellar下。

`brew install hadoop`

#### 2. 修改配置文件 ####

进入目录/usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop，修改hadoop的配置文件。

（1）修改**[hadoop-env.sh][]**，配置java jdk路径

`export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home`

(2) 修改**core-site.xml**，配置内容如下

<configuration>
        
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://locahost:9000</value>
        </property>
    </configuration>

（3）修改**hdfs-site.xml**，修改配置如下

<configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
    </configuration>

(4) 修改**mapred-site.xml**，修改配置如下

<configuration>
    
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>

（5）修改**yarn-site.xml**，修改内容如下

<configuration>
        
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>localhost</value>
        </property>
    </configuration>

#### 3. 配置环境变量 ####

将hadoop添加到环境变量`vim /etc/profile`，然后更新一下环境变量：`source /etc/profile`

export HADOOP_HOME=/usr/local/Cellar/hadoop/3.1.1
    PATH=$PATH:$HADOOP_HOME/bin

#### 4. 验证安装 ####

进入/usr/local/Cellar/hadoop/3.1.1目录。

**（1）格式化hadoop**

`bin/hdfs namenode -format`

执行成功，会在/tmp/hadoop-root/name/current下生成文件。

**(2) 启动hdfs**

`sbin/start-dfs.sh`

如果报错

Starting namenodes on [locahost]
    ERROR: Attempting to operate on hdfs namenode as root
    ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
    Starting datanodes
    ERROR: Attempting to operate on hdfs datanode as root
    ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
    ]tarting secondary namenodes [account.jetbrains.com
    ERROR: Attempting to operate on hdfs secondarynamenode as root
    ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
    2019-03-19 15:39:56,085 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

编辑sbin/start-dfs.sh，在文件开始空白处添加以下内容：

HDFS_DATANODE_USER=root
    HDFS_DATANODE_SECURE_USER=hdfs
    HDFS_NAMENODE_USER=root
    HDFS_SECONDARYNAMENODE_USER=root

如果报错

`WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable`

参考[Hadoop \_ 疑难杂症 解决1 - WARN util.NativeCodeLoader: Unable to load native-hadoop library for your plat][Hadoop _ _ _1 - WARN util.NativeCodeLoader_ Unable to load native-hadoop library for your plat]解决

**(3) 启动yarn**

`sbin/start-yarn.sh`

如果报错

Starting resourcemanager
    ERROR: Attempting to operate on yarn resourcemanager as root
    ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
    Starting nodemanagers
    ERROR: Attempting to operate on yarn nodemanager as root
    ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.

编辑sbin/start-yarn.sh，在文件开始空白处添加以下内容：

YARN_RESOURCEMANAGER_USER=root
    HADOOP_SECURE_DN_USER=yarn
    YARN_NODEMANAGER_USER=root

**(4)命令行查看和web访问**

验证是否成功，使用命令：jps，输出如下即表示配置成功。

12272 Jps
    4135 JobTracker
    9500 SecondaryNameNode
    9943 NodeManager
    9664 ResourceManager
    8898 NameNode
    9174 DataNode

也可以在浏览器中查看hdfs和mr的状态。

*  hdfs管理界面：[http://localhost:9870][http_localhost_9870]，3.0.0之前的版本端口
 *  MR的管理界面：[http://localhost:8088][http_localhost_8088]

### 3、安装Hive ###

（1）下载解压压缩包

在[Hive官网][Hive 1]下载对应版本的压缩包，我下载的是Hive3.1.1。解压到/usr/local目录下：

`sudo tar -zxvf apache-hive-3.1.1-bin.tar.gz -C /usr/local/`

将文件重命名为hive文件：

`mv apache-hive-3.1.1-bin hive`

（2）修改环境变量

修改环境变量/etc/profile：

`[root@s100 local]# vim /etc/profile`

#hive
    export HIVE_HOME=/usr/local/hive
    export PATH=$PATH:$HIVE_HOME/bin

执行`source /etc.profile`

（3）验证安装

执行`hive --version`

有hive的版本显现，即安装成功！

（4）配置hive

修改hive-site.xml文件

<property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://Master:3306/hive?createDatabaseIfNotExist=true</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

（5）将mysql的连接jar包拷贝到$HIVE\_HOME/lib目录下

（6）启动hive

`bin/hive`

[Hadoop3.0 _Hadoop3.0]: https://www.cnblogs.com/forbeat/p/8179877.html
[hadoop]: https://www.cnblogs.com/gyouxu/p/4183417.html
[hadoop 1]: https://blog.csdn.net/china1987427/article/details/82057250
[Hadoop_ Setting up a Single Node Cluster]: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation
[hadoop3.0]: https://blog.csdn.net/qq_32635069/article/details/80859790
[Hadoop _ _ _1 - WARN util.NativeCodeLoader_ Unable to load native-hadoop library for your plat]: https://blog.csdn.net/u010003835/article/details/81127984
[Unable to load native-hadoop library for your platform]: https://www.cnblogs.com/kevinq/p/5103653.html
[Hive]: https://www.cnblogs.com/dxxblog/p/8193967.html
[mysql]: https://dev.mysql.com/downloads/mysql/
[Hadoop]: https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
[hadoop-env.sh]: http://hadoop-env.sh
[http_localhost_9870]: http://localhost:9870
[http_localhost_8088]: http://localhost:8088
[Hive 1]: https://mirrors.tuna.tsinghua.edu.cn/apache/hive/