Centos7.x 搭建部署 Hive 3.1.2单台+集群

短命女 2022-10-05 12:59 346阅读 0赞

## 1.基础环境Hadoop3.2.2集群 ##

mysql数据库  
[基于centos部署hadoop集群][centos_hadoop]

## 2.Hive 的安装 ##

1.下载hive安装文件

wget https://mirrors.bfsu.edu.cn/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz

2.下载后进行解压 tar -zxf  
3.进行配置文件修改，进入conf文件夹

cp hive-default.xml.template hive-site.xml
    cp hive-env.sh.template hive-env.sh
    cp hive-log4j2.properties.template hive-log4j2.properties

4.修改hive-site.xml，一下mysqlip需替换，以及密码信息

<configuration>
        <property>
            <name>hive.metastore.schema.verification</name>
            <value>false</value>
        </property>
        <property>
            <name>hive.metastore.warehouse.dir</name>
            <value>/data/hive/warehouse</value>
        </property>
        <property>
            <name>hive.exec.mode.local.auto</name>
            <value>false</value>
            <description> Let Hive determine whether to run in local mode automatically </description>
        </property>
        <property>
            <name>javax.jdo.option.ConnectionURL</name>
            <value>jdbc:mysql://mysqlip:3306/hive??useUnicode=true&characterEncoding=utf8&useSSL=false&allowMultiQueries=true</value>
        </property>
        <property>
            <name>javax.jdo.option.ConnectionDriverName</name>
            <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
            <name>javax.jdo.option.ConnectionUserName</name>
            <value>root</value>
        </property>
        <property>
            <name>javax.jdo.option.ConnectionPassword</name>
            <value>root</value>
        </property>
        
        <property>
          <name>hive.cli.print.header</name>
          <value>true</value>
        </property>
        
        <property>
          <name>hive.cli.print.current.db</name>
          <value>true</value>
        </property>
        <property>
          <name>hive.exec.local.scratchdir</name>
          <value>/data/apache-hive-3.1.2/tmp/</value>
        <description>Local scratch space for Hive jobs</description>
        </property>
        <property>
          <name>hive.downloaded.resources.dir</name>
          <value>/data/apache-hive-3.1.2/tmp/</value>
          <description>Temporary local directory for added resources in the remote file system.</description>
        </property>
    </configuration>

5.修改hive-env.sh

export HADOOP_HEAPSIZE=1024
    HADOOP_HOME=/data/hadoop-3.2.2
    export HIVE_CONF_DIR=/data/apache-hive-3.1.2/conf

6.修改hive-log4j2.properties

property.hive.log.dir = /data/apache-hive-3.1.2/logs

7.编辑bin目录下的hive-config.sh

export JAVA_HOME=/usr/java/java-1.8.0
    export HADOOP_HOME=/data/hadoop-3.2.2
    export HIVE_HOME=/data/apache-hive-3.1.2

8.修改环境变量

export HADOOP_HOME="/data/hadoop-3.2.2"
    export HIVE_HOME="/data/apache-hive-3.1.2"
     
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"
     
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin
    export CLASSPATH=$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/common/lib/*:$hive_HOME/lib

9.下载mysql驱动包放到lib目录下

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.48/mysql-connector-java-5.1.48.jar

10.检查guava版本是否一直，不一致则统一替换为高版本，不然初始化sql会报错  
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)  
jar包目录  
/apache-hive-3.1.2-bin/lib/  
hadoop/share/hadoop/common/lib/  
这里将hive的guava的jar包删除，从hadoop的进行复制

11.执行数据库初始化命令

schematool -initSchema -dbType mysql

Initialization script completed  
schemaTool completed  
表示初始化成功，打开数据库，可以看到相关表信息。

12.启动hive，执行命令

hive

## 三.基础命令 ##

1.查看所有数据库列表

show databases;

2.创建数据库

create database db_test;

3.查看创建的数据库在 HDFS 中的信息（随意找一个 HDFS 客户端节点）

hadoop fs -ls /data/hive/warehouse
    hdfs dfs -ls /user/hive/warehouse

4.创建表

use db_test;
    create table t_test(id string,name string,age int);
    insert into table_test values ('1000','名称',21);
    select * from table_test;

## 四.集群部署 ##

1.在hive-site.xml添加以下配置

<property>
        <name>hive.server2.active.passive.ha.enable</name>
        <value>true</value>
        <description>Whether HiveServer2 Active/Passive High Availability be enabled when Hive Interactive sessions are enabled.This will also require hive.server2.support.dynamic.service.discovery to be enabled.</description>
      </property>

2.拷贝hive安装包到其他服务器

scp -r apache-hive-3.1.2 hadoop2:/data
    scp -r apache-hive-3.1.2 hadoop3:/data

3.将其他两台的环境变量保持和当前一直  
4.hiveserver2 服务启动

[root@master ~]# nohup hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10001 &

5.访问控制台  
http://hadoop1:10002/

[centos_hadoop]: https://blog.csdn.net/runner1920/article/details/117731154