centos7 Hive2.3.6安装与配置

清疚 2023-06-18 14:51 12阅读 0赞

#### 一.简介 ####

• Hive 由 **Facebook** 实现并开源，是基于 Hadoop 的一个数据仓库工具，可以将`结构化`的数据映射为一张数据库表，并提供 **HQL**(Hive SQL)查询功能，底层数据是存储在 HDFS 上。  
• Hive 本质: 将 SQL 语句转换为 MapReduce 任务运行，使不熟悉 MapReduce 的用户很方便地利用 HQL 处理和计算 HDFS 上的结构化的数据,是一款基于 HDFS 的 MapReduce 计算框架  
主要用途：用来做`离线数据分析`，比直接用 MapReduce 开发效率更高。

#### 二.安装与配置 ####

基于<<[Hadoop2.7.7 HA完全分布式集群搭建][Hadoop2.7.7 HA]>>

##### 1.下载 Hive2.3.6 #####

cd /usr/local
    wget https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz
    
    tar -zxvf apache-hive-2.3.6-bin.tar.gz

##### 2.配置hive环境变量 #####

# 编辑 /etc/profile 文件, 在其中添加以下内容 
    vim /etc/profile
    export HIVE_HOME=/usr/local/apache-hive-2.3.6-bin
    export PATH=$PATH:$HIVE_HOME/bin

# 激活配置
    source /etc/profile

##### 3.在hadoop下创建hive所用文件夹 #####

$HADOOP_HOME/bin/hadoop fs -mkdir  -p     /usr/local/apache-hive-2.3.6/tmp
    $HADOOP_HOME/bin/hadoop fs -mkdir  -p     /usr/local/apache-hive-2.3.6/warehouse
    $HADOOP_HOME/bin/hadoop fs -chmod g+w   /usr/local/apache-hive-2.3.6/tmp
    $HADOOP_HOME/bin/hadoop fs -chmod g+w   /usr/local/apache-hive-2.3.6/warehouse

##### 4.下载对应的mysql连接jar #####

https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-versions.html  
https://dev.mysql.com/downloads/connector/j/

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5_size_16_color_FFFFFF_t_70 1]  
将目标文件移动到 hive的lib文件夹下

mv mysql-connector-java-5.1.48.jar $HIVE_HOME/lib

创建数据库,配置用户和权限

create database metastore;
    grant all on metastore.* to hive@'%'  identified by 'hive';
    grant all on metastore.* to hive@'localhost'  identified by 'hive';
    flush privileges;

##### 5.配置 Hive #####

cd apache-hive-2.3.6-bin/conf/
    cp hive-env.sh.template hive-env.sh 
    cp hive-default.xml.template hive-site.xml 
    cp hive-log4j2.properties.template hive-log4j2.properties 
    cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties

修改 vim hive-env.sh

#Java路径
    export JAVA_HOME=/usr/local/java
    #Hadoop安装路径
    export HADOOP_HOME=/usr/local/hadoop-2.7.7
    #Hive安装路径
    export HIVE_HOME=/usr/local/apache-hive-2.3.6-bin
    #Hive配置文件路径
    export HIVE_CONF_DIR=/usr/local/apache-hive-2.3.6-bin/conf
    # hive存放外部jar包的位置
    export HIVE_AUX_JARS_PATH=/usr/local/apache-hive-2.3.6-bin/bin

修改hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <configuration>
    
    
    <property>
        <name>hive.exec.scratchdir</name>
        <value>/usr/local/apache-hive-2.3.6/tmp</value>
    </property>
    <property>
        <name>hive.exec.local.scratchdir</name>
       
        
        
        <value>/usr/local/apache-hive-2.3.6/tmp/${ user.name}</value>
        <description>Local scratch space for Hive jobs</description>
      </property>
    <property>
        <name>hive.downloaded.resources.dir</name>
        <value>/usr/local/apache-hive-2.3.6/tmp/${ hive.session.id}_resources</value>
        <description>Temporary local directory for added resources in the remote file system.</description>
      </property>
    
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/usr/local/apache-hive-2.3.6/warehouse</value>
    </property>
    <property>
        <name>hive.querylog.location</name>
        <value>/usr/local/apache-hive-2.3.6/logs/${ user.name}</value>
    </property>
     
    <property>
      <name>hive.server2.logging.operation.log.location</name>
      
        
      <value>/usr/local/apache-hive-2.3.6/logs/${ user.name}/operation_logs</value>
      <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
    </property>
    
    
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false</value>
      </property>
      <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
      </property>
      <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
      </property>
      <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive</value>
      </property>
    </configuration>

##### 6.初始化hive #####

cd $HIVE_HOME/bin
    ./schematool -dbType mysql -initSchema hive hive

##### 7.启动 hive的metastore元数据服务 #####

hive --service metastore

##### 8.启动hive #####

hive

Exception in thread "main" org.apache.thrift.transport.TTransportException:
    # 查看端口情况
    lsof -i:9083 
    # 再关闭端口即可
    kill -9 pid

参考:https://blog.csdn.net/wodedipang\_/article/details/72718138  
参考:https://blog.csdn.net/baixf/article/details/101052279  
参考:https://www.cnblogs.com/xibuhaohao/p/11772481.html

[Hadoop2.7.7 HA]: https://blog.csdn.net/Cocktail_py/article/details/102631199
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191203082810260.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191203082910471.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NvY2t0YWlsX3B5,size_16,color_FFFFFF,t_70