SpringBoot 3 集成Hive 3

ゞ浴缸里的玫瑰 2024-04-23 16:01 70阅读 0赞

**前提条件:**

运行环境：Hadoop 3.\* + Hive 3.\* + MySQL 8 ，如果还未安装相关环境，请参考：[Hive 一文读懂][Hive]

[Centos7 安装Hadoop3 单机版本（伪分布式版本）][Centos7 _Hadoop3]

## **SpringBoot 2 集成Hive 3** ##

### pom.xml ###

<?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <parent>
            <artifactId>SpringBootCase</artifactId>
            <groupId>org.example</groupId>
            <version>1.0-SNAPSHOT</version>
        </parent>
        <modelVersion>4.0.0</modelVersion>
    
        <artifactId>SpringBoot-Hive3</artifactId>
    
        <properties>
            <maven.compiler.source>8</maven.compiler.source>
            <maven.compiler.target>8</maven.compiler.target>
        </properties>
    
        <dependencies>
            <dependency>
                <groupId>org.apache.hive</groupId>
                <artifactId>hive-jdbc</artifactId>
                <version>3.1.2</version>
                <exclusions>
                    <exclusion>
                        <artifactId>slf4j-log4j12</artifactId>
                        <groupId>org.slf4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-api</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-core</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j</artifactId>
                        <groupId>log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-slf4j-impl</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <groupId>org.eclipse.jetty</groupId>
                        <artifactId>jetty-runner</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
    
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-jdbc</artifactId>
            </dependency>
        </dependencies>
    
    </project>

### **配置application.properties** ###

server.port=8083
    # hive 驱动名称
    spring.datasource.driver-class-name=org.apache.hive.jdbc.HiveDriver
    # hive 数据库地址 = jdbc:hive2://hive 服务器地址:10000/default(默认数据库名称)
    spring.datasource.url=jdbc:hive2://192.168.43.11:10000/default
    # hive 服务器用户名
    spring.datasource.username=root
    # hive  服务器密码
    spring.datasource.password=123456

### 编写Controller和应用入口 ###

我这边编写一个简单的Controller，打印Hive 默认数据库包含数据库名称。

package cn.zzg.hive.controller;
    
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.jdbc.core.JdbcTemplate;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.bind.annotation.RestController;
    
    import java.util.List;
    import java.util.Map;
    
    @RestController
    @RequestMapping("/hive")
    public class HiveController {
        @Autowired
        private JdbcTemplate jdbcTemplate;
    
        @RequestMapping("/list")
        public List<Map<String, Object>> list() {
            String sql = "show databases";
            List<Map<String, Object>> list = jdbcTemplate.queryForList(sql);
            return list;
        }
    
    }

package cn.zzg.hive;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class Application {
        public static void main(String[] args) {
            SpringApplication.run(Application.class, args);
        }
    }

效果截图：

![0a171b8a68e540f68a94d6c290d78cf3.png][]

## **SpringBoot 2 集成Hive 3 遇到的问题** ##

**问题一：**Class path contains multiple SLF4J bindings，日志依赖重复冲突。

造成此问题的原因是:spring boot 默认日志为logback, 而引用的hive-jdbc 及其关联jar 使用的日志为 log4j ，造成SLF4J 绑定冲突。

解决办法：移除冲突的日志：log4j

<exclusion>
                        <artifactId>slf4j-log4j12</artifactId>
                        <groupId>org.slf4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-api</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-core</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j</artifactId>
                        <groupId>log4j</groupId>
                    </exclusion>
                    <exclusion>
                        <artifactId>log4j-slf4j-impl</artifactId>
                        <groupId>org.apache.logging.log4j</groupId>
                    </exclusion>

**问题二：SpringBoot 自带容器Tomcat 与Hive JDBC 关联Jetty 容器冲突**

An attempt was made to call the method org.apache.tomcat.util.ExceptionUtils.preload()V but it does not exist. Its class, org.apache.tomcat.util.ExceptionUtils, is available from the following locations:
    
        jar:file:/E:/maven_repository/org/eclipse/jetty/jetty-runner/9.3.20.v20170531/jetty-runner-9.3.20.v20170531.jar!/org/apache/tomcat/util/ExceptionUtils.class
        jar:file:/E:/maven_repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.12/tomcat-embed-core-9.0.12.jar!/org/apache/tomcat/util/ExceptionUtils.class

解决办法：移除Hive JDBC 依赖的Jetty 容器。

<exclusion>
                        <groupId>org.eclipse.jetty</groupId>
                        <artifactId>jetty-runner</artifactId>
                    </exclusion>

**问题三：通过JDBC 连接Hive 数据库提示：**

java.net.ConnectException: Connection refused

造成此类 问题的原因：hiveserver2 服务没有正常启动。

解决办法: 切换至hive 安装目录的bin/ 文件夹下（/usr/local/hive/bin），执行如下命令：

# 方式一
    hiveserver2 &
    # 方式二
    hive --service hiveserver2

**问题四：访问Hive 数据库，提示无权限问题：**

造成此类问题的原因：hadoop 没有配置权限导致。

解决办法：切换至hadoop 安装目录的/etc文件夹下(/usr/local/hadoop/etc/hadoop/core-site.xml),添加如下配置:

<property>
                    <name>hadoop.proxyuser.root.hosts</name>
                    <value>*</value>
            </property>
            <property>
                    <name>hadoop.proxyuser.root.groups</name>
                    <value>*</value>
            </property>

[Hive]: https://blog.csdn.net/zhouzhiwengang/article/details/129228750
[Centos7 _Hadoop3]: https://blog.csdn.net/zhouzhiwengang/article/details/128991335?spm=1001.2014.3001.5501
[0a171b8a68e540f68a94d6c290d78cf3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/23/8061c0780140447cb68cb7ceef184bdf.png