eclipse集成hadoop+spark+hive本地开发图文详解

喜欢ヅ旅行 2022-06-16 01:52 186阅读 0赞

上一篇文章我们实现了[Java+Spark+Hive+Maven实现和异常处理][Java_Spark_Hive_Maven]，测试的实例是打包运行在linux环境，但当直接在Windows系统运行时，会有Hive相关异常的输出，本文将帮助您如何在Windows系统上集成Hadoop+Spark+Hive开发环境。

# 一.开发环境 #

系统：windows 7

JDK:jdk1.7

eclipse:Mars.2 Release (4.5.2)

Hadoop:hadoop-2.6.5

Spark:spark-1.6.2-bin-hadoop2.6

Hive:hive-2.1.1

# 二.前期准备 #

#### 1.系统环境配置 ####

JDK,Hadoop和Spark配置系统环境

#### 2.Hadoop相关文件 ####

winutils.exe和hadoop.dll，下载地址：[hadoop2.6.5中winutils和hadoop][hadoop2.6.5_winutils_hadoop]

将上面2个文件放置..\\hadoop-2.6.5\\bin目录下；

将winutils.exe同时放置到C:\\Windows\\System32目录下；

#### 3.新建tmp/hive目录 ####

在应用工程目录中新建tmp/hive目录，由于我的工程是放置在E盘，顾可以在E盘新建tmp/hive目录

# 三.hive配置 #

1.Hive环境

本系统的Hive是部署在远程linux集群环境上的。主安装目录ip地址：10.32.19.50:9083

具体Hive在linux环境的部署，请查看相关文档，本文不介绍。

2.Windows中hive-site.xml文件配置

<?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
    	
        <property>
            <name>hive.metastore.warehouse.dir</name>
            <value>/user/hive/warehouse</value>
        </property>
    
    	
        <property>
            <name>hive.metastore.local</name>
            <value>false</value>
        </property>
    
    	
        <property>
            <name>hive.metastore.uris</name>
            <value>thrift://10.32.19.50:9083</value>
        </property>
        
    </configuration>

![1240][]

windows中hive-site.xml配置

# 四.实例测试 #

需求：查询hive数据，eclipse正常显示

#### 1.实例工程结构 ####

![1240 1][]

实例工程

#### 2.pom文件 ####

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    	<modelVersion>4.0.0</modelVersion>
    
    	<groupId>com.lm.hive</groupId>
    	<artifactId>SparkHive</artifactId>
    	<version>0.0.1-SNAPSHOT</version>
    	<packaging>jar</packaging>
    
    	<name>SparkHive</name>
    	<url>http://maven.apache.org</url>
    
    	 <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      </properties>
    
     
        <dependencies>
            
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.10</artifactId>
                <version>1.6.0</version>
                <exclusions>
                    <exclusion>
                        <groupId>org.apache.hadoop</groupId>
                        <artifactId>hadoop-client</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_2.10</artifactId>
                <version>1.6.0</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-hive_2.10</artifactId>
                <version>1.6.0</version>
            </dependency>
    
            <dependency>
                <groupId>org.mongodb.spark</groupId>
                <artifactId>mongo-spark-connector_2.10</artifactId>
                <version>1.1.0</version>
            </dependency>
            <dependency>
                <groupId>org.apache.derby</groupId>
                <artifactId>derby</artifactId>
                <version>10.10.2.0</version>
            </dependency>
            
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>2.6.4</version>
                <exclusions>
                    <exclusion>
                        <groupId>javax.servlet</groupId>
                        <artifactId>*</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
    
        </dependencies>
        
        <build>
    		<sourceDirectory>src/main/java</sourceDirectory>
    		<testSourceDirectory>src/main/test</testSourceDirectory>
    
    		<plugins>
    			<plugin>
    				<artifactId>maven-assembly-plugin</artifactId>
    				<configuration>
    					<descriptorRefs>
    						<descriptorRef>jar-with-dependencies</descriptorRef>
    					</descriptorRefs>
    					<archive>
    						<manifest>
    							<mainClass></mainClass>
    						</manifest>
    					</archive>
    				</configuration>
    				<executions>
    					<execution>
    						<id>make-assembly</id>
    						<phase>package</phase>
    						<goals>
    							<goal>single</goal>
    						</goals>
    					</execution>
    				</executions>
    			</plugin>
    
    			<plugin>
    				<groupId>org.codehaus.mojo</groupId>
    				<artifactId>exec-maven-plugin</artifactId>
    				<version>1.2.1</version>
    				<executions>
    					<execution>
    						<goals>
    							<goal>exec</goal>
    						</goals>
    					</execution>
    				</executions>
    				<configuration>
    					<executable>java</executable>
    					<includeProjectDependencies>true</includeProjectDependencies>
    					<includePluginDependencies>false</includePluginDependencies>
    					<classpathScope>compile</classpathScope>
    					<mainClass></mainClass>
    				</configuration>
    			</plugin>
    
    			<plugin>
    				<groupId>org.apache.maven.plugins</groupId>
    				<artifactId>maven-compiler-plugin</artifactId>
    				<version>3.1</version>
    				<configuration>
    					<source>1.7</source>
    					<target>1.7</target>
    					<showWarnings>true</showWarnings>
    				</configuration>
    			</plugin>
    
    		</plugins>
    	</build>
    </project>

#### 3.测试用例实现 ####

package com.lm.hive.SparkHive;
    
    import org.apache.spark.SparkConf;
    import org.apache.spark.api.java.JavaSparkContext;
    import org.apache.spark.sql.hive.HiveContext;
    
    /**
     * Spark sql获取Hive数据
     *
     */
    public class App 
    {
        public static void main( String[] args ) 
        {
            SparkConf sparkConf = new SparkConf().setAppName("SparkHive").setMaster("local[2]");
            JavaSparkContext sc = new JavaSparkContext(sparkConf);
            
            //不要使用SQLContext,部署异常找不到数据库和表
            HiveContext hiveContext = new HiveContext(sc);
    //        SQLContext sqlContext = new SQLContext(sc);
            //查询表前10条数据
            hiveContext.sql("select * from bi_ods.owms_m_locator limit 10").show();
            
            sc.stop();
        }
    }

#### 4.测试结果展示 ####

![1240 2][]

测试结果展示

代码下载地址：[eclispe集成hadoop+spark+hive开发实例代码][eclispe_hadoop_spark_hive]

[Java_Spark_Hive_Maven]: http://blog.csdn.net/a123demi/article/details/70766843
[hadoop2.6.5_winutils_hadoop]: http://download.csdn.net/detail/a123demi/9831205
[1240]: /images/20220616/bbe1c1adc2bb4b8c865a2a67bdbc4d05.png
[1240 1]: /images/20220616/5d0c50c67b594d77935d18a9ed91e25b.png
[1240 2]: /images/20220616/65a4efaa77eb4916b5e8af8f13b315d2.png
[eclispe_hadoop_spark_hive]: http://download.csdn.net/detail/a123demi/9831220