安装Hive3.1.1

ゞ浴缸里的玫瑰 2022-03-28 11:54 215阅读 0赞

## 安装Hive3.1.1 ##

前提：安装了hadoop，本例是伪分布模式下的hadoop，hadoop伪分布模式安装见[Hadoop:单节点设置][Hadoop]  
1、下载hive，将安装包放置到服务器目录，进入你放置hive安装包的目录，解压。

cd /opt
    tar -zxvf apache-hive-3.1.1-bin.tar.gz

2、配置hive环境变量。

echo "export HIVE_HOME=$PWD/apache-hive-3.1.1-bin" > /etc/profile.d/hive.sh
    echo "PATH=$PATH:$HIVE_HOME/bin" >> /etc/profile.d/hive.sh
    source /etc/profile

3、进入conf目录生成hive-site.xml配置文件

cd conf/
    cp hive-default.sh.template hive-site.sh

4、配置hive-site.xml

<property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/user/hive/warehouse</value>
        <description>location of default database for the warehouse</description>
      </property>

同时创建该目录

mkdir -p /user/hive/warehouse

5、初始化数据库

schematool -initSchema -dbType derby

6、运行hive

hive

会报错：

Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: **Illegal character entity: expansion character (code 0x8
     at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]**
    	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
    	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
    	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
    	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
    	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
    	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
    	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
    	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5098)
    	at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
    	at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
    Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
     at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]
    	at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
    	at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
    	at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
    	at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
    	at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
    	at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
    	at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
    	at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
    	at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
    	at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
    	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
    	... 15 more

非法字符，hive-site.xml的3210行有错，

vim +3210 conf/hive-site.xml
    
    3206   <property>
    3207     <name>hive.txn.xlock.iow</name>
    3208     <value>true</value>
    3209     <description>
    3210       Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks fortransactional tables.  This ensures that inserts (w/o overwrite) running concurrently
    3211       are not hidden by the INSERT OVERWRITE.
    3212     </description>
    3213   </property>

删除&\#8;  
7、运行hive  
报错：

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    	at org.apache.hadoop.fs.Path.initialize(Path.java:259)
    	at org.apache.hadoop.fs.Path.<init>(Path.java:217)
    	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:707)
    	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:624)
    	at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:588)
    	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
    	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
    Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    	at java.net.URI.checkPath(URI.java:1823)
    	at java.net.URI.<init>(URI.java:745)
    	at org.apache.hadoop.fs.Path.initialize(Path.java:256)
    	... 12 more

URI路径问题，在配置文件hive-site.xml里面将$\{system:java.io.tmpdir\} 全部替换为/opt/apache-hive-3.1.1-bin/tmp 将\{system:user.name\} 全部替换为 \{ [user.name][]\}  
vim命令行模式输入以下命令全局替换。

:1,$s/${system:java.io.tmpdir}/\/opt\/apache-hive-3.1.1-bin\/tmp/g
    :1,$s/{system:user.name}/{user.name}/g

8、运行hive

hive

成功！以上练习hive环境搭配hadoop的伪分布模式，使用的是Derby数据库，只能提供有限的，单进程的存储服务，例如用户不可以执行2个并发的hive命令行界面。个人计算机上或者某些开发任务上使用的话没有问题，对于集群来说，需要使用MySQL或者类似的关系型数据库。  
9、其他问题  
进入hive后使用hql报错：

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

不能初始化元数据库，需要配置第四步，指定默认数据库仓库的位置，删掉你之前工作目录（即在哪个目录运行了hive这个命令）中的metastore\_db，并重新初始化。

rm -rf metastore_db
    schematool -initSchema -dbType derby

10、设置mysql为元数据库

*  1.在mysql中创建hive元数据库：

create database hive_db;

*  2.修改hive-site.xml中一下属性：

<property>
        	
            <name>javax.jdo.option.ConnectionUserName</name>
            <value>root</value>
        </property>
        <property>
    	   	
            <name>javax.jdo.option.ConnectionPassword</name>
            <value>123456</value>
        </property>
       <property>
       		
            <name>javax.jdo.option.ConnectionURL</name>
            <value>jdbc:mysql://192.168.1.68:3306/hive_db?useSSL=TRUE</value>
        </property>
        <property>
        	
            <name>javax.jdo.option.ConnectionDriverName</name>
            <value>com.mysql.jdbc.Driver</value>
        </property>

*  3.复制mysql的驱动程序到hive/lib下面。
 *  4.初始化数据库:

schematool -dbType mysql -initSchema

*  5.运行hive

[Hadoop]: https://blog.csdn.net/qiulinsama/article/details/85601194
[user.name]: http://user.name