[Hadoop] Sqoop 从MySQL导入数据到hive表报错“Could not load org.apache.hadoop.hive.conf.HiveConf”解决

我就是我 2022-05-21 01:04 188阅读 0赞

Sqoop从MySQL导入数据到Hive表报错：

ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE\_CONF\_DIR is set correctly.

[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;
    Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
    Please set $HBASE_HOME to the root of your HBase installation.
    Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
    Please set $HCAT_HOME to the root of your HCatalog installation.
    Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
    Please set $ACCUMULO_HOME to the root of your Accumulo installation.
    Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
    Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
    18/06/26 22:39:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
    18/06/26 22:39:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    18/06/26 22:39:46 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
    18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
    18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
    18/06/26 22:39:48 INFO hive.HiveImport: Loading uploaded data into Hive
    18/06/26 22:39:48 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
    18/06/26 22:39:48 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    	at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
    	at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
    	at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
    	at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
    	at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    	at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
    	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:264)
    	at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
    	... 11 more
    
    [hadoop@hadoop01 ~]$

报错提示：java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

首先检查HiveConf类是否存在：

[hadoop@hadoop01 app]$ cd apache-hive-1.1.0-cdh5.7.0-bin/lib/
    [hadoop@hadoop01 lib]$ jar tf hive-common-1.1.0-cdh5.7.0.jar  | grep HiveConf.class
    org/apache/hadoop/hive/conf/HiveConf.class     
    [hadoop@hadoop01 lib]$

可以看到HiveConf.class是存在的，只是环境没有找到。

解决方法：

修改环境配置，将hive的lib添加到HADOOP\_CLASSPATH中

[root@hadoop01 ~]# vi /etc/profile
    export HADOOP_CLASSPATH=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/*
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/*

再次执行，成功。