Oozie WorkFlow中Sqoop Action使用讲解

浅浅的花香味﹌ 2022-12-19 06:22 113阅读 0赞

**第一步：** 将examples目录下的sqoop目录拷贝到oozie-apps目录下

cp -r sqoop /opt/modules/oozie/oozie-apps/

**第二步：** 将sqopp目录下的db.hsqldb.properties db.hsqldb.script两个文件删除

rm -rf db.hsqldb.properties db.hsqldb.script

**第三步：** 创建lib目录

mkdir lib

**第四步：** 拷贝mysql jar包到lib目录下

cp /opt/modules/hive/lib/mysql-connector-java-5.1.48-bin.jar .

**第五步：** 这里我们测试的是hive表导出到mysql表。需要检查hive表和mysql表，分别是hive表（db\_hive.user\_info）和mysql表(db\_sqoop.user\_info\_export)  
关于hive表导出到mysql表的内容，以及创建表的格式和数据请参考这篇文章的第四节：  
[https://blog.csdn.net/weixin\_45366499/article/details/109403343][https_blog.csdn.net_weixin_45366499_article_details_109403343]

**第六步：** 编写sqoop导出脚本文件export.txt，内容如下：

--connect 
    jdbc:mysql://bigdata-pro-m01:3306/db_sqoop 
    --username 
    root 
    --password 
    199911 
    --num-mappers 
    1  
    --table 
    user_info_export 
    --input-fields-terminated-by 
    ',' 
    --export-dir 
    /user/hive/warehouse/db_hive.db/user_info

export.txt在`/opt/modules/oozie/oozie-apps/sqoop`目录下  
**第七步：** 编写job.properties文件，参考官方example，内容如下：

nameNode=hdfs://bigdata-pro-m01:9000
    jobTracker=bigdata-pro-m01:8032
    
    queueName=default
    oozieAppRoot=user/caizhengjie/oozie-apps
    oozieDataRoot=user/caizhengjie/oozie-datas
    
    
    oozie.use.system.libpath=true
    
    oozie.wf.application.path=${nameNode}/${oozieAppRoot}/sqoop
    
    
    EXEC=export.txt

**第八步：** 编写workflow.xml，内容如下：

<workflow-app xmlns="uri:oozie:workflow:0.5" name="sqoop-wf">
        <start to="sqoop-node"/>
    
        <action name="sqoop-node">
            <sqoop xmlns="uri:oozie:sqoop-action:0.3">
                <job-tracker>${jobTracker}</job-tracker>
                <name-node>${nameNode}</name-node>
    
                <configuration>
                    <property>
                        <name>mapred.job.queue.name</name>
                        <value>${queueName}</value>
                    </property>
                </configuration>
    
                <command>export --options-file ${EXEC}</command>
                <file>${nameNode}/${oozieAppRoot}/sqoop/${EXEC}#${EXEC}</file>
                
            </sqoop>
            <ok to="end"/>
            <error to="fail"/>
        </action>
    
        <kill name="fail">
            <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
        <end name="end"/>
    </workflow-app>

**第九步：** 上传sqoop整个目录到HDFS上

bin/hdfs dfs -put /opt/modules/oozie/oozie-apps/sqoop/ /user/caizhengjie/oozie-apps

**第十步：** 运行测试

bin/oozie job -oozie http://bigdata-pro-m01:11000/oozie -config oozie-apps/sqoop/job.properties -run

**查看测试结果：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTM2NjQ5OQ_size_16_color_FFFFFF_t_70_pic_center]  
查看导出的mysql数据表

mysql> select * from user_info_export;
    +----+----------+----------+
    | id | username | password |
    +----+----------+----------+
    |  1 | admin    | admin    |
    |  2 | wang     | 111111   |
    |  3 | zhang    | 000000   |
    |  4 | lili     | 000000   |
    |  5 | henry    | 000000   |
    |  6 | cherry   | 000000   |
    |  7 | ben      | 111111   |
    |  8 | leo      | 000000   |
    |  9 | test     | test     |
    | 10 | system   | 000000   |
    | 11 | xiao     | 111111   |
    +----+----------+----------+
    11 rows in set (0.00 sec)

--------------------

以上内容仅供参考学习，如有侵权请联系我删除！  
如果这篇文章对您有帮助，左下角的大拇指就是对博主最大的鼓励。  
您的鼓励就是博主最大的动力！

[https_blog.csdn.net_weixin_45366499_article_details_109403343]: https://blog.csdn.net/weixin_45366499/article/details/109403343
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTM2NjQ5OQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/47629301a4bf4dd19dee3c0327eadcf2.png