使用Azkaban调度执行spark-sql任务的小Demo

亦凉 2022-02-20 11:30 846阅读 0赞

# 1 、准备数据 #

准备表**dm\_action\_log** 数据如下：

<table> 
 <thead> 
  <tr> 
   <th>bdp_day</th> 
   <th>action</th> 
   <th>uv</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>20190101</td> 
   <td>click</td> 
   <td>11173</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>exit</td> 
   <td>11109</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>install</td> 
   <td>11139</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>launch</td> 
   <td>11083</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>login</td> 
   <td>11220</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>page_enter_h5</td> 
   <td>11016</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>page_enter_native</td> 
   <td>11076</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>page_exit</td> 
   <td>11120</td> 
  </tr> 
  <tr> 
   <td>20190101</td> 
   <td>register</td> 
   <td>11064</td> 
  </tr> 
  <tr> 
   <td>20190102</td> 
   <td>click</td> 
   <td>11168</td> 
  </tr> 
 </tbody> 
</table>

*···················只取前十条·························*

# 2 、编写azkaban调度job #

![201904131141432.png][]  
***dm\_action\_log.sh***

#!/bin/bash
    /usr/local/spark/bin/spark-sql --master yarn --name action_launch_job -S -f dm_action_log.hql

***dm\_action\_log.job***

type=command
    command=sh dm_action_log.sh

***dm\_action\_log.hql***

use databasename;
    create table if not exists databasename.dm_action_log_bak(
    bdp_day string comment '时间',
    action string comment '行为',
    uv int comment 'uv'
    )
    stored as parquet;
    
    insert overwrite table databasename.dm_action_log_bak
    select bdp_day,action,count(userid) uv from databasename.dw_action_log 
    group by bdp_day,action;

***这里注意这里需要是Unix环境（notepad++为例）***  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbm9rMTQ3_size_16_color_FFFFFF_t_70][]

# 3 、配置spark-sql运行环境 #

##### 3、1修改 *yarn-site.xml*，添加两行配置 #####

<property> 
       <name>yarn.nodemanager.vmem-check-enabled</name> 
       <value>false</value> 
    </property>
    
     <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4</value>
        <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
     </property>

##### 3、2 发送配置文件 #####

*  把修改后的 yarn-site.xml 发送到集群其他节点
 *  把$HADOOP\_HOME/etc/hadoop中的 core-site.xml 和 hdfs-site.xml发送到$SPARK\_HOME/conf/中（如果是spark集群，需要发送到每一个）
 *  把hive-site.xml同样也发送到spark的每个节点（$SPARK\_HOME\\conf\\）  
    ![20190413152025908.png][]

# 4 、启动hadoop集群和spark集群 #

# 5 、启动Azkaban并创建任务 #

![20190413152133851.png][]

# 6 、上传zip包执行任务 #

把 ***dm\_action\_log.sh***、***dm\_action\_log.job***、***dm\_action\_log.hql***，打成zip包  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbm9rMTQ3_size_16_color_FFFFFF_t_70 1][]

# 7 、等待执行成功 #

![20190413155809113.png][]

*发生ERROR怎么办？  
推荐以下两个网站*  
[网站一][Link 1]  
[网站二][Link 2]

[201904131141432.png]: /images/20220220/334845ff69434bbb995b57ce64e5e03d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbm9rMTQ3_size_16_color_FFFFFF_t_70]: /images/20220220/7b9c364c62344e86959147722ce7edbc.png
[20190413152025908.png]: /images/20220220/6d1268b11d3d4a2486bc41ea656e86ce.png
[20190413152133851.png]: /images/20220220/22292b6a39f64ac88bdcacb95deb5689.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbm9rMTQ3_size_16_color_FFFFFF_t_70 1]: /images/20220220/c2e41766dc2146f6b611ebb2aaf16509.png
[20190413155809113.png]: /images/20220220/c2841bb144094116bcff0a4548e447f5.png
[Link 1]: https://www.baidu.com
[Link 2]: http://www.google.com