Apache DolphinScheduler(海豚调度) - 1.3 系列核心表结构剖析

末蓝、 2023-02-22 12:30 28阅读 0赞

Apache DolphinScheduler 是一个分布式去中心化，易扩展的可视化 DAG 工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。

近日，伯毅同学给社区贡献了工作流核心表结构的剖析文章，非常细致，喜欢的伙伴请转走

# 1. 工作流总体存储结构 #

在 dolphinscheduler 库中创建的所有工作流定义(模板)都保存在 t\_ds\_process\_definition 表中.

该数据库表结构如下表所示:

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>字段</th> 
   <th>类型</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td>int(11)</td> 
   <td>主键</td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>name</td> 
   <td>varchar(255)</td> 
   <td>流程定义名称</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>version</td> 
   <td>int(11)</td> 
   <td>流程定义版本</td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>release_state</td> 
   <td>tinyint(4)</td> 
   <td>流程定义的发布状态：0 未上线 , 1已上线</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td>project_id</td> 
   <td>int(11)</td> 
   <td>项目id</td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td>user_id</td> 
   <td>int(11)</td> 
   <td>流程定义所属用户id</td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td>process_definition_json</td> 
   <td>longtext</td> 
   <td>流程定义JSON</td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td>description</td> 
   <td>text</td> 
   <td>流程定义描述</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td>global_params</td> 
   <td>text</td> 
   <td>全局参数</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td>flag</td> 
   <td>tinyint(4)</td> 
   <td>流程是否可用：0 不可用，1 可用</td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td>locations</td> 
   <td>text</td> 
   <td>节点坐标信息</td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>connects</td> 
   <td>text</td> 
   <td>节点连线信息</td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>receivers</td> 
   <td>text</td> 
   <td>收件人</td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td>receivers_cc</td> 
   <td>text</td> 
   <td>抄送人</td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td>create_time</td> 
   <td>datetime</td> 
   <td>创建时间</td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td>timeout</td> 
   <td>int(11)</td> 
   <td>超时时间</td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>tenant_id</td> 
   <td>int(11)</td> 
   <td>租户id</td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>update_time</td> 
   <td>datetime</td> 
   <td>更新时间</td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>modify_by</td> 
   <td>varchar(36)</td> 
   <td>修改用户</td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>resource_ids</td> 
   <td>varchar(255)</td> 
   <td>资源ids</td> 
  </tr> 
 </tbody> 
</table>

其中 process\_definition\_json 字段为核心字段, 定义了 DAG 图中的任务信息.该数据以JSON 的方式进行存储.

公共的数据结构如下表:

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>字段</th> 
   <th>类型</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>globalParams</td> 
   <td>Array</td> 
   <td>全局参数</td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>tasks</td> 
   <td>Array</td> 
   <td>流程中的任务集合 [ 各个类型的结构请参考如下章节]</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>tenantId</td> 
   <td>int</td> 
   <td>租户id</td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>timeout</td> 
   <td>int</td> 
   <td>超时时间</td> 
  </tr> 
 </tbody> 
</table>

数据示例:

{ 
        "globalParams":[
            { 
                "prop":"golbal_bizdate",
                "direct":"IN",
                "type":"VARCHAR",
                "value":"${system.biz.date}"
            }
        ],
        "tasks":Array[1],
        "tenantId":0,
        "timeout":0
    }

# 2. 各任务类型存储结构详解 #

## 2.1 Shell 节点 ##

\*\* Shell 节点数据结构如下:\*\*

**Shell 节点数据样例:**

{ 
        "type":"SHELL",
        "id":"tasks-80760",
        "name":"Shell Task",
        "params":{ 
            "resourceList":[
                { 
                    "id":3,
                    "name":"run.sh",
                    "res":"run.sh"
                }
            ],
            "localParams":[
            ],
            "rawScript":"echo "This is a shell script""
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.2 SQL节点 ##

通过 SQL 对指定的数据源进行数据查询、更新操作.

\*\* SQL 节点数据结构如下:\*\*

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>SQL</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>type</td> 
   <td>String</td> 
   <td>数据库类型</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>datasource</td> 
   <td>Int</td> 
   <td>数据源id</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>sql</td> 
   <td>String</td> 
   <td>查询SQL语句</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>udfs</td> 
   <td>String</td> 
   <td>udf函数</td> 
   <td>UDF函数id,以逗号分隔.</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>sqlType</td> 
   <td>String</td> 
   <td>SQL节点类型</td> 
   <td>0 查询 , 1 非查询</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>title</td> 
   <td>String</td> 
   <td>邮件标题</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>receivers</td> 
   <td>String</td> 
   <td>收件人</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td></td> 
   <td>receiversCc</td> 
   <td>String</td> 
   <td>抄送人</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td></td> 
   <td>showType</td> 
   <td>String</td> 
   <td>邮件显示类型</td> 
   <td>TABLE 表格 , ATTACHMENT附件</td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>connParams</td> 
   <td>String</td> 
   <td>连接参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>preStatements</td> 
   <td>Array</td> 
   <td>前置SQL</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>postStatements</td> 
   <td>Array</td> 
   <td>后置SQL</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>24</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>25</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>26</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>27</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>28</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>29</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

\*\* SQL 节点数据样例:\*\*

{ 
        "type":"SQL",
        "id":"tasks-95648",
        "name":"SqlTask-Query",
        "params":{ 
            "type":"MYSQL",
            "datasource":1,
            "sql":"select id , namge , age from emp where id = ${id}",
            "udfs":"",
            "sqlType":"0",
            "title":"xxxx@xxx.com",
            "receivers":"xxxx@xxx.com",
            "receiversCc":"",
            "showType":"TABLE",
            "localParams":[
                { 
                    "prop":"id",
                    "direct":"IN",
                    "type":"INTEGER",
                    "value":"1"
                }
            ],
            "connParams":"",
            "preStatements":[
                "insert into emp ( id,name ) value (1,'Li' )"
            ],
            "postStatements":[
            ]
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.3 Spark 节点 ##

\*\* Spark 节点数据结构如下:\*\*

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>SPARK</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>mainClass</td> 
   <td>String</td> 
   <td>运行主类</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>mainArgs</td> 
   <td>String</td> 
   <td>运行参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>others</td> 
   <td>String</td> 
   <td>其他参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>mainJar</td> 
   <td>Object</td> 
   <td>程序 jar 包</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>deployMode</td> 
   <td>String</td> 
   <td>部署模式</td> 
   <td>local,client,cluster</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>driverCores</td> 
   <td>String</td> 
   <td>driver核数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>driverMemory</td> 
   <td>String</td> 
   <td>driver 内存数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td></td> 
   <td>numExecutors</td> 
   <td>String</td> 
   <td>executor数量</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td></td> 
   <td>executorMemory</td> 
   <td>String</td> 
   <td>executor内存</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>executorCores</td> 
   <td>String</td> 
   <td>executor核数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>programType</td> 
   <td>String</td> 
   <td>程序类型</td> 
   <td>JAVA,SCALA,PYTHON</td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>sparkVersion</td> 
   <td>String</td> 
   <td>Spark 版本</td> 
   <td>SPARK1 , SPARK2</td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td></td> 
   <td>resourceList</td> 
   <td>Array</td> 
   <td>资源文件</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>24</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>25</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>26</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>27</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>28</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>29</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>30</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

\*\* Spark 节点数据样例:\*\*

{ 
        "type":"SPARK",
        "id":"tasks-87430",
        "name":"SparkTask",
        "params":{ 
            "mainClass":"org.apache.spark.examples.SparkPi",
            "mainJar":{ 
                "id":4
            },
            "deployMode":"cluster",
            "resourceList":[
                { 
                    "id":3,
                    "name":"run.sh",
                    "res":"run.sh"
                }
            ],
            "localParams":[
            ],
            "driverCores":1,
            "driverMemory":"512M",
            "numExecutors":2,
            "executorMemory":"2G",
            "executorCores":2,
            "mainArgs":"10",
            "others":"",
            "programType":"SCALA",
            "sparkVersion":"SPARK2"
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.4 MapReduce(MR)节点 ##

\*\* MapReduce(MR) 节点数据结构如下:\*\*

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>MR</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>mainClass</td> 
   <td>String</td> 
   <td>运行主类</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>mainArgs</td> 
   <td>String</td> 
   <td>运行参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>others</td> 
   <td>String</td> 
   <td>其他参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>mainJar</td> 
   <td>Object</td> 
   <td>程序 jar 包</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>programType</td> 
   <td>String</td> 
   <td>程序类型</td> 
   <td>JAVA,PYTHON</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>resourceList</td> 
   <td>Array</td> 
   <td>资源文件</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

\*\* MapReduce(MR) 节点数据样例:\*\*

{ 
        "type":"MR",
        "id":"tasks-28997",
        "name":"MRTask",
        "params":{ 
            "mainClass":"wordcount",
            "mainJar":{ 
                "id":5
            },
            "resourceList":[
                { 
                    "id":3,
                    "name":"run.sh",
                    "res":"run.sh"
                }
            ],
            "localParams":[
            ],
            "mainArgs":"/tmp/wordcount/input /tmp/wordcount/output/",
            "others":"",
            "programType":"JAVA"
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.5 Python节点 ##

\*\* Python 节点数据结构如下:\*\*

**Python节点数据样例:**

{ 
        "type":"PYTHON",
        "id":"tasks-5463",
        "name":"Python Task",
        "params":{ 
            "resourceList":[
                { 
                    "id":3,
                    "name":"run.sh",
                    "res":"run.sh"
                }
            ],
            "localParams":[
            ],
            "rawScript":"print("This is a python script")"
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.6 Flink节点 ##

**Flink 节点数据结构如下:**

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>FLINK</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>mainClass</td> 
   <td>String</td> 
   <td>运行主类</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>mainArgs</td> 
   <td>String</td> 
   <td>运行参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>others</td> 
   <td>String</td> 
   <td>其他参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>mainJar</td> 
   <td>Object</td> 
   <td>程序 jar 包</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>deployMode</td> 
   <td>String</td> 
   <td>部署模式</td> 
   <td>local,client,cluster</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>slot</td> 
   <td>String</td> 
   <td>slot数量</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>taskManager</td> 
   <td>String</td> 
   <td>taskManage数量</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td></td> 
   <td>taskManagerMemory</td> 
   <td>String</td> 
   <td>taskManager内存数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td></td> 
   <td>jobManagerMemory</td> 
   <td>String</td> 
   <td>jobManager内存数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>programType</td> 
   <td>String</td> 
   <td>程序类型</td> 
   <td>JAVA,SCALA,PYTHON</td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>resourceList</td> 
   <td>Array</td> 
   <td>资源文件</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>24</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>25</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>26</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>27</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>38</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

\*\* Flink 节点数据样例:\*\*

{ 
        "type":"FLINK",
        "id":"tasks-17135",
        "name":"FlinkTask",
        "params":{ 
            "mainClass":"com.flink.demo",
            "mainJar":{ 
                "id":6
            },
            "deployMode":"cluster",
            "resourceList":[
                { 
                    "id":3,
                    "name":"run.sh",
                    "res":"run.sh"
                }
            ],
            "localParams":[
            ],
            "slot":1,
            "taskManager":"2",
            "jobManagerMemory":"1G",
            "taskManagerMemory":"2G",
            "executorCores":2,
            "mainArgs":"100",
            "others":"",
            "programType":"SCALA"
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.7 Http 节点 ##

**Http 节点数据结构如下:**

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>HTTP</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>url</td> 
   <td>String</td> 
   <td>请求地址</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>httpMethod</td> 
   <td>String</td> 
   <td>请求方式</td> 
   <td>GET,POST,HEAD,PUT,DELETE</td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>httpParams</td> 
   <td>Array</td> 
   <td>请求参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>httpCheckCondition</td> 
   <td>String</td> 
   <td>校验条件</td> 
   <td>默认响应码200</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>condition</td> 
   <td>String</td> 
   <td>校验内容</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

\*\* Http 节点数据样例:\*\*

{ 
        "type":"HTTP",
        "id":"tasks-60499",
        "name":"HttpTask",
        "params":{ 
            "localParams":[
            ],
            "httpParams":[
                { 
                    "prop":"id",
                    "httpParametersType":"PARAMETER",
                    "value":"1"
                },
                { 
                    "prop":"name",
                    "httpParametersType":"PARAMETER",
                    "value":"Bo"
                }
            ],
            "url":"https://www.xxxxx.com:9012",
            "httpMethod":"POST",
            "httpCheckCondition":"STATUS_CODE_DEFAULT",
            "condition":""
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.8 DataX节点 ##

\*\* DataX 节点数据结构如下:\*\*

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>DATAX</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>customConfig</td> 
   <td>Int</td> 
   <td>自定义类型</td> 
   <td>0定制 , 1自定义</td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>dsType</td> 
   <td>String</td> 
   <td>源数据库类型</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>dataSource</td> 
   <td>Int</td> 
   <td>源数据库ID</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>dtType</td> 
   <td>String</td> 
   <td>目标数据库类型</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>dataTarget</td> 
   <td>Int</td> 
   <td>目标数据库ID</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>sql</td> 
   <td>String</td> 
   <td>SQL语句</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>targetTable</td> 
   <td>String</td> 
   <td>目标表</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td></td> 
   <td>jobSpeedByte</td> 
   <td>Int</td> 
   <td>限流(字节数)</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td></td> 
   <td>jobSpeedRecord</td> 
   <td>Int</td> 
   <td>限流(记录数)</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>preStatements</td> 
   <td>Array</td> 
   <td>前置SQL</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>postStatements</td> 
   <td>Array</td> 
   <td>后置SQL</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>json</td> 
   <td>String</td> 
   <td>自定义配置</td> 
   <td>customConfig=1时生效</td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td>customConfig=1时生效</td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>24</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>25</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>26</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>27</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>28</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>29</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

**DataX 节点数据样例:**

{ 
        "type":"DATAX",
        "id":"tasks-91196",
        "name":"DataxTask-DB",
        "params":{ 
            "customConfig":0,
            "dsType":"MYSQL",
            "dataSource":1,
            "dtType":"MYSQL",
            "dataTarget":1,
            "sql":"select id, name ,age from user ",
            "targetTable":"emp",
            "jobSpeedByte":524288,
            "jobSpeedRecord":500,
            "preStatements":[
                "truncate table emp "
            ],
            "postStatements":[
                "truncate table user"
            ]
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                ""
            ],
            "failedNode":[
                ""
            ]
        },
        "dependence":{ 
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
        ]
    }

## 2.9 Sqoop节点 ##

**Sqoop 节点数据结构如下:**

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>SQOOP</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>JSON 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>concurrency</td> 
   <td>Int</td> 
   <td>并发度</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>modelType</td> 
   <td>String</td> 
   <td>流向</td> 
   <td>import,export</td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>sourceType</td> 
   <td>String</td> 
   <td>数据源类型</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td></td> 
   <td>sourceParams</td> 
   <td>String</td> 
   <td>数据源参数</td> 
   <td>JSON格式</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td></td> 
   <td>targetType</td> 
   <td>String</td> 
   <td>目标数据类型</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td></td> 
   <td>targetParams</td> 
   <td>String</td> 
   <td>目标数据参数</td> 
   <td>JSON格式</td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>22</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>23</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

**Sqoop 节点数据样例:**

{ 
                "type":"SQOOP",
                "id":"tasks-82041",
                "name":"Sqoop Task",
                "params":{ 
                    "concurrency":1,
                    "modelType":"import",
                    "sourceType":"MYSQL",
                    "targetType":"HDFS",
                    "sourceParams":"{"srcType":"MYSQL","srcDatasource":1,"srcTable":"","srcQueryType":"1","srcQuerySql":"selec id , name from user","srcColumnType":"0","srcColumns":"","srcConditionList":[],"mapColumnHive":[{"prop":"hivetype-key","direct":"IN","type":"VARCHAR","value":"hivetype-value"}],"mapColumnJava":[{"prop":"javatype-key","direct":"IN","type":"VARCHAR","value":"javatype-value"}]}",
                    "targetParams":"{"targetPath":"/user/hive/warehouse/ods.db/user","deleteTargetDir":false,"fileType":"--as-avrodatafile","compressionCodec":"snappy","fieldsTerminated":",","linesTerminated":"@"}",
                    "localParams":[
                    ]
                },
                "description":"",
                "runFlag":"NORMAL",
                "conditionResult":{ 
                    "successNode":[
                        ""
                    ],
                    "failedNode":[
                        ""
                    ]
                },
                "dependence":{ 
                },
                "maxRetryTimes":"0",
                "retryInterval":"1",
                "timeout":{ 
                    "strategy":"",
                    "interval":null,
                    "enable":false
                },
                "taskInstancePriority":"MEDIUM",
                "workerGroup":"default",
                "preTasks":[
                ]
            }

## 2.10 条件分支节点 ##

**条件分支节点数据结构如下:**

**条件分支节点数据样例:**

{ 
        "type":"CONDITIONS",
        "id":"tasks-96189",
        "name":"条件",
        "params":{ 
        },
        "description":"",
        "runFlag":"NORMAL",
        "conditionResult":{ 
            "successNode":[
                "test04"
            ],
            "failedNode":[
                "test05"
            ]
        },
        "dependence":{ 
            "relation":"AND",
            "dependTaskList":[
            ]
        },
        "maxRetryTimes":"0",
        "retryInterval":"1",
        "timeout":{ 
            "strategy":"",
            "interval":null,
            "enable":false
        },
        "taskInstancePriority":"MEDIUM",
        "workerGroup":"default",
        "preTasks":[
            "test01",
            "test02"
        ]
    }

## 2.11子流程节点 ##

**子流程节点数据结构如下:**

**子流程节点数据样例:**

{ 
                "type":"SUB_PROCESS",
                "id":"tasks-14806",
                "name":"SubProcessTask",
                "params":{ 
                    "processDefinitionId":2
                },
                "description":"",
                "runFlag":"NORMAL",
                "conditionResult":{ 
                    "successNode":[
                        ""
                    ],
                    "failedNode":[
                        ""
                    ]
                },
                "dependence":{ 
                },
                "timeout":{ 
                    "strategy":"",
                    "interval":null,
                    "enable":false
                },
                "taskInstancePriority":"MEDIUM",
                "workerGroup":"default",
                "preTasks":[
                ]
            }

## 2.12 依赖(DEPENDENT)节点 ##

**依赖(DEPENDENT)节点数据结构如下:**

<table> 
 <thead> 
  <tr> 
   <th>序号</th> 
   <th>参数名</th> 
   <th></th> 
   <th>类型</th> 
   <th>描述</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>1</td> 
   <td>id</td> 
   <td></td> 
   <td>String</td> 
   <td>任务编码</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>type</td> 
   <td></td> 
   <td>String</td> 
   <td>类型</td> 
   <td>DEPENDENT</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>name</td> 
   <td></td> 
   <td>String</td> 
   <td>名称</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>params</td> 
   <td></td> 
   <td>Object</td> 
   <td>自定义参数</td> 
   <td>Json 格式</td> 
  </tr> 
  <tr> 
   <td>5</td> 
   <td></td> 
   <td>rawScript</td> 
   <td>String</td> 
   <td>Shell脚本</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td></td> 
   <td>localParams</td> 
   <td>Array</td> 
   <td>自定义参数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>7</td> 
   <td></td> 
   <td>resourceList</td> 
   <td>Array</td> 
   <td>资源文件</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>8</td> 
   <td>description</td> 
   <td></td> 
   <td>String</td> 
   <td>描述</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td>runFlag</td> 
   <td></td> 
   <td>String</td> 
   <td>运行标识</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td>conditionResult</td> 
   <td></td> 
   <td>Object</td> 
   <td>条件分支</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>11</td> 
   <td></td> 
   <td>successNode</td> 
   <td>Array</td> 
   <td>成功跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td></td> 
   <td>failedNode</td> 
   <td>Array</td> 
   <td>失败跳转节点</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>13</td> 
   <td>dependence</td> 
   <td></td> 
   <td>Object</td> 
   <td>任务依赖</td> 
   <td>与params互斥</td> 
  </tr> 
  <tr> 
   <td>14</td> 
   <td></td> 
   <td>relation</td> 
   <td>String</td> 
   <td>关系</td> 
   <td>AND,OR</td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td></td> 
   <td>dependTaskList</td> 
   <td>Array</td> 
   <td>依赖任务清单</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>16</td> 
   <td>maxRetryTimes</td> 
   <td></td> 
   <td>String</td> 
   <td>最大重试次数</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>17</td> 
   <td>retryInterval</td> 
   <td></td> 
   <td>String</td> 
   <td>重试间隔</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>18</td> 
   <td>timeout</td> 
   <td></td> 
   <td>Object</td> 
   <td>超时控制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>19</td> 
   <td>taskInstancePriority</td> 
   <td></td> 
   <td>String</td> 
   <td>任务优先级</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>workerGroup</td> 
   <td></td> 
   <td>String</td> 
   <td>Worker 分组</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>21</td> 
   <td>preTasks</td> 
   <td></td> 
   <td>Array</td> 
   <td>前置任务</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

**依赖(DEPENDENT)节点数据样例:**

{ 
                "type":"DEPENDENT",
                "id":"tasks-57057",
                "name":"DenpendentTask",
                "params":{ 
                },
                "description":"",
                "runFlag":"NORMAL",
                "conditionResult":{ 
                    "successNode":[
                        ""
                    ],
                    "failedNode":[
                        ""
                    ]
                },
                "dependence":{ 
                    "relation":"AND",
                    "dependTaskList":[
                        { 
                            "relation":"AND",
                            "dependItemList":[
                                { 
                                    "projectId":1,
                                    "definitionId":7,
                                    "definitionList":[
                                        { 
                                            "value":8,
                                            "label":"MRTask"
                                        },
                                        { 
                                            "value":7,
                                            "label":"FlinkTask"
                                        },
                                        { 
                                            "value":6,
                                            "label":"SparkTask"
                                        },
                                        { 
                                            "value":5,
                                            "label":"SqlTask-Update"
                                        },
                                        { 
                                            "value":4,
                                            "label":"SqlTask-Query"
                                        },
                                        { 
                                            "value":3,
                                            "label":"SubProcessTask"
                                        },
                                        { 
                                            "value":2,
                                            "label":"Python Task"
                                        },
                                        { 
                                            "value":1,
                                            "label":"Shell Task"
                                        }
                                    ],
                                    "depTasks":"ALL",
                                    "cycle":"day",
                                    "dateValue":"today"
                                }
                            ]
                        },
                        { 
                            "relation":"AND",
                            "dependItemList":[
                                { 
                                    "projectId":1,
                                    "definitionId":5,
                                    "definitionList":[
                                        { 
                                            "value":8,
                                            "label":"MRTask"
                                        },
                                        { 
                                            "value":7,
                                            "label":"FlinkTask"
                                        },
                                        { 
                                            "value":6,
                                            "label":"SparkTask"
                                        },
                                        { 
                                            "value":5,
                                            "label":"SqlTask-Update"
                                        },
                                        { 
                                            "value":4,
                                            "label":"SqlTask-Query"
                                        },
                                        { 
                                            "value":3,
                                            "label":"SubProcessTask"
                                        },
                                        { 
                                            "value":2,
                                            "label":"Python Task"
                                        },
                                        { 
                                            "value":1,
                                            "label":"Shell Task"
                                        }
                                    ],
                                    "depTasks":"SqlTask-Update",
                                    "cycle":"day",
                                    "dateValue":"today"
                                }
                            ]
                        }
                    ]
                },
                "maxRetryTimes":"0",
                "retryInterval":"1",
                "timeout":{ 
                    "strategy":"",
                    "interval":null,
                    "enable":false
                },
                "taskInstancePriority":"MEDIUM",
                "workerGroup":"default",
                "preTasks":[
                ]
            }

### DolphinScheduler 社区介绍： ###

Apache DolphinScheduler 是一个非常多样化的社区，至今贡献者已近100名， 他们分别来自 30 多家不同的公司。 微信群用户3000人。

![file][]

### Apache DolphinScheduler 部分用户案例(排名不分先后) ###

已经有300多家企业和科研机构在使用DolphinScheduler，来处理各类调度和定时任务，另有500多家公司开通了海豚调度的试用：  
![file][file 1]

### Apache DolphinScheduler项目起源 - 需求决定 ###

![file][file 2]

### Apache DolphinScheduler四大特性 ###

![file][file 3]

### Apache DolphinScheduler 能力: ###

*  以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态
 *  支持丰富的任务类型：Shell、MR、Spark、Flink、SQL(mysql、postgresql、hive、sparksql)、Python、Http、Sub\_Process、Procedure等
 *  支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复，同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作
 *  支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败
 *  支持工作流全局参数及节点自定义参数设置
 *  支持资源文件的在线上传/下载，管理等，支持在线文件创建、编辑
 *  支持任务日志在线查看及滚动、在线下载日志等
 *  实现集群HA，通过Zookeeper实现Master集群和Worker集群去中心化
 *  支持对Master/Worker cpu load，memory，cpu在线查看
 *  支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计
 *  支持补数
 *  支持多租户
 *  支持国际化

### Apache DolphinScheduler 1.3 新特性 ###

* Worker实现重构，提升Worker性能
    
    * Master和Worker引入Netty通信
    
    * 去zookeeper任务队列
    
    * Worker节点的三种选择：随机、循环和CPU和内存的线性加权负载平衡
    
    * Worker去数据库操作
    
    * 资源中心支持多目录
    
    * 添加 if/else 条件任务
    
    * 添加 sqoop/datax 任务
    
    * 支持 k8s 部署
    
    * 添加DAG流程图一键格式化
    
    * 流程图美化
    
    * 支持 ambari 插件安装
    
    * 批量导出和导入工作流
    
    * 流程定义支持复制
    
    * 大幅简化配置项，简化部署

### 在线DEMO试用 ###

http://106.75.43.194:8888/

### DolphinScheduler Slogan ###

![file][file 4]

### 加入 Apache DolphinScheduler ###

在使用 DolphinScheduler 的过程中，如果您有任何问题或者想法、建议，都可以通过Apache 邮件列表参与到 DolphinScheduler 的社区建设中来。

欢迎加入贡献的队伍，加入开源社区从提交第一个 PR开始,

*  找到带有”easy to fix”标记或者一些非常简单的issue(比如拼写错误等)，先通过第一个PR熟悉提交流程，如果有任何疑问，欢迎联系  
    ![file][file 5]

[file]: https://img-blog.csdnimg.cn/20200703171028636.png
[file 1]: https://img-blog.csdnimg.cn/20200703171028958.png
[file 2]: https://img-blog.csdnimg.cn/20200703171029394.png
[file 3]: https://img-blog.csdnimg.cn/20200703171029644.png
[file 4]: https://img-blog.csdnimg.cn/20200703171029903.png
[file 5]: https://img-blog.csdnimg.cn/20200703171030236.png