[Hadoop] Azkaban 安装插件

系统管理员 2022-05-11 06:58 309阅读 0赞

前面我们完成了[Azkaban-3.58.0的源码编译和multiple-executor mode部署][Azkaban-3.58.0_multiple-executor mode]，现在来安装hdfs-viewer插件。

## 1. 拷贝hdfs-viewer到目标主机 ##

hdfs-viewer插件是安装在azkaban-web-server端的，首先去目标主机的$AK\_HOME/下新建plugins/viewer文件夹，然后解压编译好的将编译好的hdfs-viewer插件压缩包到plugins/viewer文件夹下，并重命名为hdfs。

# 新建plugins/viewer文件夹
    [root@vm azkaban-web-server]# mkdir plugins
    [root@vm azkaban-web-server]# mkdir plugins/viewer
    
    # 解压编译好的az-hdfs-viewer-xxx.tar.gz压缩包到viewer文件夹下，重命名成hdfs
    [root@vm azkaban-web-server]# cd /data/software/azkaban/
    [root@vm azkaban]# tar -zxvf az-hdfs-viewer-3.58.0-10-g609d4d8.tar.gz -C $AK_HOME/plugins/viewer/
    [root@vm azkaban]# cd $AK_HOME/plugins/viewer
    [root@vm viewer]# mv az-hdfs-viewer-3.58.0-10-g609d4d8/ hdfs

修改azkaban-web-server配置文件

[root@g12-1 azkaban-web-server]# vi conf/azkaban.properties 
    viewer.plugins=hdfs    # 这句话会告诉azkaban去$AK_HOME/plugins/viewer/hdfs路径下去寻找新的插件

拷贝hadoop的相关jar包到$AK\_HOME/extlib/路径下

[root@vm azkaban-web-server]# mkdir plugins/viewer/hdfs/extlib
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/commons-cli-1.2.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/commons-configuration-1.6.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-auth-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-common-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-core-2.6.0-mr1-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-hdfs-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-common-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-yarn-api-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-yarn-common-2.6.0-cdh5.13.1.jar plugins/viewer/hdfs/extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/htrace-core-3.0.4.jar plugins/viewer/hdfs/extlib/      
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/protobuf-java-2.5.0.jar plugins/viewer/hdfs/extlib/
    
    
    
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/commons-cli-1.2.jar extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-auth-2.6.0-cdh5.13.1.jar extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-common-2.6.0-cdh5.13.1.jar extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/hadoop-hdfs-2.6.0-cdh5.13.1.jar extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/protobuf-java-2.5.0.jar extlib/  
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/htrace-core-3.0.4.jar extlib/
    [root@vm azkaban-web-server]# cp /opt/cloudera/parcels/CDH/jars/htrace-core4-4.0.1-incubating.jar plugins/viewer/hdfs/extlib/

安装Job Type Plugins

这是一个executor插件，它可以为为hadoop生态圈的一些组件提供支持。

解压编译好的jobtypes安装包到executor-server的$AK\_HOME/下新建plugins/viewer文件夹

修改配置文件

azkaban.jobtype.plugin.dir=plugins/jobtypes

TO BE UPDATE LATER ON ...

[Azkaban-3.58.0_multiple-executor mode]: https://blog.csdn.net/wawa8899/article/details/82761913