Flume推送数据到DataHub

向右看齐 2022-11-14 14:39 354阅读 0赞

# Flume推送数据到DataHub #

## 一、Flume-DataHub插件安装 ##

Flume默认是不支持DataHub的，所以要给Flume安装DataHub的Sink插件

1.  首先在Flume安装目录建立插件文件夹

[hadoop@hadoop102 flume]$ mkdir plugins.d

1.  利用SecureCRT工具把aliyun-flume-datahub-sink-2.0.2.tar.gz拷贝到该/opt/module/flume/plugins.d目录下，并原地解压缩

[hadoop@hadoop102 plugins.d]$ ls
    aliyun-flume-datahub-sink-2.0.2.tar.gz
    [hadoop@hadoop102 plugins.d]$ tar -zxvf aliyun-flume-datahub-sink-2.0.2.tar.gz

## 二、广告基础表Flume配置文件修改 ##

[hadoop@hadoop102 conf]$ vim /opt/module/flume/jobs/base_ad_log-file-flume-datahub.conf

# 定义组件名称
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    ### source 部分
    a1.sources.r1.type = TAILDIR
    a1.sources.r1.positionFile = /opt/module/flume/test/base_ad_log_taildir_position.json
    a1.sources.r1.channels = c1
    a1.sources.r1.filegroups = f1
    a1.sources.r1.filegroups.f1 = /opt/module/logs/baseadlog.+
    a1.sources.r1.fileHeader = true
    
    ### channel 部分
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 1000
    
    ### sink 部分 
    a1.sinks.k1.type = com.aliyun.datahub.flume.sink.DatahubSink
    a1.sinks.k1.datahub.accessID = LTAI4G8mugWrSrSynde2xQ8X
    a1.sinks.k1.datahub.accessKey = MMnDsQFGYbW2fJeBiis5sdg0UWMy2M
    a1.sinks.k1.datahub.endPoint = http://dh-cn-shenzhen-int-vpc.aliyuncs.com
    a1.sinks.k1.datahub.project = education_online_datahub
    a1.sinks.k1.datahub.topic = base_ad_log
    a1.sinks.k1.batchSize = 100
    a1.sinks.k1.serializer = DELIMITED
    a1.sinks.k1.serializer.delimiter = "\\u007C"
    a1.sinks.k1.serializer.fieldnames = logstring
    a1.sinks.k1.serializer.charset = UTF-8
    a1.sinks.k1.shard.number = 1
    a1.sinks.k1.shard.maxTimeOut = 60
    
    # 把source和sink绑定到channel中
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

说明：

<table> 
 <thead> 
  <tr> 
   <th>选择参数</th> 
   <th>含义</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>type</td> 
   <td>必须是：com.aliyun.datahub.flume.sink.DatahubSink</td> 
  </tr> 
  <tr> 
   <td>datahub.accessID</td> 
   <td>accessID 下面小节会介绍</td> 
  </tr> 
  <tr> 
   <td>datahub.accessKey</td> 
   <td>accessKey</td> 
  </tr> 
  <tr> 
   <td>datahub.endPoint</td> 
   <td>不同的地区有不同的接入点，参考https://help.aliyun.com/document_detail/47442.html</td> 
  </tr> 
  <tr> 
   <td>project</td> 
   <td>datahub中创建的project名</td> 
  </tr> 
  <tr> 
   <td>topic</td> 
   <td>datahub中project下的topic</td> 
  </tr> 
  <tr> 
   <td>batchSize</td> 
   <td>每个批次多少条记录</td> 
  </tr> 
  <tr> 
   <td>serializer</td> 
   <td>序列化方式，DELIMITED 表示按分隔符切分</td> 
  </tr> 
  <tr> 
   <td>serializer.delimiter</td> 
   <td>具体分隔符是什么，这里“\u007C” 表示 \</td> 
  </tr> 
  <tr> 
   <td>http://www.msxindl.com/tools/unicode16.asp</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>serializer.fieldnames</td> 
   <td>切分后对应的字段名</td> 
  </tr> 
  <tr> 
   <td>serializer.charset</td> 
   <td>字符集</td> 
  </tr> 
  <tr> 
   <td>shard.number</td> 
   <td>分片数</td> 
  </tr> 
  <tr> 
   <td>shard.maxTimeOut</td> 
   <td>超时时间，单位是秒</td> 
  </tr> 
 </tbody> 
</table>

## 三、获取AccessID和AccessKey ##

DataHub服务并不是靠IP来定位的，而是靠阿里云账号，每个阿里云账号只能有一个DataHub，每个阿里云账号也会有唯一的AccessID和AccessKey。所以通过AccessId和AccessKey就可以直接锁定某个阿里云账号的DataHub。

1.  悬浮鼠标到阿里云账号头像上->点击accesskeys  
    ![在这里插入图片描述][20210329221849731.png]
2.  点击继续使用AccessKey  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70]
3.  新用户需要点击创建AccessKey  
    ![在这里插入图片描述][20210329221915454.png]
4.  获取到AccessKeyID和AccessKeySecret值  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70 1]

## 四、查看接收数据 ##

1）启动Flume进程

[hadoop@hadoop102 flume]$ /opt/module/flume/bin/flume-ng agent -n a1 -c /opt/module/flume/conf/ 
     -f /opt/module/flume/jobs/base_ad_log-file-flume-datahub.conf -Dflume.root.logger=info,console

2）观察DataHub中数据量  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70 2]

可以去DataHub后台看到对应的数据已经进入主题。

[20210329221849731.png]: /images/20221022/6df65b8b7139427ebd045c33c380169c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70]: /images/20221022/d6dfb44df50c4c06aa832b1080133c4f.png
[20210329221915454.png]: /images/20221022/805fdbd2a9294a38b29826d58e1af8b6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70 1]: /images/20221022/c9f5da5bb2d64273a22b9e746149d214.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjc5NjQwMw_size_16_color_FFFFFF_t_70 2]: /images/20221022/ad4e101992ab42eb9c52b417c7bd1740.png