大数据Flume系列之Flume集群搭建

柔情只为你懂 2022-05-15 12:18 481阅读 0赞

# 1. 概念 #

集群的意思是多台机器，最少有2台机器，一台机器从数据源中获取数据，将数据传送到另一台机器上，然后输出。接下来就要实现Flume集群搭建。集群如下图所示。

![E6_9E_B6_E6_9E_84.jpg][]

# 2. Flume搭建 #

## 2.1 部署准备 ##

*  **部署主机**

192.168.9.139    host14 
    192.168.9.128    host15

*  **host14主机下载flume软件包**

# cd /opt/tools
    # wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

*  **上传解压flume**

# mkdir -p /apps/svr/flume/
    # tar -zxvf /opt/tools/apache-flume-1.7.0-bin.tar.gz -C /apps/svr/flume/

## 2.2 部署Flume ##

部署的是集群，需要在2台机安装Flume，host14作为push推送数据，host15作为pull获取数据后显示出来。

*  **修改配置文件**

# cd /apps/svr/flume/apache-flume-1.7.0-bin/conf/
    # cp flume-env.sh.template flume-env.sh
    # cp flume-conf.properties.template flume-telent.conf
    
    # vim flume-env.sh
    export JAVA_HOME=/apps/svr/java/jdk1.8.0_172

*  **host15主机部署Flume**

# scp -r /apps/svr/flume/ host15:/apps/svr/

*  **验证flume**

# /apps/svr/flume/apache-flume-1.7.0-bin/bin/flume-ng version
    Flume 1.7.0
    Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
    Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
    Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
    From source with checksum 0d21b3ffdc55a07e1d08875872c00523

## 2.3 Flume集群配置 ##

*  **配置push.conf **

\[host14\]

\# cd /apps/svr/flume/apache-flume-1.7.0-bin/conf  
\# vim push.conf

# Name the components on this agent
    a2.sources= r1
    a2.sinks= k1
    a2.channels= c1
     
    # Describe/configure the source
    a2.sources.r1.type= spooldir
    a2.sources.r1.spoolDir= /apps/svr/flume/logs
    a2.sources.r1.channels= c1
     
    # Use a channel which buffers events in memory
    a2.channels.c1.type= memory
    a2.channels.c1.keep-alive= 10
    a2.channels.c1.capacity= 100000
    a2.channels.c1.transactionCapacity= 100000
     
    # Describe/configure the source
    a2.sinks.k1.type= avro
    a2.sinks.k1.channel= c1
    a2.sinks.k1.hostname= host15
    a2.sinks.k1.port= 8899

*  **配置pull.conf**

\[host15\]  
\# cd /apps/svr/flume/apache-flume-1.7.0-bin/conf  
\# vim pull.conf

# Name the components on this agent
    a1.sources= r1
    a1.sinks= k1
    a1.channels= c1
     
    # Describe/configure the source
    a1.sources.r1.type= avro
    a1.sources.r1.channels= c1
    a1.sources.r1.bind= host15
    a1.sources.r1.port= 8899
     
    # Describe the sink
    a1.sinks.k1.type= logger
    a1.sinks.k1.channel = c1
     
    # Use a channel which buffers events in memory
    a1.channels.c1.type= memory
    a1.channels.c1.keep-alive= 10
    a1.channels.c1.capacity= 100000
    a1.channels.c1.transactionCapacity= 100000

*  **创建spoolDir目录**

\[host14\]  
\# mkdir -p /apps/svr/flume/logs

## 2.4 Flume集群启动 ##

*  **启动pull主机**

\[host15\]  
\# cd /apps/svr/flume/apache-flume-1.7.0-bin/  
\# ./bin/flume-ng agent -c conf -f conf/pull.conf -n a1 -Dflume.root.logger=INFO,console

显示如图所示则为启动成功

![70][]

*  **启动push主机**

\[host14\]  
\# cd /apps/svr/flume/apache-flume-1.7.0-bin/  
\# ./bin/flume-ng agent -n a2 -c conf -f conf/push.conf -Dflume.root.logger=INFO,console

显示如图所示则为启动成功

![70 1][]

*  **验证连接**

\[host15\]

显示如图所示表示连接成功

![70 2][]

# 3. Flume测试 #

## 3.1 创建测试用例 ##

\[host14\]  
\# cd /apps/svr/flume/logs/  
\# vim flume-use-case-test.log

HELLO WORLD!!!
    HELLO FLUME!!!

## 3.2 验证测试 ##

*  **pull主机**

显示如图所示表示测试成功

![70 3][]

*  **push主机**

显示如图所示表示测试成功

![70 4][]

结论：用例测试成功，证明Flume集群搭建成功。

原文地址：[https://1csh1.github.io/2016/04/21/Flume%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/][https_1csh1.github.io_2016_04_21_Flume_E9_9B_86_E7_BE_A4_E6_90_AD_E5_BB_BA]

[E6_9E_B6_E6_9E_84.jpg]: /images/20220515/b29132bbe962423ab50df8c1df7a9465.png
[70]: /images/20220515/e3131bff73bf4af8a105eb0383fdc4d9.png
[70 1]: /images/20220515/054535d01c4d4d64b3a514a8b2026d77.png
[70 2]: /images/20220515/a6ca5bc1cc4b43c0a132b1cb7a641826.png
[70 3]: /images/20220515/878810c7722244f9bfdb3c4e4bb55ec4.png
[70 4]: /images/20220515/d616e728662a4bbf81bb787e921e10dc.png
[https_1csh1.github.io_2016_04_21_Flume_E9_9B_86_E7_BE_A4_E6_90_AD_E5_BB_BA]: https://1csh1.github.io/2016/04/21/Flume%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/