2.5.4 hadoop体系之离线计算-Flume日志收集系统-Agent级联

清疚 2022-12-19 03:58 124阅读 0赞

**目录**

1.需求分析

2.功能实现

2.1 Node02安装Flume

2.2 开发脚本向文件中写入数据

2.3 node03中Flume配置文件

2.4 顺序启动

--------------------

# 1.需求分析 #

*  第一个agent负责收集文件当中的数据，通过网络发送到第二个agent当中去
 *  第二个agent负责接收第一个agent发送的数据，并将数据保存到hdfs上面去

# 2.功能实现 #

## 2.1 Node02安装Flume ##

我们设计让node02（avro）搜集数据给node03，首先将node03机器上面解压后的flume文件夹拷贝到node02机器上面去

cd /export/servers/ apache-flume-1.8.0-bin/conf

vim tail-avro-avro-logger.conf

##################
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /export/servers/taillogs/access_log
    a1.sources.r1.channels = c1
    # Describe the sink
    ##sink端的avro是一个数据发送者
    a1.sinks = k1
    a1.sinks.k1.type = avro
    a1.sinks.k1.channel = c1
    a1.sinks.k1.hostname = 192.168.174.120
    a1.sinks.k1.port = 4141
    a1.sinks.k1.batch-size = 10
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

## 2.2 开发脚本向文件中写入数据 ##

直接将node03下面的脚本和数据拷贝到node02即可，node03机器上执行以下命令

cd /export/servers

scp -r shells/ taillogs/ node02:$PWD

## 2.3 node03中Flume配置文件 ##

在node03机器上开发flume的配置文件

cd /export/servers/apache-flume-1.8.0-bin/conf

vim avro-hdfs.conf

# Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    # Describe/configure the source
    ##source中的avro组件是一个接收者服务
    a1.sources.r1.type = avro
    a1.sources.r1.channels = c1
    a1.sources.r1.bind = 192.168.174.120
    a1.sources.r1.port = 4141
    # Describe the sink
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = hdfs://node01:8020/av /%y-%m-%d/%H%M/
    a1.sinks.k1.hdfs.filePrefix = events
    a1.sinks.k1.hdfs.round = true
    a1.sinks.k1.hdfs.roundValue = 10
    a1.sinks.k1.hdfs.roundUnit = minute
    a1.sinks.k1.hdfs.rollInterval = 3
    a1.sinks.k1.hdfs.rollSize = 20
    a1.sinks.k1.hdfs.rollCount = 5
    a1.sinks.k1.hdfs.batchSize = 1
    a1.sinks.k1.hdfs.useLocalTimeStamp = true
    #生成的文件类型，默认是Sequencefile，可用DataStream，则为普通文本
    a1.sinks.k1.hdfs.fileType = DataStream
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

## 2.4 顺序启动 ##

node03机器启动Flume进程

![20201105145151193.png][]

node02机器启动Flume进程

![20201105145159570.png][]

node02机器启shell脚本生成文件

cd /export/servers/shells

sh tail-file.sh

![20201105145412400.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70][]

[20201105145151193.png]: /images/20221120/72209f6506d242ffa63fd65325b1b4ce.png
[20201105145159570.png]: https://img-blog.csdnimg.cn/20201105145159570.png
[20201105145412400.png]: https://img-blog.csdnimg.cn/20201105145412400.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20201105145437506.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU=,size_16,color_FFFFFF,t_70