Flume个人心得笔记

布满荆棘的人生 2021-12-16 12:25 432阅读 0赞

# 一、概述 #

1.  Flume是Apache提供的开源的、分布式的、可靠的日志收集系统
2.  能够有效的收集、聚合、传输大量的日志数据
3.  Flume有2个版本：Flume-og（Flume0.9x）和Flume-ng(Flume1.X)。Flume-og和Flume-ng不兼容

# 二.Flume的特性 #

1.  可靠性：事务型的数据传递，保证数据的可靠性。一个日志交给flume来处理，不会出现此日志丢失或未被处理的情况
2.  可恢复性：通道可以以内存或文件的方式实现，内存更快，但不可恢复。文件较慢但提供了可恢复性

## 三、基本概念 ##

1.  Event：将收集到的每一条日志封装成一个Event对象 - 在Flume中流动的是Event。Event的格式采用了json格式来定义的，这个json串中包含了2个部分：headers和body
2.  Flume中的组件是以Agent形式来出现的。Agent包含三个部分：Source、Channel和Sink
    
    1.  Source：从源头采集数据
    2.  Channel：临时存储数据
    3.  Sink：将数据发往目的地

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 2][]

# 四、自定义 --- 扩展 #

**如果Flume中提供的Source不符合当前场景，需要自定义Source。如果在Flume中自定义Source，那么需要确定Source的数据获取方式：PollableSource（主动拉取数据）和EventDrivenSource（事件驱动方式）**

## **1.创建元数据** ##

![20190705191843817.png][]

## **2.代码** ##

public class AuthSource extends AbstractSource 
    	implements Configurable, Source, EventDrivenSource {
    	private String path;
    	private ExecutorService es;
    
    	// 获取配置文件中的属性
    	@Override
    	public void configure(Context context) {
    		path = context.getString("file");
    	}
    	// 启动Source
    	@Override
    	public synchronized void start() {
    		// 获取Channel
    		ChannelProcessor cp = this.getChannelProcessor();
    		es = Executors.newFixedThreadPool(10);
    		es.submit(new FileThread(path, cp));
    	}
    	@Override
    	public synchronized void stop() {
    		es.shutdown();
    	}
    }
    class FileThread implements Runnable {
    	private BufferedReader br;
    	private ChannelProcessor cp;
    	public FileThread(String path, ChannelProcessor cp) {
    		this.cp = cp;
    		File file = new File(path);
    		if (file.isFile())
    			try {
    				br = new BufferedReader(new FileReader(path));
    			} catch (FileNotFoundException e) {
    				e.printStackTrace();
    			}
    	}
    	@Override
    	public void run() {
    		while (true) {
    			try {
    				// 读取数据
    				String name = br.readLine();
    				if (name == null)
    					break;
    				String age = br.readLine();
    				String description = br.readLine();
    				// 定义头部信息
    				Map<String, String> headers = new HashMap<>();
    				headers.put("name", name);
    				headers.put("age", age);
    				// 创建一个Event对象
    				Event e = EventBuilder.withBody(description.getBytes(), headers);
    				// 将event放到通道中
    				cp.processEvent(e);
    			} catch (IOException e) {
    				e.printStackTrace();
    			}
    		}
    		try {
    			br.close();
    		} catch (IOException e) {
    			e.printStackTrace();
    		}
    	}
    }

## 3.打成jar包上传到flume的lib目录下 ##

**auth.jar**

![20190705191507400.png][]

## 4.编辑一个文件 更新配置 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 3][]

## 5.测试 ##

![2019070519155789.png][]

![20190705191754544.png][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20190705192104273.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 1]: /images/20211214/3b23946a1b2a4843b39c3b81363b5025.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 2]: /images/20211214/514d1eced59c4ca49ef2468da8063e6a.png
[20190705191843817.png]: /images/20211214/b7682faef87649d88b2bd084e2f439d3.png
[20190705191507400.png]: /images/20211214/56610799e42648a5b62647cbb3c8a0b0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 3]: /images/20211214/72e9c06890134adeb52b0921542c86fc.png
[2019070519155789.png]: /images/20211214/64893911d6a34f2ba0b3b9faa24bacd8.png
[20190705191754544.png]: /images/20211214/7ecb4f0317a94ee2bb02df330bb786d5.png