log4j输出日志到flume

- 日理万妓 2022-06-04 10:26 275阅读 0赞

Log4JAppender和LoadBalancingLog4jAppender可以将应用服务器的日志通过AvroSource实时的把日志传输到日志服务器，然后在传输到监控系统或者是HDFS中存储，FLume的Log4JAppender和LoadBalancingLog4jAppender必须使用Log4j的异步加载器，否则日志服务器down机，将会导致应用服务器异常，影响线上环境的使用。 应用程序如果要用Log4JAppender就要依赖相关的jar，在应用服务器可以通过如下Maven配置获取到Log4JAppender的jar和log4j的相关jar：

<dependency>    
    <groupId>org.apache.flume.flume-ng-clients</groupId>    
    <artifactId>flume-ng-log4jappender</artifactId>    
    <version>1.6.0</version>    
</dependency>    
    
    
    
<dependency>      
    <groupId>log4j</groupId>      
    <artifactId>log4j</artifactId>      
    <version>1.2.17</version>      
</dependency>      
<dependency>      
    <groupId>org.slf4j</groupId>      
    <artifactId>slf4j-api</artifactId>      
    <version>1.7.5</version>      
</dependency>      
<dependency>      
    <groupId>org.slf4j</groupId>      
    <artifactId>slf4j-log4j12</artifactId>      
    <version>1.7.5</version>      
</dependency>

Log4jAppender和LoadBalancingLog4jAppender有如下区别：

1，LoadBalancingLog4jAppender支持一组AvroSource。  
2，LoadBalancingLog4jAppender支持round-robin，random或自定义的负载均衡方式。  
3，当Event发送失败的情况下，LoadBalancingLog4jAppender支持backoff策略。

### Log4JAppender配置文件 ###

<table style="border-collapse:collapse; border-spacing:0px; max-width:100%; color:rgb(69,69,69); font-size:16px; margin-top:10px; margin-bottom:10px; font-family:微软雅黑"> 
 <thead style="background-color:inherit"> 
  <tr style="background-color:inherit"> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 属性名</th> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 默认</th> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 描述</th> 
  </tr> 
 </thead> 
 <tbody style="background-color:inherit"> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> <span style="background-color:inherit">Hostname</span></td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 运行在远程Flume Agent中Avro Source的Hostname</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> <span style="background-color:inherit">Port</span></td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 运行在远程Flume Agent中Avro Source的监听端口</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> UnsafeMode</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> false</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 如果设置为true，那么该Appender在发送Event失败不会抛出异常。</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> AvroReflectionEnabled</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> false</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> AvroSchemaUrl</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 一个可以检索Avro schema的URL</td> 
  </tr> 
 </tbody> 
</table>

### LoadBalancingLog4jAppender配置文件 ###

<table style="border-collapse:collapse; border-spacing:0px; max-width:100%; color:rgb(69,69,69); font-size:16px; margin-top:10px; margin-bottom:10px; font-family:微软雅黑"> 
 <thead style="background-color:inherit"> 
  <tr style="background-color:inherit"> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 属性名</th> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 默认</th> 
   <th style="text-align:left; border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 描述</th> 
  </tr> 
 </thead> 
 <tbody style="background-color:inherit"> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> <span style="background-color:inherit">Hosts</span></td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 通过AvroSource监听Event的host:port列表，用空格分隔</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> Selector</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> ROUND_ROBIN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br style="background-color:inherit"> </td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 选择机制。选择机制包含：ROUND_ROBIN，RANDOM或者是自定义</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> MaxBackoff</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> MaxBackoff是一个long型值，代表一个节点消费一个event失败的情况下，负载均衡客户端最大的backoff时间（单位:毫秒），默认不启动backoff</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> UnsafeMode</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> false</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 如果设置为true，那么该Appender在发送Event失败不会抛出异常。</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> AvroReflectionEnabled</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> false</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）</td> 
  </tr> 
  <tr style="background-color:inherit"> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> AvroSchemaUrl</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> –</td> 
   <td style="border:1px solid rgb(153,153,153); padding:5px 16px 5px 12px; min-height:25px; min-width:25px; background-color:inherit"> 一个可以检索Avro schema的URL</td> 
  </tr> 
 </tbody> 
</table>

### 示例 ###

应用程序端log4j配置

一，采用Log4JAppender

1，采用properties方式：

log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender    
log4j.appender.flume.Hostname=192.68.179.160  
log4j.appender.flume.Port=5555    
log4j.appender.flume.UnsafeMode=true    
log4j.appender.flume.layout=org.apache.log4j.PatternLayout    
log4j.appender.flume.layout.ConversionPattern=%d\{HH:mm:ss.SSS\} \[%thread\] %-5level %logger\{36\} - %msg%n

二，采用LoadBalancingLog4jAppender

1，采用properties方式：

log4j.appender.flume=org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender    
log4j.appender.flume.Hosts=192.168.179.160:5555 192.168.179.161:5555 10.0.1.77:5555    
log4j.appender.flume.UnsafeMode=true    
log4j.appender.flume.MaxBackoff=30000    
log4j.appender.flume.Selector=RANDOM    
log4j.appender.flume.layout=org.apache.log4j.PatternLayout    
log4j.appender.flume.layout.ConversionPattern=%d\{HH:mm:ss.SSS\} \[%thread\] %-5level %logger\{36\} - %msg%n

服务端Flume Agent

a1.sources = r1    
a1.sinks = k1    
a1.channels = c1    
    
\# Describe/configure the source    
a1.sources.r1.type = avro    
a1.sources.r1.bind = 0.0.0.0    
a1.sources.r1.port = 5555    
    
\# Describe the sink    
a1.sinks.k1.type = file\_roll    
a1.sinks.k1.sink.directory = /data/    
a1.sinks.k1.sink.rollInterval = 0    
a1.sinks.k1.sink.serializer = TEXT    
a1.sinks.k1.sink.batchSize = 100    
    
\# Use a channel which buffers events in memory    
a1.channels.c1.type = memory    
a1.channels.c1.capacity = 1000    
a1.channels.c1.transactionCapacity = 100    
    
\# Bind the source and sink to the channel    
a1.sources.r1.channels = c1    
a1.sinks.k1.channel = c1

如果是采用LoadBalancingLog4jAppender方式，需要部署多台Flume Agent。配置类似。