filebeat+logstash对springcloud项目日志进行解析

我就是我 2022-05-31 10:53 279阅读 0赞

项目地址：[https://gitee.com/lwydyby/springcloud-adplatform][https_gitee.com_lwydyby_springcloud-adplatform]  
该项目的日志目前可以分为三种：  
(1)项目启动日志  
(2)controller层的操作日志（通过aop实现记录）  
(3)项目非controller层异常日志  
由于分为3种格式，则需要三种macth进行匹配：

grok {       
            match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}",
            "message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)",
             "message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]
        }

使用logstash自带表达式的格式一般为：%\{SYNTAX:SEMANTIC\}  
即： SYNTAX`代表匹配值的类型，例如，`0.11`可以`NUMBER`类型所匹配，`10.222.22.25`可以使用`IP`匹配。 *`SEMANTIC`表示存储该值的一个变量声明，`client\_ip\_address\`这个字段里边，类似数据库的列名  
具体可参考：[http://blog.csdn.net/liukuan73/article/details/52318243][http_blog.csdn.net_liukuan73_article_details_52318243]  
而自定义匹配需要使用正则表达式，格式为

( ?<列名>正则表达式)

注：\[\\s\\S\]\*为匹配剩余所有的字符，我这里是用来匹配错误日志的详细信息的  
由于controller层日志是以json格式进行的存储，除使用grok之外还需使用 json进行解析

json {  
            source => "log_json"  
            target => "content"  
            remove_field=>["logjson"]  
         }

完整配置文件为：

input {
      beats {
        port => 5044
      }
    }
    filter{
        grok {       
            match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}",
            "message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)",
             "message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]
        }   
         json {  
            source => "log_json"  
            target => "content"  
            remove_field=>["logjson"]  
         }  
    }
    output {
      elasticsearch {
        hosts => ["localhost"]
        manage_template => false
        index => "%{[fields][logIndex]}-%{+YYYY.MM.dd}"
        document_type => "%{[fields][docType]}"
      }
    }

除需要配置logstash之外，还需要对filebeat进行配置，以设置如何发送日志（处理多行日志）：

filebeat:
      prospectors:
      -
        paths:
          - "/data/logs/*.log"
        fields:
          logIndex: userservice
          docType: springcloud
          project: adplatform
        multiline:
          pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
          negate: true
          match: after
          timeout: 5s
    output:
      logstash:
        hosts: ["localhost:5044"]

日志文件样例：

2018-02-22 16:09:59.876 [main] INFO  o.s.integration.channel.PublishSubscribeChannel - Channel 'user-service:dev:8763.errorChannel' has 1 subscriber(s).
    2018-02-22 16:09:59.876 [main] INFO  o.s.integration.endpoint.EventDrivenConsumer - started _org.springframework.integration.errorLogger
    2018-02-22 16:09:59.913 [main] INFO  o.s.i.endpoint.SourcePollingChannelAdapter - started sleuthStreamSpanReporter.poll.inboundChannelAdapter
    2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147482647
    2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147483647
    2018-02-22 16:09:59.966 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8763"]
    2018-02-22 16:09:59.991 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8763"]
    2018-02-22 16:09:59.992 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
    2018-02-22 16:10:00.081 [main] INFO  o.s.b.c.e.tomcat.TomcatEmbeddedServletContainer - Tomcat started on port(s): 8763 (http)
    2018-02-22 16:10:00.082 [main] INFO  o.s.c.n.e.s.EurekaAutoServiceRegistration - Updating port to 8763
    2018-02-22 16:10:00.088 [main] INFO  com.gameley.UserServiceApplication - Started UserServiceApplication in 281.136 seconds (JVM running for 285.806)
    2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  o.a.c.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring FrameworkServlet 'dispatcherServlet'
    2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization started
    2018-02-22 16:10:13.301 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization completed in 67 ms
    2018-02-22 16:12:26.850 [http-nio-8763-exec-10] INFO  com.gameley.common.Interceptor.HttpAspect - {
       "address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":1041}
    2018-02-22 16:12:27.971 [task-scheduler-5] INFO  o.s.integration.codec.kryo.CompositeKryoRegistrar - registering [40, java.io.File] with serializer org.springframework.integration.codec.kryo.FileSerializer
    2018-02-22 16:12:28.276 [http-nio-8763-exec-9] INFO  com.gameley.common.Interceptor.HttpAspect - {
       "address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":7}

[https_gitee.com_lwydyby_springcloud-adplatform]: https://gitee.com/lwydyby/springcloud-adplatform
[http_blog.csdn.net_liukuan73_article_details_52318243]: http://blog.csdn.net/liukuan73/article/details/52318243