logstash通过grok插件匹配springBoot默认日志

た入场券 2023-02-12 06:25 97阅读 0赞

**背景：在 [Logstash数据源为日志文件操作][Logstash] 基础上进行**

### 一、修改配置 ###

**1. logstash.conf文件修改为如下**

input { 
        stdin { } 
        file {
            # 容器中日志所在目录的文件
            path => ["/usr/share/logstash/logs/*.log"]
            # 多行匹配方法1
            codec => multiline {
                pattern => "^(%{TIMESTAMP_ISO8601})"
                negate => true
                what => "previous"
            }
            sincedb_path => "NUL"
            type => "spring"
            start_position => "beginning"
        }
    }
    
    filter {
        if [type] == "spring" {
            # 多行匹配方法2
           # multiline {
                # pattern => "^(%{TIMESTAMP_ISO8601})"
                # negate => true
                # what => "previous"
           # }
            grok {
                # Do multiline matching with (?m) as the above mutliline filter may add newlines to the log messages.
                match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{NUMBER:pid}%{SPACE}---%{SPACE}%{SYSLOG5424SD:threadName}%{SPACE}%{NOTSPACE:loggerName}%{SPACE}:%{SPACE}%{GREEDYDATA:message}" ]
                # 覆盖原有的message
                overwrite=> [ "message" ]
            }
       }
    }
    
    
    output {
        if [type] == "spring" {
            elasticsearch {
                hosts => ["es:9200"]
                index => "spring-%{+YYYY.MM.dd}"
            }
        }
    	stdout { codec => rubydebug }
    }

### 二、模拟日志 ###

**1. D:\\usr\\local\\logs目录下添加日志文件spring-2020-05-25.log，内容如下**

2020-05-15 17:54:50.805 DEBUG 8296 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-15 17:54:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-21 19:55:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-21 19:55:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-21 19:55:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-21 19:55:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-21 19:55:52.945 DEBUG 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-22 10:15:12.945 INFO 1012 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select id from user
    2020-05-18 03:39:31.174 ERROR 9 --- [http-nio-8002-exec-6] org.skr.security.JwtAuthenticationToken  : org.skr.common.exception.AuthException: 无法找到有效的登录状态
    	at org.skr.security.JwtAuthenticationToken.authenticate(JwtAuthenticationToken.java:64)
    	at org.skr.security.JwtAuthenticationFilter.doFilterInternal(JwtAuthenticationFilter.java:45)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.skr.security.JwtAuthExceptionFilter.doFilterInternal(JwtAuthExceptionFilter.java:42)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)
    	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
    	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
    	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
    	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
    	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
    	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
    	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
    	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    	at java.base/java.lang.Thread.run(Thread.java:830)
    
    2020-05-18 03:45:00.001 DEBUG 9 --- [scheduling-1] org.jooq.tools.LoggerListener            : Executing query          : select "butler"."rb_deleted_data"."id", "butler"."rb_deleted_data"."uid", "butler"."rb_deleted_data"."created_at", "butler"."rb_deleted_data"."created_by", "butler"."rb_deleted_data"."updated_at", "butler"."rb_deleted_data"."updated_by", "butler"."rb_deleted_data"."ver", "butler"."rb_deleted_data"."realm_code", "butler"."rb_deleted_data"."model_name", "butler"."rb_deleted_data"."data_id", "butler"."rb_deleted_data"."schema", "butler"."rb_deleted_data"."table_name", "butler"."rb_deleted_data"."data_key", "butler"."rb_deleted_data"."data_summary", "butler"."rb_deleted_data"."scheduled_deleting_date", "butler"."rb_deleted_data"."reference_info" from "butler"."rb_deleted_data" where ("butler"."rb_deleted_data"."scheduled_deleting_date" is not null and "butler"."rb_deleted_data"."scheduled_deleting_date" < cast(? as timestamp) and "butler"."rb_deleted_data"."id" > ?) order by "butler"."rb_deleted_data"."id" asc limit ?

### 三、重启Logstash ###

**1. 重启logstash**

docker restart logstash

![20200525093323708.png][]

### 四、查看结果 ###

**1. 打开浏览器，查看请求[http://localhost:5601/][http_localhost_5601]**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70][]

**从以上结果可以看出匹配成功**

[Logstash]: https://blog.csdn.net/zsx18273117003/article/details/106116428
[20200525093323708.png]: https://img-blog.csdnimg.cn/20200525093323708.png
[http_localhost_5601]: http://localhost:5601/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200525093609178.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz,size_16,color_FFFFFF,t_70