SpringBoot+Logback+ELK+Kafka+Sleuth链路追踪做日志收集

心已赠人 2023-06-17 07:00 24阅读 0赞

# ELK搭建 #

[https://blog.csdn.net/qq\_37598011/article/details/103260806][https_blog.csdn.net_qq_37598011_article_details_103260806]

# 配置修改 #

**先到logstash的config目录下**

cd /application/logstash/config/

** 新建logstash-test.conf配置**

vim logstash-test.conf

input {
      kafka{
        id => "my_plugin_id"
        bootstrap_servers=>"192.168.78.133:9092"
        topics=>["logger-platform","logger-user","logger-gateway"]
      #  codec=>plain
        codec=>json_lines
        auto_offset_reset => "latest" 
      }
      #stdin { }
    }
    #filter {
    #    grok {
    #      patterns_dir => ["./patterns"]
    #        match => { "message" => "%{WORD:module} \| %{LOGBACKTIME:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}" }
    #    }   
    #}
    output {
        elasticsearch {
            #action => "index"
            hosts => "127.0.0.1:9200"
            index  => "logger-%{+YYYY-MM}"
        }
        #stdout { codec=> rubydebug }
    }

**重启**

nohub ../bin/logstash -f ./logstash-test.conf >/dev/null 2>&1 &

# **SpringBoot修改** #

### **新增pom依赖** ###

<dependency>
                <groupId>org.springframework.kafka</groupId>
                <artifactId>spring-kafka</artifactId>
                <version>2.2.0.RELEASE</version>
            </dependency>
    
            
            <dependency>
                <groupId>com.github.danielwegener</groupId>
                <artifactId>logback-kafka-appender</artifactId>
                <version>0.2.0-RC1</version>
                <scope>runtime</scope>
            </dependency>
    
            
            
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-core</artifactId>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>1.2.3</version>
                <scope>runtime</scope>
            </dependency>
            <dependency>
                <groupId>net.logstash.logback</groupId>
                <artifactId>logstash-logback-encoder</artifactId>
                <version>5.1</version>
            </dependency>
    
            
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-starter-sleuth</artifactId>
            </dependency>

### logback文件修改 ###

<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
        
        <springProperty scope="context" name="bootstrapServers" source="spring.kafka.bootstrap-servers"
                        defaultValue="localhost:9092"/>
        <springProperty scope="context" name="server-name" source="spring.application.name"/>
        <springProperty scope="context" name="ip" source="spring.cloud.client.ipAddress" />
    
        
        <appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
            <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <timestamp>
                        <timeZone>UTC</timeZone>
                    </timestamp>
                    <pattern>
                        <pattern>
                            {
                            "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ}",
                            "traceId": "%X{X-B3-TraceId:-}",
                            "parentSpanId" : "%X{X-B3-ParentSpanId:-}",
                            "spanId": "%X{X-B3-SpanId:-}",
                            "exportable": "%X{X-Span-Export:-}",
                            "pid": "${PID:-}",
                            "logger": "%logger",
                            "level": "%p",
                            "message": "%msg",
                            "host": "%property",
                            "serverName": "${server-name}",
                            "stackTrace": "%ex"
                            }
                        </pattern>
                    </pattern>
                </providers>
            </encoder>
            <topic>logger-platform</topic>
            
            <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy" />
    
            
            <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy" />
    
            
            
            
            <producerConfig>bootstrap.servers=${bootstrapServers}</producerConfig>
            
            <producerConfig>acks=0</producerConfig>
            
            <producerConfig>linger.ms=1000</producerConfig>
            
            <producerConfig>max.block.ms=0</producerConfig>
            
            <producerConfig>client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed</producerConfig>
        </appender>
    
    
    
        
        <root level="INFO">
            <appender-ref ref="kafkaAppender"/>
        </root>

### 配置application.yml ###

spring:
      kafka:
        bootstrap-servers: 192.168.78.133:9092

### ** Slf4jUtils类** ###

import org.slf4j.LoggerFactory;
    import org.slf4j.spi.LocationAwareLogger;
    import org.springframework.stereotype.Component;
    import sun.misc.JavaLangAccess;
    import sun.misc.SharedSecrets;
    
    import java.io.PrintWriter;
    import java.io.StringWriter;
    import java.text.MessageFormat;
    
    /**
     * @Classname Slf4jUtils
     * @Description TODO
     * @Date 2019/11/29 5:58
     * @Created zzf
     */
    @Component
    public class Slf4jUtils {
    
        /**
         * 空数组
         */
        private static final Object[] EMPTY_ARRAY = new Object[]{};
        /**
         * 全类名
         */
        private static final String FQCN = Slf4jUtils.class.getName();
    
        /**
         * 获取栈中类信息
         *
         * @param stackDepth
         * @return
         */
        private LocationAwareLogger getLocationAwareLogger(final int stackDepth) {
            /**通过堆栈信息获取调用当前方法的类名和方法名*/
            JavaLangAccess access = SharedSecrets.getJavaLangAccess();
            Throwable throwable = new Throwable();
            StackTraceElement frame = access.getStackTraceElement(throwable, stackDepth);
            return (LocationAwareLogger) LoggerFactory.getLogger(frame.getClassName() + "-" +
                    frame.getMethodName().split("\\$")[0] + "-" +
                    frame.getLineNumber());
        }
    
        /**
         * 封装Debug级别日志
         *
         * @param msg
         * @param arguments
         */
        public void debug(String msg, Object... arguments) {
            if (arguments != null && arguments.length > 0) {
                MessageFormat temp = new MessageFormat(msg);
                msg = temp.format(arguments);
            }
            getLocationAwareLogger(2).log(null, FQCN, LocationAwareLogger.DEBUG_INT, msg, EMPTY_ARRAY, null);
        }
    
        /**
         * 封装Info级别日志
         *
         * @param msg
         * @param arguments
         */
        public void info(String msg, Object... arguments) {
            if (arguments != null && arguments.length > 0) {
                MessageFormat temp = new MessageFormat(msg);
                msg = temp.format(arguments);
            }
            getLocationAwareLogger(2).log(null, FQCN, LocationAwareLogger.INFO_INT, msg, EMPTY_ARRAY, null);
        }
    
        /**
         * 封装Warn级别日志
         *
         * @param msg
         * @param arguments
         */
        public void warn(String msg, Object... arguments) {
            if (arguments != null && arguments.length > 0) {
                MessageFormat temp = new MessageFormat(msg);
                msg = temp.format(arguments);
            }
            getLocationAwareLogger(2).log(null, FQCN, LocationAwareLogger.WARN_INT, msg, EMPTY_ARRAY, null);
        }
    
        /**
         * 封装Error级别日志
         *
         * @param msg
         * @param arguments
         */
        public void error(String msg, Object... arguments) {
            if (arguments != null && arguments.length > 0) {
                MessageFormat temp = new MessageFormat(msg);
                msg = temp.format(arguments);
            }
            getLocationAwareLogger(2).log(null, FQCN, LocationAwareLogger.ERROR_INT, msg, EMPTY_ARRAY, null);
        }
    
        /**
         * 异常堆栈转字符串
         *
         * @param e
         * @return
         */
        public String ExceptionToString(Exception e) {
            StringWriter sw = null;
            PrintWriter pw = null;
            try {
                if (e == null) {
                    return "无具体异常信息";
                }
                sw = new StringWriter();
                pw = new PrintWriter(sw);
                e.printStackTrace(pw);
                return sw.toString();
            } catch (Exception ex) {
                this.error("异常堆栈转字符串异常", ex);
                return "";
            } finally {
                sw.flush();
                pw.flush();
                pw.close();
            }
    
        }
    }

**使用如下 **

![2019120214454473.png][]

![20191202144618979.png][]

**跟log使用方式相同！**

**启动Boot程序**

# **配置Kibana** #

**配置Kibana创建索引logger**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 1][]

**链路追踪！ **

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 2][]

**OK！可以看到日志已经输出Kafka了，并且有链路追踪功能！！！！！**

# **通过AOP切面做日志收集** #

**例子：（通过AOP切面做日志收集）**

import com.alibaba.fastjson.JSON;
    import com.alibaba.fastjson.JSONObject;
    import com.ils.intelab.common.slf4j.Slf4jUtils;
    import org.aspectj.lang.ProceedingJoinPoint;
    import org.aspectj.lang.annotation.Around;
    import org.aspectj.lang.annotation.Aspect;
    import org.aspectj.lang.annotation.Pointcut;
    import org.slf4j.MDC;
    import org.springframework.stereotype.Component;
    import org.springframework.web.context.request.RequestContextHolder;
    import org.springframework.web.context.request.ServletRequestAttributes;
    
    import javax.annotation.Resource;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;
    import java.util.ArrayList;
    import java.util.List;
    
    /**
     * @Classname LogAspect
     * @Description TODO
     * @Date 2019/12/2 15:19
     * @Created zzf
     */
    @Aspect
    @Component
    public class LogAspect {
    
        @Resource
        Slf4jUtils log;
    
        /**
         * 拦截所有控制器方法
         */
        @Pointcut("execution(public * com.CC.CCC.CCCCC.controller..*.*(..))")
        public void webLog() {
        }
    
        @Around("webLog()")
        public Object aroundMethod(ProceedingJoinPoint joinPoint) throws Exception {
            /*result为连接点的放回结果*/
            Object result = null;
            /*result为连接点的放回结果*/
            ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
            HttpServletRequest request = attributes.getRequest();
            // 记录下请求内容
            //如果是表单，参数值是普通键值对。如果是application/json否则request.getParameter是取不到的。
            String ip = request.getRemoteAddr();
            String method = joinPoint.getSignature().getDeclaringTypeName() + "." + joinPoint.getSignature().getName();
            JSONObject jsonObject = new JSONObject();
            List<Object> argList = new ArrayList<>();
            if ("application/json".equals(request.getHeader("Content-Type"))) {
                for (Object arg : joinPoint.getArgs()) {
                    // request/response无法使用toJSON
                    if (arg instanceof HttpServletRequest) {
                        argList.add("request");
                    } else if (arg instanceof HttpServletResponse) {
                        argList.add("response");
                    } else {
                        argList.add(JSONObject.toJSON(arg));
                    }
                }
            } else {
                //记录请求的键值对
                for (String key : request.getParameterMap().keySet()) {
                    JSONObject js = new JSONObject();
                    js.put(key, request.getParameter(key));
                    argList.add(js);
                }
            }
            MDC.put("url", request.getRequestURL().toString());
            MDC.put("ip", ip);
            MDC.put("method", method);
            MDC.put("arg", JSON.toJSON(argList).toString());
    
            /*执行目标方法*/
            try {
                /*返回通知方法*/
                result = joinPoint.proceed();
            } catch (Throwable e) {
                /*异常通知方法*/
                MDC.put("throwable", e.getMessage());
                StackTraceElement[] sta = e.getStackTrace();
                StringBuffer str = new StringBuffer();
                for (int i = 0; i < 15 && i < sta.length; i++) {
                    str.append(sta[i] + "\n");
                }
                log.error(str.toString());
                //抛出异常
                throw (Exception) e.fillInStackTrace();
            }
            /*后置通知*/
            jsonObject.put("message", result);
            log.info(jsonObject.toJSONString());
            return result;
        }
    
    
    }

**这里我通过MDC扩展了logback输出，所以需要在logback.xml中新增这几个，如：**

"ip": "%X{ip}",
                   "url": "%X{url}",
                   "method": "%X{method}",
                   "arg": "%X{arg}",
                   "throwable": "%X{throwable}"

**%X\{定义的字段\}**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 3][]

[https_blog.csdn.net_qq_37598011_article_details_103260806]: https://blog.csdn.net/qq_37598011/article/details/103260806
[2019120214454473.png]: https://img-blog.csdnimg.cn/2019120214454473.png
[20191202144618979.png]: https://img-blog.csdnimg.cn/20191202144618979.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191128095601939.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191202144946275.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191202144851337.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191203103045811.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70