日志读取分析算法---大结局

青旅半醒 2022-05-28 22:23 158阅读 0赞

日志读取分析算法

## 一,【前言】 ##

在所有业务系统中,不论是开发,部署,运维为了更好的发现问题,解决问题,我们都会首先从日志出发.

但是有的日志我们是看不到的, 有的日志是会被系统本身吃掉,所以对于整个业务组,部署,运维来说,如果有一个日志系统专门来优化架构,优化业务系统就能大大的提升效率了.

那业务系统中怎么读取日志呢?

## 二,【详情】 ##

###  1,读取方式: ###

(1)读一行

(2)读所有

(3)分级别读

,,,,

### 2,解决方案: ###

![Image 1][]

### 3,实现代码: ###

//日志信息分享读取算法---2018-03-08
    
    publicvoid parseLog(StringfileNamePath, List<Event> eventList) {
        do {
            long position = positionService.getNowDayPosition(fileNamePath);
            log.info("=parseLog=> fileNamePath={},position={} .", fileNamePath, position);
            if (null != fileNamePath) {
                RandomAccessFile in = null;
                try {
                    in = new RandomAccessFile(fileNamePath, "r");
                    LogLines lines = new LogLines();
                    if (in.length() >= position) {
                        in.seek(position);
                    }
                    currentPosition = 0;
                    for (String line = in.readLine(); currentPosition < in.length(); line = in.readLine()) {
    
                    //如果读到日志为null  或有时间，或大于最大数
                        if (null == line || line.startsWith(String.valueOf(DateUtil.getNowYear()))|| lines.size() > MAX_EXCEPTION_LINES) {
    
                          //如果日志不为空
                            if (!lines.isEmpty()) {
    
                           // 收集全量或异常
                                if (!isErrorParse || (isErrorParse && lines.isException)) {
                                   StringBuilder msg = new StringBuilder();
                                    for (String le : lines) {
                                       msg.append(le).append("\n");
                                    }
                                    LogEntity logEntity =createLog(fileNamePath, msg.toString(), in.getFilePointer(), LogLevel.NULL);
                                    if (null != logEntity) {
                                        synchronized (eventList) {
                                            eventList.add(EventBuilder.withBody(logEntity.toJson().getBytes(charset)));
                                            if (eventList.size() >= batchSize || timeout()) {
                                               flushEventBatch(eventList);
                                            }
                                        }
                                    }
                                   lines.clear();
                                }
                            }
                        }
                        lines.add(line);
                        currentPosition = in.getFilePointer();
                    }
                    if (currentPosition > 0) {
                        positionService.saveCurrentPosition(fileNamePath, currentPosition);
                    }
    
                    synchronized (eventList) {
                        if (!eventList.isEmpty()) {
                           flushEventBatch(eventList);
                        }
                    }
                } catch (Exception e) {
                    log.error("=parseLog=> read file " + fileNamePath + " error ", e);
                    if (currentPosition > 0) {
                        positionService.saveCurrentPosition(fileNamePath, currentPosition);
                    }
                } finally {
                    if (null != in) {
                        try {
                            in.close();
                        } catch (IOException e) {
                            log.error("=parseLog=> close file " + fileNamePath + " error ", e);
                        }
                    }
                }
            } else {
                log.info("=parseLog=> no file path to analysis.");
            }
            if (restart) {
                log.info("=parseLog=>sleep {} ms.", restartSleepTime);
                try {
                    Thread.sleep(restartSleepTime);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            } else {
                log.info("<=parseLog=>thread is over.");
            }
        } while (restart);
    }

//集合----2018-03-08
    
    
            public static class LogLines extends ArrayList<String> {
                private boolean isException = false;
    //            private int startLinenum;
    
                public boolean add(String line) {
                    boolean add = super.add(line);
                    if (!isException && exceptionPattern.matcher(line).matches()) {
                        isException = true;
                    }
    //                if (isException &&startException.matcher(line).matches()) {
    //                    startLinenum++;
    //                }
                    return add;
                }
    
                public void clear() {
                    super.clear();
                    isException = false;
                }
            }

--------------------

publicstatic void main(Stringargs[]) {
        ObjectMapper mapper = new ObjectMapper();
        String msg = "[]{}']\f\r\batorg.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)" +
                "\natorg.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)";
            msg = msg.replaceAll("[\\\b\t\f\r\"\'\\{\\}\\[\\]]", "");
        System.out.println(msg);
        msg = msg.replaceAll("\n", "\\\\n");
        System.out.println(msg);
    }

## 三,【小结】                                           ##

话说:难者不会,会者不难! 一开始我对这个逻辑是分析的非常的清楚和明白的,但是在落地实现的时候,还是花了不少时间, 后面同架构师我们一起不到15mins 就完成了, 对人家佩服的五体投地呀!这就是巨人, 自己要不断的学习, 不断的提升. 要坚信: 你不是在为公司打工, 你是在为自己的生命打工!

让自己进步, 让自己提升才能终生成长!

![Image 1][]

[Image 1]: