Logstash数据源为日志文件操作

柔光的暖阳◎ 2023-03-13 13:50 6阅读 0赞

在[Springboot整合ELK][Springboot_ELK]基础上进行

### 一、模拟日志 ###

**1. 在D:\\usr\\local\\bin\\start目录下编写python脚本save\_log.py**

#!/usr/bin/python
    # -*- coding: UTF-8 -*-
    import datetime
    import threading
    
    import os
    
    # 递归级联创建的目录
    path = "D:/usr/local/logs"
    if not os.path.exists(path):
        os.makedirs(path)
    
    # 方法1： 此方法文件夹不存在会报错：FileNotFoundError: [Errno 2] No such file or directory
    with open('D:/usr/local/logs/spring.log', mode='a+', encoding='utf-8') as f:
        now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
        thread_id = threading.currentThread().native_id
        log_str = f'{now} DEBUG {thread_id} --- [scheduling-1] org.jooq.tools.LoggerListener            : ' \
                  'Executing query          : select id from user'
        f.writelines(f'{log_str}\n')
        f.close()
        print('保存日志成功')

**2.  切换到D:\\usr\\local\\bin\\start目录，并执行脚本**

cd D:/usr/local/bin/start
    python3 save_log.py

![20200514133830788.png][]

**3. 查看结果**

![20200514133953266.png][]

### 二、修改Logstash配置 ###

**1. 修改D:\\usr\\local\\etc\\logstash\\pipeline目录下logstash.conf内容，新增文件数据源，修改后的内容为**

input { 
        stdin { } 
        tcp {
            mode => "server"
            host => "0.0.0.0"
            # 从5044端口取日志
            port => 5044
            # 需要安装logstash-codec-json_lines插件
            codec => json_lines
            type => "tcp"
        }
        file {
            # 容器中日志所在目录的文件
            path => ["/usr/share/logstash/logs/*.log"]
            #codec => "json"
            sincedb_path => "NUL"
            type => "spring"
            start_position => "beginning"
        }
    }
    
    output {
        if [type] == "spring" {
            elasticsearch {
                hosts => ["192.168.1.110:9200"]
                index => "spring-%{+YYYY.MM.dd}"
            }
        }
        if [type] == "tcp" {
            elasticsearch { 
                hosts => ["192.168.1.110:9200"] 
                index => "logstash-%{+YYYY.MM.dd}"  
            }
        }
    	stdout { codec => rubydebug }
    }

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70][]

**2. 在docker-compose-elk.yml文件中Logstash容器下增加日志挂载目录(-v /d/usr/local/logs:/usr/share/logstash/logs)**

version: '3'
    services:
      elasticsearch:
        image: elasticsearch:7.6.2
        container_name: elasticsearch
        environment:
            # 设置集群名称为elasticsearch
            cluster:
                name=elasticsearch
            # # 以单一节点模式启动
            discovery:
                type=single-node
            # 设置使用jvm内存大小
            ES_JAVA_OPTS: -Xms512m -Xmx512m
        volumes:
            # 插件文件挂载
            - /d/usr/local/opt/elasticsearch/plugins:/usr/share/elasticsearch/plugins
            # 数据文件挂载
            - /d/usr/local/var/elasticsearch/data:/usr/share/elasticsearch/data
            - /d/usr/local/etc/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
        ports:
          - 9200:9200
          - 9300:9300
      logstash:
        image: logstash:7.6.2
        container_name: logstash
        volumes:
          # 挂载logstash的配置文件
            - /d/usr/local/etc/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
            - /d/usr/local/etc/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
            - /d/usr/local/logs:/usr/share/logstash/logs
        depends_on:
            # logstash在elasticsearch启动之后再启动
            - elasticsearch
        links:
            # 可以用es这个域名访问elasticsearch服务
            - elasticsearch:es
        ports:
            - 9600:9600
            - 5044:5044
      kibana:
        image: kibana:7.6.2
        container_name: kibana
        links:
            - elasticsearch:es
        depends_on:
            - elasticsearch
        environment:
            - "elasticsearch.hosts=http://es:9200"
        ports:
            - 5601:5601

### 三、.重启Logstash容器 ###

docker restart logstash

![202005141345435.png][]

### 四、 多次追加日志 ###

python3 save_log.py

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70 1][]

### 五、查看结果 ###

**1. 打开浏览器Elasticsearch head插件**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70 2][]

**到此数据源为file配置成功 **

[Springboot_ELK]: https://blog.csdn.net/zsx18273117003/article/details/106113036
[20200514133830788.png]: /images/20230312/f07b00141048437d91a16760ae88eab6.png
[20200514133953266.png]: /images/20230312/3cf3cee543914fb1b8d6b3f8d4327275.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200514194411914.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz,size_16,color_FFFFFF,t_70
[202005141345435.png]: https://img-blog.csdnimg.cn/202005141345435.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200514135926606.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200514195314822.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz,size_16,color_FFFFFF,t_70