ELK 7.5.0（八）ELK+Filebeat采集Json日志

系统管理员 2023-06-19 11:24 62阅读 0赞

## 一、实验环境 ##

主机名        IP
    es        192.168.14.210
    kibana    192.168.14.210
    logstash  192.168.14.211
    Filebeat  192.168.14.213
    nginx     192.168.14.213

## 二、安装部署（内容比较多，已经分开写） ##

1、ELK（elasticsearch+logstash+kibana）

具体查看第二、三、四章节

2、Filebeat安装

具体查看第六章节

## 三、Nginx使用json格式日志 ##

json的好处：

客户端原生日志需要Logstash做正则匹配，比较麻烦，也会消耗系统资源；  
客户端设置Json格式的日志不需要Logstash正则分析，就能直接分段采集，减少服务器压力。

1、nginx配置文件添加Json日志格式

[root@filebeat ~]# vi /usr/local/nginx/conf/nginx.conf
        log_format json '{"@timestamp":"$time_iso8601",'
                        '"clientip":"$remote_addr",'
                        '"status":$status,'
                        '"bodysize":$body_bytes_sent,'
                        '"referer":"$http_referer",'
                        '"ua":"$http_user_agent",'
                        '"handletime":$request_time,'
                        '"url":"$uri"}';
    
        access_log  logs/access.json.log  json;

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70][]

2、启动或重载配置文件

[root@filebeat ~]# nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    [root@filebeat ~]# nginx -s reload

3、浏览器访问nginx，然后本机查看nginx日志输出是否正常

![20191206152352272.png][]

## 四、Filebeat采集Json日志格式 ##

1、修改配置文件

[root@filebeat ~]# vi  /usr/local/filebeat-7.5.0/filebeat.yml
    filebeat.inputs:
    - type: log
      tail_files: true
      backoff: "1s"
      paths:
          - /usr/local/nginx/logs/access.json.log
    
    output:
      logstash:
        hosts: ["192.168.14.211:5044"]

2、启动或者重启Filebeat服务

[root@filebeat ~]# pkill filebeat
    [root@filebeat ~]# nohup filebeat -e -c /usr/local/filebeat-7.5.0/filebeat.yml > /tmp/filebeat.log 2>&1 &

## 五、Logstash解析Json日志 ##

1、修改配置文件，根据需求删除不必要字段

[root@logstash ~]# vi /usr/local/logstash-7.5.0/config/logstash.conf 
    input {
      beats {
        host => '0.0.0.0'
        port => 5044
      }
    }
    filter {
      json {
        source => "message"
         remove_field => ["message","@version","path","input","log","agent","ecs","tags"]    #不需要记录，移除的字段
      }
    }
    output {
      elasticsearch {
        hosts => ["http://192.168.14.210:9200"]
        user => "elastic"
        password => "elkpwd"
        index => "logstash-%{+YYYY.MM.dd}"
      }
    }

2、启动或者重启服务

[root@logstash ~]# kill -9 14598(logstash进程ID)
    [root@logstash ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf > /tmp/logstash.log 2> /tmp/logstash.log &

3、在kibana查看json日志格式

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1][]

至此，json日志格式采集并展示完成。tomcat其他服务类似此操作。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191206151838869.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[20191206152352272.png]: https://img-blog.csdnimg.cn/20191206152352272.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191206153651371.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70