ELK系列(四)、Logstash读取nginx日志写入ES中

逃离我推掉我的手 2023-02-11 13:27 9阅读 0赞

前面讲了ELK的部署以及Logstash的插件的安装方式，本篇就介绍一下如何使用Logstash读取nginx的日志，并写入ES中，通过Kibana分析。

[ELK系列(一)、安装ElasticSearch+Logstash+Kibana+Filebeat-v7.7.0][ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]

[ELK系列(二)、在Kibana中使用RESTful操作ES库][ELK_Kibana_RESTful_ES]

[ELK系列(三)、安装Logstash插件及打包离线安装包][ELK_Logstash]

\-----------------------------使用Logstash读取nginx日志------------------------------------

# 日志格式 #

上面讲了如何给Logstash安装插件以及离线安装，安装好的插件就在Logstash目录下:

/opt/app/logstash-7.7.0/vendor/bundle/jruby/2.5.0/gems

![20200523213823344.png][]

nginx的日志默认结构为HTTPD\_COMBINEDLOG，可以对比/usr/local/nginx/conf/nginx.conf和/opt/app/logstash-7.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/httpd 中的格式，然后我们就可以直接使用该格式的正则作为filter内容。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70][]

# 输出到控制台 #

### 配置logstash配置文件： ###

*vim $LS\_HOME/config/logstash\_nginxlog2print.conf *

input {
       # 从文件读取日志信息
       file {
            path => "/usr/local/nginx/logs/access.log"
            type => "system"
            start_position => "beginning"
        }
     }
    
    filter {
       grok {
          match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        }
    }
    
    output {
           # 标准输出
           stdout {}
    }

### 启动验证： ###

*bin/logstash -f config/logstash\_nginxlog2print.conf *

刷新nginx页面，或使用curl命令访问nginx，然后查看logstash的打印信息：

其中setting \{:sincedb\_path=>"/opt/app/logstash-7.7.0/data/plugins/inputs/file/.sincedb\_d2343edad78a7252d2ea9cba15bbff6d", :path=>\["/usr/local/nginx/logs/access.log"\]\}表示此配置文件读取该access.log文件，且读取的位置信息记录在sincedb文件内，下次再启动对这个日志文件的监控都会从记录的offset开始读，因此如果需要从头读取需要把这个文件先删掉。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 3][]

# 输出到ES #

### 配置logstash配置文件 ###

*vim $LS\_HOME/config/logstash\_nginxlog2es.conf *

input {
       # 从文件读取日志信息
       file {
            path => "/usr/local/nginx/logs/access.log"
            type => "nginx_access"
            start_position => "beginning"
        }
     }
    
    filter {
       grok {
          match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        }
    }
    
    output {
           # 输出es
           elasticsearch {
              hosts => ["wykd:9200"]
              index => "nginx-%{+YYYY.MM.dd}"
          }
    
    }

### 启动验证 ###

先删掉刚才的sincedb文件，然后启动监控并写入ES:

*rm -rf /opt/app/logstash-7.7.0/data/plugins/inputs/file/.sincedb\_d2343edad78a7252d2ea9cba15bbff6d*

*bin/logstash -f config/logstash\_nginxlog2es.conf *

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 4][]

### Kibana中分析nginx日志 ###

可以在kibana中看到nginx日志的索引已经自动创建了，此时我们使用curl命令访问nginx，该索引内的数据量也会实时变化：

在kibana中创建index patten，并在discover中查看index数据：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 7][]

如果对你有帮助，请点个赞鼓励一下作者吧~ 谢谢！

[ELK_ElasticSearch_Logstash_Kibana_Filebeat-v7.7.0]: https://blog.csdn.net/wsdc0521/article/details/106236499
[ELK_Kibana_RESTful_ES]: https://blog.csdn.net/wsdc0521/article/details/106241596
[ELK_Logstash]: https://blog.csdn.net/wsdc0521/article/details/106292639
[20200523213823344.png]: https://img-blog.csdnimg.cn/20200523213823344.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2020052321431721.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200523220713868.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200523215807725.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200523215908403.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200523221826271.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200523222033837.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200523222048914.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/2020052322241915.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70