logstash7.5.1同步nginx日志到elasticsearch

旧城等待， 2023-07-03 02:40 18阅读 0赞

测试环境：centos

**前言：**本文在已经安装elasticsearch的基础进行，可参考我的上篇博客--[  elasticsearch7.5.1集群安装+head插件安装][elasticsearch7.5.1_head]

本文实现如下简单粗暴的日志收集架构

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70][]

上面的架构是还没有进行日志过滤的，只是简单的将nginx日志通过logstash传输到es中，过滤环节是很耗时间和资源的。所以一般都会加上一个消息中间件比如kafka或redis这种读写性能是非常高的应用。本文只简单演示如上的功能。

在开始之前先配置一下nginx日志的格式，在nginx配置文件中修改如下

log_format main '{"client_ip":"$remote_addr","log_time":"$time_local","request":"$request","status":"$status","body_bytes_sent":"$body_bytes_sent","http_referer":"$http_referer","AgentVersion":"$http_user_agent","upstream_addr":"$upstream_addr","request_time":"$request_time","upstream_response_time":"$upstream_response_time"}';

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 1][]

**一、下载logstash**

1、wget [https://www.elastic.co/downloads/past-releases/logstash-7-5-1][https_www.elastic.co_downloads_past-releases_logstash-7-5-1]

2、解压：

tar -xzf logstash-7.5.1.tar.gz

3、进入到目录logstash-7.5.1中新建一个配置文件（logstash-nginx-log.conf）用于配置输入、输出信息

加入如下配置：

input{
      file{
        #日志存放路径
        path => "/var/log/nginx/access.log"
        #从日志文件开头读取
        start_position => "beginning"
        #设置多长时间检测文件是否修改 默认是1s
        stat_interval => "2"
        codec => "json"
      }
    }
    
    filter {}
    output {  # 定义日志输出
    
        elasticsearch {
            hosts => ["localhost:9200"]       # 定义es服务器的ip，这里使用本地
            index => "nginx-log"              #定义索引名称
        }
    
        stdout {
            codec => json_lines
        }
    
    }

4、启动logstash

./bin/logstash -f logstash-nginx-log.conf

5、常见错误

如遇到如下错误：

：Logstash could not be started because there is already another instance usin

解决办法：

之前运行的实例有缓冲，保存在data/里面有.lock文件删除掉即可。

rm -rf data/.lock

启动成功后logstash回把nginx日志传输到elasticsearch中

6、head插件查看数据

可以通过head插件来查看日志传输情况

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 2][]

可以看到nginx的日志已经同步到了elasticsearch。

7、可以用kibana简单的分析一下日志（需要安装）

查看所有访问中状态分布：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 3][]

可以看出在所有请求中有22次是成功的。

到此演示结束！

[elasticsearch7.5.1_head]: https://blog.csdn.net/zzqaaasss/article/details/104119601
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200131162347391.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020013117134021.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw==,size_16,color_FFFFFF,t_70
[https_www.elastic.co_downloads_past-releases_logstash-7-5-1]: https://www.elastic.co/downloads/past-releases/logstash-7-5-1
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200131170846714.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200131175258928.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6cWFhYXNzcw==,size_16,color_FFFFFF,t_70