Logstash+Redis+Elasticsearch+Kibana 快速搭建Nginx日志查询系统

心已赠人 2022-08-26 05:53 215阅读 0赞

# Logstash+Redis+Elasticsearch+Kibana Nginx日志查询系统 #

环境

elasticsearch-0.90.5.zip
    kibana-latest.zip
    redis-2.6.16.tar.gz
    logstash-1.2.2-flatjar.jar

nginx.conf配置

log_format  main  '$remote_addr - $remote_user [$time_local] ' 
                               '"$request" $status $body_bytes_sent '
                               '"$http_referer" "$http_user_agent" ';

nginx日志

172.16.201.174 - - [25/Mar/2014:16:39:13 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1772.0 Safari/537.36"

表达式

%{ IPORHOST:source_ip} - %{ USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{ QS:request} %{ INT:status} %{ INT:body_bytes_sent} %{ QS:http_referer} %{ QS:http_user_agent}

collection

hadoop@stormspark:~/log/logstash$ cat  sp.conf 
    input {
       file {
           type => "nginx-access"
           path => "/var/log/nginx/access.log"
       }
    }
    output {
        stdout {
            debug => true
            debug_format => json
        }
    
    
        redis {
            host => "127.0.0.1"
            port => 6379
            data_type => "list"
            key => "logstash"
        }
    }

index配置

hadoop@stormspark:~/log/logstash$ cat index.conf 
    input {
      redis {
        host => "127.0.0.1"
        port => "6379"
        data_type => "list"
        key => "logstash"
        type => "redis-input"
      }
    }
    filter {
       grok {
           type => "nginx-access"
           pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}"
       }
    }
    output {
      elasticsearch {
        host => "127.0.0.1"
      }
    }

分别启动logstash,redis,es等。

java -jar logstash-1.2.2-flatjar.jar agent -f sp.conf
    java -jar logstash-1.2.2-flatjar.jar agent -f index.conf

最后来个截图：

![SouthEast][]

[SouthEast]: /images/20220824/ca8c46f85d964a19871c8f235e6373e0.png