Logstash用法实例

你的名字 2022-10-02 06:55 228阅读 0赞

[2019独角兽企业重金招聘Python工程师标准>>> ][2019_Python_] ![hot3.png][]

首先提供一些参考的网站：

[https://www.gitbook.com/book/chenryn/kibana-guide-cn][https_www.gitbook.com_book_chenryn_kibana-guide-cn]

[http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html][http_udn.yyuap.com_doc_logstash-best-practice-cn_index.html]

[https://www.gitbook.com/book/chenryn/kibana-guide-cn][https_www.gitbook.com_book_chenryn_kibana-guide-cn]  
[http://www.learnes.net/index.html][http_www.learnes.net_index.html]

下面由简单到复杂，介绍一些Logstash的配置方法

安装完Logstash后，默认安装在/opt/logstash中，在bin目录中有一些命令可以使用。

最简单的配置语法：

input {
        stdin {}
        syslog {}
    }

Logstash的Helloworld：

[root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{}}'
    !!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
    Settings: Default pipeline workers: 8
    Logstash startup completed
    hello world
    2016-03-17T03:30:02.147Z BLELOIST003.lenovo.com hello world
    [root@BLELOIST003 logstash]#
    [root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
    !!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
    Settings: Default pipeline workers: 8
    Logstash startup completed
    hello world
    {
           "message" => "hello world",
          "@version" => "1",
        "@timestamp" => "2016-03-17T03:31:30.308Z",
              "host" => "BLELOIST003.lenovo.com"
    }

最基本的配置组成是input output，可选的是filter。input中的stdin和output中的stdout代表了命令行窗口的输入输出。

监控apache日志，这里没有做过滤。

[root@BLELOIST003 logstash]# bin/logstash -e '
     input {
       file { 
         type => "apache-access" 
         path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*" 
         start_position => end 
         sincedb_path => "/dev/null"
       }
     }
     filter{
       
     }
     output {
       stdout{ codec=>rubydebug }
     }'
    !!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
    Settings: Default pipeline workers: 8
    Logstash startup completed
    {
           "message" => "10.100.90.241 - - [17/Mar/2016:11:35:00 +0800] \"GET / HTTP/1.1\" 304 - 411",
          "@version" => "1",
        "@timestamp" => "2016-03-17T03:35:01.162Z",
              "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
              "host" => "BLELOIST003.lenovo.com",
              "type" => "apache-access"
    }
    {
           "message" => "10.99.80.8 - dummy002 [17/Mar/2016:11:35:01 +0800] \"POST /LOIS/servlet/SEOTranslationService HTTP/1.0\" 200 90360 503291",
          "@version" => "1",
        "@timestamp" => "2016-03-17T03:35:03.178Z",
              "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
              "host" => "BLELOIST003.lenovo.com",
              "type" => "apache-access"
    }

start\_position => end 代表了只监控新增加的日志内容。

加上正则表达式的过滤：

[root@BLELOIST003 logstash]# bin/logstash -e '
     input {
       file { 
         type => "apache-access" 
         path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*" 
         start_position => end 
         sincedb_path => "/dev/null"
       }
     }
     filter{
       if [path] =~ "access" {
         mutate { replace => { "type" => "ApacheLogs" } }
         grok {
           match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:userName} \[%{HTTPDATE:logTime}\] \"%{WORD:verb} %{NOTSPACE:request} (?:%{NOTSPACE:httpversion}|)\" (?:%{NUMBER:state}|-) (?:%{NUMBER:bytes}|-) %{NUMBER:duration}"}
         }
       }
       date {
         match => [ "logTime" , "dd/MMM/yyyy:HH:mm:ss Z" ]
       }
     }
     output {
       stdout{ codec=>rubydebug }
     }'
    
    Settings: Default pipeline workers: 8
    Logstash startup completed
    {
            "message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /Windchill/ HTTP/1.1\" 404 208 424",
           "@version" => "1",
         "@timestamp" => "2016-03-17T04:05:27.000Z",
               "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
               "host" => "BLELOIST003.lenovo.com",
               "type" => "ApacheLogs",
           "clientip" => "10.100.90.241",
              "ident" => "-",
           "userName" => "-",
            "logTime" => "17/Mar/2016:12:05:27 +0800",
               "verb" => "GET",
            "request" => "/Windchill/",
        "httpversion" => "HTTP/1.1",
              "state" => "404",
              "bytes" => "208",
           "duration" => "424"
    }
    {
            "message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /favicon.ico HTTP/1.1\" 200 207 476",
           "@version" => "1",
         "@timestamp" => "2016-03-17T04:05:27.000Z",
               "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
               "host" => "BLELOIST003.lenovo.com",
               "type" => "ApacheLogs",
           "clientip" => "10.100.90.241",
              "ident" => "-",
           "userName" => "-",
            "logTime" => "17/Mar/2016:12:05:27 +0800",
               "verb" => "GET",
            "request" => "/favicon.ico",
        "httpversion" => "HTTP/1.1",
              "state" => "200",
              "bytes" => "207",
           "duration" => "476"
    }

这里的if \[path\] =~ "access"代表了判断语句，可以参考这里：

[http://kibana.logstash.es/content/logstash/get\_start/full\_config.html][http_kibana.logstash.es_content_logstash_get_start_full_config.html]

这里的过滤中grok中匹配：

match => \{ "message" => "%\{IPORHOST:clientip\} %\{HTTPDUSER:ident\} %\{USER:userName\} \\\[%\{HTTPDATE:logTime\}\\\] \\"%\{WORD:verb\} %\{NOTSPACE:request\} (?:%\{NOTSPACE:httpversion\}|)\\" (?:%\{NUMBER:state\}|-) (?:%\{NUMBER:bytes\}|-) %\{NUMBER:duration\}"\}

其中%\{IPORHOST:clientip\}表示具体的正则匹配，IPORHOST代表了logstash的pattern，具体可以参考如下链接，而clientip则代表了在logstash中信息传递的变量。

[https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns][https_github.com_logstash-plugins_logstash-patterns-core_blob_master_patterns_grok-patterns]

转载于:https://my.oschina.net/shawnplaying/blog/639449

[2019_Python_]: https://my.oschina.net/u/2663968/blog/3061697
[hot3.png]: /images/20220113/e7ec1f7c5bca4f3cb5a5e5fb740eaeb6.png
[https_www.gitbook.com_book_chenryn_kibana-guide-cn]: https://www.gitbook.com/book/chenryn/kibana-guide-cn
[http_udn.yyuap.com_doc_logstash-best-practice-cn_index.html]: http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html
[http_www.learnes.net_index.html]: http://www.learnes.net/index.html
[http_kibana.logstash.es_content_logstash_get_start_full_config.html]: http://kibana.logstash.es/content/logstash/get_start/full_config.html
[https_github.com_logstash-plugins_logstash-patterns-core_blob_master_patterns_grok-patterns]: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns