logstash之filter-patterns库的正则表达式非格式化的数据格式化

秒速五厘米 2023-06-16 02:29 1阅读 0赞

### patterns正则表达式库介绍 ###

默认grok调用的是：/logstash-5.5.2/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.1/patterns 这个目录下的正则  
上面的例子，可以这样写：

input {stdin{}}
    filter {
        grok {
            match => {
    # "message" => "(?<date>\d+\.\d+)\s+(?<is>\w+)\s+(?<luck>\w+)\s+(?<day>\w+)"
    
                "message" => "%{NUMBER:date:float} %{WORD:is} %{WORD:luck} %{WORD:day}"
            }
        }
    }
    output {stdout{codec => rubydebug}}

结果截图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ_size_16_color_FFFFFF_t_70]

### grok将非结构化数据进行结构化 ###

Nginx打印出的日志一般格式是：

192.168.77.1 - - [09/Apr/2018:20:08:18 +0800] "GET / HTTP/1.1" 200 156 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.
    0.3325.181 Safari/537.36"
    192.168.77.1 - - [09/Apr/2018:20:08:19 +0800] "GET /plugins/kibana/assets/dashboard.svg HTTP/1.1" 200 301 "http://hadoop01/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"

这种日志是非格式化的，通常，我们获取到日志后，还要使用mapreduce或者spark做一下清洗操作，就是将非格式化日志编程格式化日志；  
在清洗的时候，如果日志的数据量比较大，那么也是需要花费一定的时间的；  
所以可以使用logstash的grok功能，将nginx的非格式化数据采集成格式化数据：

input {stdin{}}
    filter {
        grok {
            match => {
                    "message" => "%{IPORHOST:remote_addr} - %{NGUSER:remote_addr} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:request} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NOTSPACE:http_x_forwarded_for}"
            }
        }
    }
    output {stdout{codec => rubydebug}}

启动：  
bin/logstash -f /usr/local/elk/logstash-5.5.2/conf/template/format\_log.conf  
在控制台输入日志：

192.168.77.1 - - [10/May/2018:12:12:40 +0800] "GET /plugins/ml/ml.svg HTTP/1.1" 304 0 "http://hadoop01/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" "-"

结果展示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ_size_16_color_FFFFFF_t_70 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191124074330610.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191124074433198.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDA2MjMzOQ==,size_16,color_FFFFFF,t_70