elk之filebeat安装配置并使用logstash接收数据

女爷i 2022-01-25 15:12 322阅读 0赞

下载:

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.2-x86\_64.rpm

安装:

rpm -ivh filebeat-6.4.2-x86\_64.rpm

配置:

vim /etc/filebeat/filebeat.yml

filebeat.inputs:

\# Each – is an input. Most options can be set at the input level, so  
\# you can use different inputs for various configurations.  
\# Below are the input specific configurations.

– type: log  
enabled: true  
paths:  
– /opt/lampp/logs/access\_log  
fields:  
index\_name: new\_guanwang\_access\_master

\#—————————– Logstash output ——————————–  
output.logstash:  
\# The Logstash hosts  
hosts: \[“192.168.3.124:5044”\]

重启

/etc/init.d/filebeat restart

logstash配置

input \{  
beats \{  
port => “5044”  
\}  
\}  
filter \{  
if \[fields\]\[index\_name\] == “tour\_alllegend\_access” and \[fields\]\[index\_name\] == “new\_guanwang\_access\_master” \{  
grok \{  
match => \{ “message” => “%\{COMMONAPACHELOG\}” \}  
\}  
date \{  
match => \[ “timestamp”, “dd/MMM/YYYY:H:m:s Z” \]  
target => “@timestamp”  
\}  
\} else \{  
grok \{  
match => \{ “message” => “%\{COMBINEDAPACHELOG\}” \}  
\}  
date \{  
match => \[ “timestamp”, “dd/MMM/YYYY:H:m:s Z” \]  
target => “@timestamp”  
\}  
\}  
\}  
output \{  
elasticsearch \{  
hosts => \[“localhost:9200”\]  
index => “%\{\[fields\]\[index\_name\]\}-%\{+YYYY.MM.dd\}”  
\}  
\}

删除filebeat历史上传记录

rm /var/lib/filebeat/registry

停止 删除 启动 tail

/etc/init.d/filebeat stop;rm /var/lib/filebeat/registry;/etc/init.d/filebeat start;sleep 3;tail -f /var/log/filebeat/filebeat -n 100

\} else if \[fields\]\[index\_name\] == “new\_guanwang\_access\_slave” \{  
16 grok \{  
17 match => \{ “message” => “%\{IPORHOST:clientip\} %\{USER:ident\} %\{NOTSPACE:auth\} \\\[%\{HTTPDATE:timestamp\}\\\] \\”(?:%\{WORD:verb\} %\{NOTSPACE:re  
quest\}(?: HTTP/%\{NUMBER:httpversion\})?|%\{DATA:rawrequest\})\\” %\{NUMBER:response\} (?:%\{NUMBER:bytes\}|-) \\”%\{NOTSPACE:referrer\}\\” \\”%\{GREEDYDAT  
A:agent\}\\” \\”%\{IPORHOST:http\_x\_forwarded\_for\}\\”” \}  
18 \}  
19 date \{  
20 match => \[ “timestamp”, “dd/MMM/YYYY:H:m:s Z” \]  
21 target => “@timestamp”  
22 \}  
23 \} else \{