ELK 7.5.0（十一）ELK+kafka+Filebeat

刺骨的言语ヽ痛彻心扉 2023-06-20 06:19 5阅读 0赞

## 一、实验环境 ##

主机名        IP
    es        192.168.14.210
    kibana    192.168.14.210
    logstash  192.168.14.211
    kafka     192.168.14.212
    Filebeat  192.168.14.213
    nginx     192.168.14.213
    secure    192.168.14.213

## 二、安装部署（内容比较多，已经分多个章节） ##

1、ELK（elasticsearch+logstash+kibana）

具体查看第二、三、四章节

2、Filebeat安装

具体查看第六章节

## 三、安装kafka ##

Kafka依赖于Zookeeper；两个都需要java环境

1、官网下载

[https://archive.apache.org/dist/zookeeper/][https_archive.apache.org_dist_zookeeper]

[root@kafka ~]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz

[http://kafka.apache.org/downloads][http_kafka.apache.org_downloads]

[root@kafka ~]# wget http://mirror.bit.edu.cn/apache/kafka/2.3.1/kafka_2.12-2.3.1.tgz

2、安装jdk

[root@kafka ~]# tar -zxvf jdk-11.0.5_linux-x64_bin.tar.gz  -C /usr/local/
    [root@kafka ~]# vi /etc/profile
    export JAVA_HOME=/usr/local/jdk-11.0.5/
    export PATH=$PATH:$JAVA_HOME/bin
    export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
    [root@kafka ~]# source  /etc/profile
    [root@kafka ~]# java -version
    java version "11.0.5" 2019-10-15 LTS
    Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS)
    Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode)

3、Zookeeper安装并启动（默认启动配置文件是zoo.cfg）

[root@kafka ~]# tar -zxvf zookeeper-3.4.13.tar.gz -C /usr/local/
    [root@kafka ~]# cp /usr/local/zookeeper-3.4.13/conf/zoo_sample.cfg /usr/local/zookeeper-3.4.13/conf/zoo.cfg    #拷贝默认模板
    
    #修改配置文件，添加可以访问的地址
    [root@kafka ~]# vi /usr/local/zookeeper-3.4.13/conf/zoo.cfg
    clientPortAddress=0.0.0.0
    
    #启动服务
    [root@kafka ~]# /usr/local/zookeeper-3.4.13/bin/zkServer.sh start 
    ZooKeeper JMX enabled by default
    Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    #查看服务状态
    [root@kafka ~]# /usr/local/zookeeper-3.4.13/bin/zkServer.sh status 
    ZooKeeper JMX enabled by default
    Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
    Mode: standalone

4、kafka安装并启动

[root@kafka ~]# tar -zxvf kafka_2.11-2.1.1.tgz  -C /usr/local/
    [root@kafka ~]# mv /usr/local/kafka_2.11-2.1.1 /usr/local/kafka_2.11
    
    #修改配置文件
    [root@kafka ~]# cat /usr/local/kafka_2.11/config/server.properties |grep -v ^# |egrep  'listeners|zookeeper'
    listeners=PLAINTEXT://192.168.14.212:9092
    zookeeper.connect=192.168.14.212:2181
    
    #启动服务
    nohup /usr/local/kafka_2.11/bin/kafka-server-start.sh /usr/local/kafka_2.11/config/server.properties >/tmp/kafka.log 2>&1 &

5、查看端口

[root@kafka ~]# netstat -ntlp|grep java
    tcp6       0      0 :::42294                :::*                    LISTEN      14937/java          
    tcp6       0      0 :::36354                :::*                    LISTEN      18050/java          
    tcp6       0      0 192.168.14.212:9092     :::*                    LISTEN      18050/java          
    tcp6       0      0 :::2181                 :::*                    LISTEN      14937/java

## 四、Filebeat采集日志发送到kafka ##

1、修改配置文件

[root@filebeat ~]# vi /usr/local/filebeat-7.5.0/filebeat.yml
    filebeat.inputs:
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /usr/local/nginx/logs/access.json.log
      fields:
        filetype: nginx_access
      fields_under_root: true
    
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /var/log/secure
      fields:
        filetype: linux_secure
      fields_under_root: true
    
    
    output:
      kafka:
        hosts: ["192.168.14.212:9092"]
        topic: filebeat

2、重启服务（发现filebeat后台启动会自动退出，待解决问题）

[root@filebeat ~]# nohup filebeat -e -c /usr/local/filebeat-7.5.0/filebeat.yml > /tmp/filebeat.log 2>&1 &

## 五、logstash从kafka中读取数据 ##

1、修改配置文件

root@logstash ~]# vi /usr/local/logstash-7.5.0/config/logstash.conf 
    input {
        kafka {
          bootstrap_servers => "192.168.14.212:9092"
          topics => ["filebeat"]
          group_id => "filebeat"
          codec => "json"
      }
    }
    
    filter {
      if [filetype] == "nginx_access" {
        json {
          source => "message"
          remove_field => ["message","@version","path","input","log","agent","ecs","tags"]
        }
      }
    }
    
    output{
      if [filetype] == "nginx_access" {
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "nginx_access-%{+YYYY.MM.dd}"
        }
      }
      else if [filetype] == "linux_secure" {
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "linux_secure-%{+YYYY.MM.dd}"
        }
      }
    }

2、重启服务

[root@logstash ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf > /tmp/logstash.log 2> /tmp/logstash.log &

## 六、Kibana查看日志 ##

1、查看日志索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70][]

2、访问nginx网页产生新的日志，查看日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1][]

3、登录filebeat系统主机，产生日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2][]

至此，Filebeat（多台） - -> kafka（消息队列） - -> Logstash（正则过滤）- -> Elasticsearch（存储） - ->Kibana（显示）的架构部署完成

[https_archive.apache.org_dist_zookeeper]: https://archive.apache.org/dist/zookeeper/
[http_kafka.apache.org_downloads]: http://kafka.apache.org/downloads
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191209125248551.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191209125320854.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191209125416434.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70