ELK进阶为filebeat+reids+ELK（windows版）

小鱼儿 2021-10-20 00:14 388阅读 0赞

在上一章里，我们学会了如何搭建ELK实时日志分析平台，也知道了ELK是由Elasticsearch+logstash+kibana组合而成，但是其实这里面是有一个弊端的，那就是logstash收集日志运行依赖于java环境，对内存资源和cpu的占用较大，所以还是有待优化的。而今天，日月在这里就教大家如何优化这一点，那就是使用filebeat去收集日志，并使用redis作为中转，filebeat运行无需依赖于java环境，对内存资源和cpu几乎没多少消耗，并且使用redis作为中转也可以提高并发性能。于是filebeat+reids+ELK应运而生。

OK，我们话不多说，来开始我们的ELK改造。

**1、下载filebeat**  
下载地址：https://www.elastic.co/downloads/beats  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70]

解压至F盘

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 1]

**2、filebeat配置文件**  
新建 filebeat2.yml

#=========================== Filebeat inputs =============================
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - D:/testlog.log
      encoding: utf-8
      scan_frequency: 10s
      harvester_buffer_size: 16384
      max_bytes: 10485760
      tail_files: true
    #============================= Filebeat modules ===============================
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    #==================== Elasticsearch template setting ==========================
    setup.template.settings:
      index.number_of_shards: 3
    #============================== Kibana =====================================
    setup.kibana:
    #-------------------------- reids output ------------------------------
    output.redis:
      hosts: ["127.0.0.1:6379"]
      key: filebeat
      timeout: 5s
      max_retries: 3

**3、下载安装redis**

redis的下载安装我这里就不在陈述了，大家参考 [springBoot集成redis][springBoot_redis] 即可

我们先启动redis  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 2]

**4、启动filebeat**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 3]

doc下进入filebeat安装目录，执行 `filebeat -c filebeat2.yml`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 4]

启动后会在安装目录下生成data和logs两个文件夹，我们去logs里面查看启动日志

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 5]  
可以看到，启动成功。

**5、修改logstash配置文件**  
logstash-sample.conf

# For detail structure of this file
    # Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
    input {
      redis {
        host => "127.0.0.1"
        port => "6379"
        data_type => "list"
        key => "filebeat"
        type => "redis-input"
      }
    }
    filter {
    }
    output {
      # For detail config for elasticsearch as output, 
      # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
      elasticsearch {
        action => "index"          #The operation on ES
        hosts  => "127.0.0.1:9200"   #ElasticSearch host, can be array.
        index  => "chenqilog"         #The index to write data to.
      }
    }

将input指向redis

**6、依次启动Elasticsearch、logstash、kibana**

启动es,在bin目录下双击elasticsearch.bat文件即可

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 6]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 7]

**启动logstash**  
进入logstash的bin目录  
输入 `logstash -f ../config/logstash-sample.conf`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 8]

**启动kibana**  
双击运行bin目录下的 kibana.bat  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 9]

7、浏览器访问 localhost:5601  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 10]

可以看到，成功查询到了之前的20条日志，我们继续往日志文件里添加10条

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 11]

再次刷新kibana

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 12]

说明，filebeat成功的收集到了日志内容，并通过redis传输到logstash,在保存至elasticsearch中，最后由kibana查询展示

至此，filebeat + redis + ELK 搭建完成

如果该文章有帮助到您，就留言点个赞吧！您的支持与肯定是我持续更新最大的动力。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70]: /images/20211019/81eff0b6e5cc4553a5d02fb4a45d5969.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 1]: /images/20211019/85171753eca34c42a93652a4f82e208f.png
[springBoot_redis]: https://blog.csdn.net/weixin_40623736/article/details/98097708
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 2]: /images/20211019/3acc51e845b640fb9d6306228450ed71.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 3]: /images/20211019/3af82c98939c421f8bac66ae406bfcae.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 4]: /images/20211019/6f0e36fad55e43b9b2aa24177f897509.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 5]: /images/20211019/f31aa59f71724f848e8156a3f2b83e64.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 6]: /images/20211019/fc6eb45f749c45fa878075e8202cf684.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 7]: /images/20211019/168edc48ea634495800962133be9db92.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 8]: /images/20211019/374af745e0534cca8b6a5156d5e17098.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 9]: /images/20211019/db2bfc31d4f2406caa2682066b7b95dc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 10]: /images/20211019/e420e25a604e4359b9dd1cef9053dcac.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 11]: /images/20211019/3db938a73a2f4956b941f2a23980ca7f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDYyMzczNg_size_16_color_FFFFFF_t_70 12]: /images/20211019/c4385c375e09493f8ea4481cb6083aee.png