elk7.5 之 filebeat部署

悠悠 2023-06-20 13:56 6阅读 0赞

原理篇请查看`https://blog.csdn.net/xiaokangtongxue410/article/details/82838872`  
这里我就不copy了

部署、配置过程：  
安装略。。。  
配置目录放在：`/etc/filebeat`

$ cd /etc/filebeat
    $ ll
    -rw-r--r-- 1 root root  938 Oct 14 10:46 filebeat-htjfweb.yml
    drwxr-xr-x 2 root root 4096 Nov  8 14:39 prospectors.d
    $ cat filebeat-htjfweb.yml
    path.config: /etc/filebeat
    path.logs: /var/log/filebeat-kafka-htjfweb
    path.data: /var/lib/filebeat/data-kafka-htjfweb
    filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb
    filebeat.shutdown_timeout: 0
    logging.level: info
    logging.metrics.enabled: false
    logging.files.rotateeverybytes: 104857600
    logging.files.keepfiles: 10
    logging.files.permissions: 0600
    
    setup.template.name: "filebeat"
    setup.template.pattern: "filebeat-*"
    filebeat.config:
        prospectors:
            enabled: true
            path: ${path.config}/prospectors.d/product-htjfweb*.yml ##热加载配置
            reload.enabled: true
            reload.period: 10s
    
    output.kafka:
          enabled: true
          hosts: ["10.100.202.177:9092","10.100.202.191:9092","10.100.202.192:9092"]
          topic: logstash-htjfweb
          partition.round_robin: # 集群是否开启kafka的partition分区
            reachable_only: false
          compression: gzip
          max_message_bytes: 10000000
          required_acks: 1

我的配置文件放在prospectors.d目录中

$ cat prospectors.d/product-htjfweb-account-web.yml
    - type: log
      enabled: true
      paths:
          - /data/WEBLOG/product-htjfweb-account-web/*/*.log
      scan_frequency: 10s
      fields_under_root: true
      multiline.pattern: '^\d{4}-\d{1,2}-\d{1,2}'
      multiline.negate: true
      multiline.match: after
      multiline.max_lines: 500
      multiline.timeout: 5s
      exclude_lines: ['WARN']
    
      fields:
          env: product
          index: logstash-htjf
          logtype: product-htjfweb-account-web
          topic: logstash-htjf
          k8s_host: node
           
      tail_files: false
      close_inactive: 2h
      close_eof: false
      close_removed: true
      clean_removed: true
      close_renamed: false

启动filebeat

$ /usr/local/bin/filebeat -c /etc/filebeat/filebeat-htjfweb.yml

如有多个配置文件 需要将以下配置变更成不同的文件，不然启动报错。

path.config: /etc/filebeat
    path.logs: /var/log/filebeat-kafka-htjfweb
    path.data: /var/lib/filebeat/data-kafka-htjfweb
    filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb

通过查看kafka 的topic里是否有数据来判断收集是否成功。

$ cd /usr/local/kafka/bin/
    $ ./kafka-console-consumer.sh --bootstrap-server 10.100.202.191:9092 --topic logstash-htjfweb --from-beginning ##从头查看收集到的数据
    $  ./kafka-consumer-groups.sh --group logstash-htjfweb --describe --bootstrap-server 10.100.202.177:9092     ##查看消费情况 
    
    GROUP            TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
    logstash-htjfweb logstash-htjfweb 0          13965817        13965822        5               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0
    logstash-htjfweb logstash-htjfweb 1          13965595        13965599        4               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0
    logstash-htjfweb logstash-htjfweb 2          13965756        13965761        5               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0