使用apt-get安装filebeat

冷不防 2023-05-31 06:48 15阅读 0赞

*   [apt-get安装方法][apt-get]

apt-get update
    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
    apt-get install apt-transport-https
    echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list  #ubuntu系统使用7.4.0版本时报错Syscall filter could not be installed because the kernel does not support seccomp
    apt-get install filebeat

*  [filebeat reference][]. [filebeat官方设置和运行命令][filebeat]，[对应中文版][Link 1]

<table> 
 <tbody> 
 <tr> 
 <th>类型</th> 
 <th>描述</th> 
 <th>使用apt-get安装后的位置</th> 
 </tr> 
 </tbody> 
 <tbody> 
 <tr> 
 <td> home </td> 
 <td> Filebeat安装的主页。 </td> 
 <td> <code>/usr/share/filebeat</code> </td> 
 </tr> 
 <tr> 
 <td> bin </td> 
 <td> 二进制文件的位置。 </td> 
 <td> <code>/usr/share/filebeat/bin</code> </td> 
 </tr> 
 <tr> 
 <td> config </td> 
 <td> 配置文件的位置。 </td> 
 <td> <code>/etc/filebeat</code> </td> 
 </tr> 
 <tr> 
 <td> data </td> 
 <td> 持久数据文件的位置。 </td> 
 <td> <code>/var/lib/filebeat</code> </td> 
 </tr> 
 <tr> 
 <td> logs </td> 
 <td> 由Filebeat创建的日志的位置。 </td> 
 <td> <code>/var/log/filebeat</code> </td> 
 </tr> 
 </tbody> 
</table>

*  [filebeat输入类型的配置][filebeat 1]
 *  [filebeat输出类型的配置][filebeat 2]
 *  简单调试命令

#指定测试文件
    filebeat -e --path.config /serving/myfilebeat
    #测试模式(日志输出在终端)
    filebeat -e  -d "publish" --path.config /serving/myfilebeat

*  配置文件-输入部分

#=========================== Filebeat inputs =============================
    
    filebeat.inputs:
    
    - type: log
    
      # Change to true to enable this input configuration.
      enabled: True    #自定义日志目录时，须修改为True
    
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        - /workspace/test/kafka/filebeat/filebeat_log/dmcv_*.log
        #- /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*
      #exclude_lines: ['WARN', 'ERROR']
      #include_lines: ['INFO']
      #exclude_files: ['.gz$']

*  配置文件-输出部分

#-------------------------- Kafka output ------------------------------
    output.kafka:
      # initial brokers for reading cluster metadata
      hosts: ["localhost:9093", "localhost:9094", "localhost:9095"] #生产者port
      version: "0.10.1.0"
    
      # message topic selection + partitioning
      topic: "face_kafka"
      partition.round_robin:
        reachable_only: false
    
      required_acks: 1
      compression: gzip
      #compression: none
      max_message_bytes: 1000000

问题及解决方法：

*  Unable to locate package filebeat

apt-get update

*  [gpg: no valid OpenPGP data found][gpg_ no valid OpenPGP data found]

apt-get install ca-certificates

*  [Failed to connect to kafka(localhost:9092): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)][Failed to connect to kafka_localhost_9092_ kafka_ client has run out of available brokers to talk to _Is your cluster reachable]

连接单机的kafka报错，因为需要多个broker。启动3个broker后可以运行filebeat.

参考文献：

1.  [Filebeat Reference][filebeat reference]
2.  [使用filebeat收集日志到kafka][filebeat_kafka]
3.  [apt and yum install filebeat][apt-get]
4.  [gpg: no valid OpenPGP data found][gpg_ no valid OpenPGP data found]
5.  [Setting up and running Filebeat][]
6.  [设置和运行Filebeat][Link 1]
7.  [卡夫卡：客户已经没有可用的经纪人与他们交谈了][Failed to connect to kafka_localhost_9092_ kafka_ client has run out of available brokers to talk to _Is your cluster reachable]

[apt-get]: https://www.elastic.co/guide/en/beats/filebeat/7.4/setup-repositories.html#_apt
[filebeat reference]: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
[filebeat]: https://www.elastic.co/guide/en/beats/filebeat/current/setting-up-and-running.html
[Link 1]: https://my.oschina.net/ch66880/blog/1619087
[filebeat 1]: https://www.elastic.co/guide/en/beats/filebeat/7.4/configuration-filebeat-options.html
[filebeat 2]: https://www.elastic.co/guide/en/beats/filebeat/7.4/configuring-output.html
[gpg_ no valid OpenPGP data found]: https://stackoverflow.com/questions/21338721/gpg-no-valid-openpgp-data-found
[Failed to connect to kafka_localhost_9092_ kafka_ client has run out of available brokers to talk to _Is your cluster reachable]: https://xbuba.com/questions/49988530
[filebeat_kafka]: https://www.zhoujiangang.com/p/filebeat-collect-log2kafka/
[Setting up and running Filebeat]: https://www.elastic.co/guide/en/beats/filebeat/current/setting-up-and-running.html#setting-up-and-running