Elasticsearch6.x之filebeat安装-yellowcong

￡神魔★判官ぃ 2022-02-16 13:29 321阅读 0赞

下载filebeat包，收集日志到logstash

## 下载地址 ##

https://www.elastic.co/cn/downloads/beats/filebeat
    
    
    #下载压缩包
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.0.0-linux-x86_64.tar.gz 
    
    #下载这个版本，我测试使用的这个
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.1-linux-x86_64.tar.gz

## 安装filebeat ##

#解压缩
    tar -zxvf  filebeat-7.0.0-linux-x86_64.tar.gz
    
    #移动文件
    mv filebeat-7.0.0-linux-x86_64 filebeat
    
    #启用nginx插件
    ./filebeat modules enable nginx
    
    #停用nginx插件
    ./filebeat modules disable nginx
    
    #查看模块列表
    ./filebeat modules list

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70]  
查看模块列表

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 1]

## 2 filebeat配置案例 ##

### 2.1 配置fielbeat输出到控制台 ###

这个配置，是将nginx的日志，直接输出到控制台上面。

#=========================== Filebeat inputs =============================
    
    filebeat.inputs:
    
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.
    
    - type: log
    
      # Change to true to enable this input configuration.
      enabled: true
    
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        - /var/log/*.log
        - /home/wwwlogs/*.log
    
    
    #----------------------------- Console output ---------------------------------
    output.console:
      # Boolean flag to enable or disable the output module.
      enabled: true
    
      # Configure JSON encoding
      #codec.json:
        # Pretty-print JSON event
        #pretty: false
    
        # Configure escaping HTML symbols in strings.
        #escape_html: false

### 2.2 启动服务 ###

#启动filebeat
    #-e 输出日志文件
    # -c 执行配置
     ./filebeat -e -c filebeat.yml

我访问nginx多次，然后查看 filebeat的控制台输出  
![在这里插入图片描述][2019042216224668.gif]

输出的数据格式

{ 
    	"@timestamp": "2019-04-22T08:22:24.341Z",
    	"@metadata": { 
    		"beat": "filebeat",
    		"type": "_doc",
    		"version": "7.0.0"
    	},
    	"log": { 
    		"offset": 6372,
    		"file": { 
    			"path": "/home/wwwlogs/access.log"
    		}
    	},
    	"message": "127.0.0.1 - - [22/Apr/2019:16:22:23 +0800] \"GET / HTTP/1.1\" 404 146 \"-\" \"curl/7.29.0\" \"-\" 0.000 \"-\" \"-\" \"-\"",
    	"input": { 
    		"type": "log"
    	},
    	"ecs": { 
    		"version": "1.0.0"
    	},
    	"host": { 
    		"name": "iz2ze5k5w9eimkab9rxz1xz",
    		"id": "f0f31005fb5a436d88e3c6cbf54e25aa",
    		"containerized": true,
    		"hostname": "iz2ze5k5w9eimkab9rxz1xz",
    		"architecture": "x86_64",
    		"os": { 
    			"kernel": "3.10.0-693.2.2.el7.x86_64",
    			"codename": "Core",
    			"platform": "centos",
    			"version": "7 (Core)",
    			"family": "redhat",
    			"name": "CentOS Linux"
    		}
    	},
    	"agent": { 
    		"type": "filebeat",
    		"ephemeral_id": "c3d10e31-b619-4f55-9817-16ac6167da00",
    		"hostname": "iz2ze5k5w9eimkab9rxz1xz",
    		"id": "f3e2c514-09a5-4bcc-a460-f6b72119a8e6",
    		"version": "7.0.0"
    	},
    	"cloud": { 
    		"availability_zone": "cn-beijing-e",
    		"instance": { 
    			"id": "i-2ze5k5w9eimkab9rxz1x"
    		},
    		"region": "cn-beijing",
    		"provider": "ecs"
    	}
    }

### 2.3 输入日志到es ###

filebeat.prospectors:
    - input_type: log
      paths: /var/log/nginx/access.log
      json.message_key:
    
    output.elasticsearch:
      #指定主机
      hosts: ["172.16.11.199"]
      #指定index
      index: "filebeat-nginx-%{+yyyy.MM.dd}"

### 2.配置nginx到filebeat ###

#启用nginx的插件
    ./filebeat modules enable nginx
    
    #查看模块列表
    ./filebeat modules list

可以看到启动的模块，有nginx模块。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 2]

## 常见问题 ##

### This Beat requires the default distribution of Elasticsearch ###

必须是一个elasticsearch的集群，才可以使用filebeat来搞事情啊

2019-04-22T19:52:16.118+0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://192.168.156.120:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 3]

## 参考文章 ##

https://www.jianshu.com/p/39453a671ce8  
https://www.tuicool.com/articles/Zji63qA  
https://blog.51cto.com/michaelkang/2317953  
https://www.cnblogs.com/jingmoxukong/p/8185321.html  
https://www.cnblogs.com/kingleft/p/7682678.html

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70]: /images/20220216/6a345c09a5244280a2b28851c2a992c5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 1]: /images/20220216/6a80fc49570542ae8419b438d464c093.png
[2019042216224668.gif]: /images/20220216/2af5a4c7440f4917a9a73bd8a011cee6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 2]: /images/20220216/b4cdc5a0e3f44c0394f85322bf6c5bf9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3llbGxsb3djb25n_size_16_color_FFFFFF_t_70 3]: /images/20220216/469feb58585846c6aad5a60d090d01a9.png