EFK教程 - EFK快速入门指南

古城微笑少年丶 2023-06-27 05:31 7阅读 0赞

# 目录 #

▪ 用途  
▪ 实验架构  
▪ EFK软件安装  
▪ elasticsearch配置  
▪ filebeat配置  
▪ kibana配置  
▪ 启动服务  
▪ kibana界面配置  
▪ 测试  
▪ 后续文章

--------------------

# 用途 #

▷ 通过filebeat实时收集nginx访问日志、传输至elasticsearch集群  
▷ filebeat将收集的日志传输至elasticsearch集群  
▷ 通过kibana展示日志

--------------------

# 实验架构 #

▷ 服务器配置

![format_png][]

image

▷ 架构图

![format_png 1][]

image

--------------------

# EFK软件安装 #

**版本说明**

▷ elasticsearch 7.3.2  
▷ filebeat 7.3.2  
▷ kibana 7.3.2

**注意事项**

▷ 三个组件版本必须一致  
▷ elasticsearch必须3台以上且总数量为单数

**安装路径**

▷ /opt/elasticsearch  
▷ /opt/filebeat  
▷ /opt/kibana

**elasticsearch安装**：3台es均执行相同的安装步骤

mkdir -p /opt/software && cd /opt/software
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.2-linux-x86_64.tar.gz
    tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
    mv elasticsearch-7.3.2 /opt/elasticsearch
    useradd elasticsearch -d /opt/elasticsearch -s /sbin/nologin
    mkdir -p /opt/logs/elasticsearch
    chown elasticsearch.elasticsearch /opt/elasticsearch -R
    chown elasticsearch.elasticsearch /opt/logs/elasticsearch -R
    
    # 限制一个进程可以拥有的VMA(虚拟内存区域)的数量要超过262144，不然elasticsearch会报max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
    echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
    sysctl -p

**filebeat安装**

mkdir -p /opt/software && cd /opt/software
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.2-linux-x86_64.tar.gz
    mkdir -p /opt/logs/filebeat/
    tar -zxvf filebeat-7.3.2-linux-x86_64.tar.gz
    mv filebeat-7.3.2-linux-x86_64 /opt/filebeat

**kibana安装**

mkdir -p /opt/software && cd /opt/software
    wget https://artifacts.elastic.co/downloads/kibana/kibana-7.3.2-linux-x86_64.tar.gz
    tar -zxvf kibana-7.3.2-linux-x86_64.tar.gz
    mv kibana-7.3.2-linux-x86_64 /opt/kibana
    useradd kibana -d /opt/kibana -s /sbin/nologin
    chown kibana.kibana /opt/kibana -R

**nginx安装**（用于生成日志，被filebeat收集）

# 只在192.168.1.11安装
    yum install -y nginx
    /usr/sbin/nginx -c /etc/nginx/nginx.conf

--------------------

# elasticsearch配置 #

▷ 192.168.1.31 /opt/elasticsearch/config/elasticsearch.yml

# 集群名字
    cluster.name: my-application
    
    # 节点名字
    node.name: 192.168.1.31
    
    # 日志位置
    path.logs: /opt/logs/elasticsearch
    
    # 本节点访问IP
    network.host: 192.168.1.31
    
    # 本节点访问
    http.port: 9200
    
    # 节点运输端口
    transport.port: 9300
    
    # 集群中其他主机的列表
    discovery.seed_hosts: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 首次启动全新的Elasticsearch集群时，在第一次选举中便对其票数进行计数的master节点的集合
    cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 启用跨域资源共享
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    
    # 只要有2台数据或主节点已加入集群，就可以恢复
    gateway.recover_after_nodes: 2

▷ 192.168.1.32 /opt/elasticsearch/config/elasticsearch.yml

# 集群名字
    cluster.name: my-application
    
    # 节点名字
    node.name: 192.168.1.32
    
    # 日志位置
    path.logs: /opt/logs/elasticsearch
    
    # 本节点访问IP
    network.host: 192.168.1.32
    
    # 本节点访问
    http.port: 9200
    
    # 节点运输端口
    transport.port: 9300
    
    # 集群中其他主机的列表
    discovery.seed_hosts: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 首次启动全新的Elasticsearch集群时，在第一次选举中便对其票数进行计数的master节点的集合
    cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 启用跨域资源共享
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    
    # 只要有2台数据或主节点已加入集群，就可以恢复
    gateway.recover_after_nodes: 2

▷ 192.168.1.33 /opt/elasticsearch/config/elasticsearch.yml

# 集群名字
    cluster.name: my-application
    
    # 节点名字
    node.name: 192.168.1.33
    
    # 日志位置
    path.logs: /opt/logs/elasticsearch
    
    # 本节点访问IP
    network.host: 192.168.1.33
    
    # 本节点访问
    http.port: 9200
    
    # 节点运输端口
    transport.port: 9300
    
    # 集群中其他主机的列表
    discovery.seed_hosts: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 首次启动全新的Elasticsearch集群时，在第一次选举中便对其票数进行计数的master节点的集合
    cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32", "192.168.1.33"]
    
    # 启用跨域资源共享
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    
    # 只要有2台数据或主节点已加入集群，就可以恢复
    gateway.recover_after_nodes: 2

--------------------

# filebeat配置 #

192.168.1.11 /opt/filebeat/filebeat.yml

# 文件输入
    filebeat.inputs:
      # 文件输入类型
      - type: log
        # 开启加载
        enabled: true
        # 文件位置
        paths:
          - /var/log/nginx/access.log
        # 自定义参数
        fields:
          type: nginx_access  # 类型是nginx_access,和上面fields.type是一致的
    
    # 输出至elasticsearch
    output.elasticsearch:
      # elasticsearch集群
      hosts: ["http://192.168.1.31:9200",
              "http://192.168.1.32:9200",
              "http://192.168.1.33:9200"]
    
      # 索引配置
      indices:
        # 索引名
        - index: "nginx_access_%{+yyy.MM}"
          # 当类型是nginx_access时使用此索引
          when.equals:
            fields.type: "nginx_access"
    
    # 关闭自带模板
    setup.template.enabled: false
    
    # 开启日志记录
    logging.to_files: true
    # 日志等级
    logging.level: info
    # 日志文件
    logging.files:
      # 日志位置
      path: /opt/logs/filebeat/
      # 日志名字
      name: filebeat
      # 日志轮转期限，必须要2~1024
      keepfiles: 7
      # 日志轮转权限
      permissions: 0600

--------------------

# kibana配置 #

192.168.1.21 /opt/kibana/config/kibana.yml

# 本节点访问端口
    server.port: 5601
    
    # 本节点IP
    server.host: "192.168.1.21"
    
    # 本节点名字
    server.name: "192.168.1.21"
    
    # elasticsearch集群IP
    elasticsearch.hosts: ["http://192.168.1.31:9200",
                          "http://192.168.1.32:9200",
                          "http://192.168.1.33:9200"]

--------------------

# 启动服务 #

# elasticsearch启动（3台es均启动）
    sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch
    
    # filebeat启动
    /opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml -d "publish"
    
    # kibana启动(如果kibana用户启动不了，则可使用root用户启动，添加参数--allow-root)
    sudo -u kibana /opt/kibana/bin/kibana -c /opt/kibana/config/kibana.yml

上面的启动方法是位于前台运行。systemd配置方法，会在《EFK教程》系列后续文章中提供，敬请关注！

--------------------

# kibana界面配置 #

1️⃣ 使用浏览器访问192.168.1.21:5601，看到以下界面表示启动成功

![format_png 2][]

image

2️⃣ 点"Try our sample data"

![format_png 3][]

image

3️⃣ "Help us improve the Elastic Stack by providing usage statistics for basic features. We will not share this data outside of Elastic"点"no”

![format_png 4][]

image

4️⃣ "Add Data to kibana"点"Add data"

![format_png 5][]

image

5️⃣ 进入视图

![format_png 6][]

image

--------------------

# 测试 #

**访问nginx，生成日志**

curl -I "http://192.168.1.11"

**在kibana上查看数据**

1️⃣ 创建索引模板

![format_png 7][]

image

2️⃣ 输入你要创建的索引模板的名字

![format_png 8][]

image

![format_png 9][]

image

3️⃣ 查看之前CURL的数据

![format_png 10][]

image

![format_png 11][]

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWUwZjkxMWVkNzgxZGY3NzkucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTA1Mi9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTA4ODFkMDI4MTM5YWZiN2EucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTQ3MmIyNWYyNjgzYzQ3OTYucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWZiNjJkMTE5NjgxYmI5NWEucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWU5ZjZiZjBiMjk3ZDQwYzkucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTgzNjllMTZlYzNmZWRhNTcucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWUxYTIzOTBlYzNhYzVhNDcucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTVmNTNiOGQwZWNlNzM4NmEucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTVjZTAyODAwOTU2ZDc4YWQucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTE3Ny9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 9]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWEyZGU5Njc4OTkyODcyMDIucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTE3Ny9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 10]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLWQ0ZGEwZjhjMmVlYmI2MDgucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 11]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85MjA2NTczLTIxMWE4YWRhODU1ZjRjYTAucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png