ELFK 7.x 集群部署

左手的ㄟ右手 2022-10-27 15:49 223阅读 0赞

### ELFK介绍 ###

ELK 是 elastic 公司旗下三款产品ElasticSearch、Logstash、Kibana的首字母组合，也即Elastic Stack包含ElasticSearch、Logstash、Kibana、Beats。ELK提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用，是目前主流的一种日志系统。

ElasticSearch   一个基于 JSON 的分布式的搜索和分析引擎，作为 ELK 的核心，它集中存储数据，
                    用来搜索、分析、存储日志。它是分布式的，可以横向扩容，可以自动发现，索引自动分片
    
    Logstash        一个动态数据收集管道，支持以 TCP/UDP/HTTP 多种方式收集数据（也可以接受 Beats 传输来的数据），
                    并对数据做进一步丰富或提取字段处理。用来采集日志，把日志解析为json格式交给ElasticSearch
    
    Kibana          一个数据可视化组件，将收集的数据进行可视化展示（各种报表、图形化数据），并提供配置、管理 ELK 的界面
    
    Beats           一个轻量型日志采集器，单一用途的数据传输平台，可以将多台机器的数据发送到 Logstash 或 ElasticSearch
    
    X-Pack          一个对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，不过收费

官网：https://www.elastic.co/cn/ ，中文文档：https://elkguide.elasticsearch.cn/

下载elk各组件的旧版本：

https://www.elastic.co/downloads/past-releases

grok正则表达式参考：

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

当前最新版本：7.10.1

--------------------

### 环境准备 ###

环境准备工作请在所有节点上完成。

*  角色划分：

es主节点/es数据节点/kibana/head             192.168.100.132
    
    es主节点/es数据节点/logstash                192.168.100.133
    
    es主节点/es数据节点/filebeat                192.168.100.134

*  关闭防火墙和selinux：

systemctl stop firewalld && systemctl disable firewalld
    
    sed -i 's/=enforcing/=disabled/g' /etc/selinux/config && setenforce 0

*  配置系统：

cat >> /etc/security/limits.conf << EOF * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 EOF
    
    [ `grep 'vm.max_map_count' /etc/sysctl.conf |wc -l` -eq 0 ] && echo 'vm.max_map_count=655360' >> /etc/sysctl.conf
    
    sysctl -p

*  安装Java：

mkdir /software && cd /software             #将jdk安装包放到 /software 下
    
    tar xf jdk-11.0.9_linux-x64_bin.tar.gz && mv jdk-11.0.9 /usr/local/jdk
    
    vim /etc/profile
    
    JAVA_HOME=/usr/local/jdk
    PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
    CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib
    export JAVA_HOME PATH CLASSPATH
    
    source !$
    
    java -version

--------------------

### elastic-head ###

elastic-head 部署工作请在 head 节点上完成。

*  安装docker：

curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo
    
    yum makecache fast
    
    yum install -y docker-ce
    
    systemctl enable docker && systemctl start docker

*  安装elastic-head：

docker pull mobz/elasticsearch-head:5
    
    docker run -d --name head -p 9100:9100 mobz/elasticsearch-head:5
    
    docker ps
    
    CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS                    NAMES
    c0ecb506aeb4   mobz/elasticsearch-head:5   "/bin/sh -c 'grunt s…"   3 seconds ago   Up 2 seconds   0.0.0.0:9100->9100/tcp   head

访问`ip:9100`，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center]

--------------------

### elasticsearch ###

elasticsearch 集群部署工作请在 es 节点上完成。

*  安装elasticsearch：

rpm -ivh https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.1-x86_64.rpm

*  修改配置：

vim /etc/elasticsearch/elasticsearch.yml

192.168.30.132

cluster.name: elk               #集群名，同一集群必须相同
    node.name: elk-132 #指定节点主机名
    node.master: true #允许成为主节点
    node.data: true #数据节点
    path.data: /var/lib/elasticsearch               #数据存放路径
    path.logs: /var/log/elasticsearch               #日志路径
    bootstrap.memory_lock: false #关闭锁定内存，设置为true会报错
    network.host: 192.168.30.132                #监听ip
    http.port: 9200 #http端口
    transport.tcp.port: 9300
    discovery.seed_hosts: ["192.168.30.132", "192.168.30.133", "192.168.30.134"]                #master资格节点列表
    cluster.initial_master_nodes: ["elk-132", "elk-133", "elk-134"]             #集群初始节点列表
    discovery.zen.minimum_master_nodes: 2
    gateway.recover_after_nodes: 3
    http.cors.enabled: true #允许head插件访问es
    http.cors.allow-origin: "*"

192.168.30.133

cluster.name: elk               #集群名，同一集群必须相同
    node.name: elk-133 #指定节点主机名
    node.master: true #允许成为主节点
    node.data: true #数据节点
    path.data: /var/lib/elasticsearch               #数据存放路径
    path.logs: /var/log/elasticsearch               #日志路径
    bootstrap.memory_lock: false #关闭锁定内存，设置为true会报错
    network.host: 192.168.30.133                #监听ip
    http.port: 9200 #http端口
    transport.tcp.port: 9300
    discovery.seed_hosts: ["192.168.30.132", "192.168.30.133", "192.168.30.134"]                #master资格节点列表
    cluster.initial_master_nodes: ["elk-132", "elk-133", "elk-134"]             #集群初始节点列表
    discovery.zen.minimum_master_nodes: 2
    gateway.recover_after_nodes: 3
    http.cors.enabled: true #允许head插件访问es
    http.cors.allow-origin: "*"

192.168.30.134

cluster.name: elk               #集群名，同一集群必须相同
    node.name: elk-134 #指定节点主机名
    node.master: true #允许成为主节点
    node.data: true #数据节点
    path.data: /var/lib/elasticsearch               #数据存放路径
    path.logs: /var/log/elasticsearch               #日志路径
    bootstrap.memory_lock: false #关闭锁定内存，设置为true会报错
    network.host: 192.168.30.134                #监听ip
    http.port: 9200 #http端口
    transport.tcp.port: 9300
    discovery.seed_hosts: ["192.168.30.132", "192.168.30.133", "192.168.30.134"]                #master资格节点列表
    cluster.initial_master_nodes: ["elk-132", "elk-133", "elk-134"]             #集群初始节点列表
    discovery.zen.minimum_master_nodes: 2
    gateway.recover_after_nodes: 3
    http.cors.enabled: true #允许head插件访问es
    http.cors.allow-origin: "*"

在机器负载不是非常高的时候，建议将所有节点设置允许成为主节点和数据节点，实现集群高可用。

*  启动服务：

systemctl enable elasticsearch && systemctl start elasticsearch

*  查看集群状态：

curl '192.168.30.132:9200/_cluster/health?pretty'               #查看集群健康状态

{ 
      "cluster_name" : "elk",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 3,
      "active_primary_shards" : 0,
      "active_shards" : 0,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }

curl '192.168.30.132:9200/_cat/master?v'                #查看集群master
    
    id                     host           ip             node
    oql8XBNFTqSSaiiImwjiEg 192.168.30.133 192.168.30.133 elk-133

curl '192.168.30.132:9200/_cluster/state?pretty'                #查看集群详细信息

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 1]

--------------------

### kibana ###

kibana 部署工作请在 kibana 节点上完成。

*  安装kibana：

rpm -ivh https://artifacts.elastic.co/downloads/kibana/kibana-7.10.1-x86_64.rpm

*  修改配置：

vim /etc/kibana/kibana.yml

server.port: 5601
    server.host: "192.168.30.132"
    elasticsearch.hosts: ["http://192.168.30.132:9200","http://192.168.30.133:9200","http://192.168.30.134:9200"]
    kibana.index: ".kibana"
    elasticsearch.username: elastic
    elasticsearch.password: changeme
    i18n.locale: "zh-CN"

*  启动服务：

systemctl enable kibana && systemctl start kibana

访问`ip:5601`，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 2]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 3]

--------------------

### filebeat ###

filebeat 部署工作请在 filebeat 节点上完成。

*  安装filebeat：

rpm -ivh https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.1-x86_64.rpm

*  收集nginx日志：

yum install -y nginx && systemctl start nginx

*  修改配置：

vim /etc/filebeat/filebeat.yml

filebeat.inputs:
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
        - /var/log/nginx/access.log
      fields:
        type: nginx_access
      fields_under_root: true
      multiline.pattern: '\d+\.\d+\.\d+\.\d+'
      multiline.negate: true
      multiline.match: after
        
    output.logstash:
      hosts: ["192.168.30.133:5040"]
      enabled: true
      worker: 1
      compression_level: 3

*  启动filebeat：

systemctl enable filebeat && systemctl start filebeat

--------------------

### logstash ###

logstash 部署工作请在 logstash 节点上完成。

*  安装logstash：

rpm -ivh https://artifacts.elastic.co/downloads/logstash/logstash-7.10.1-x86_64.rpm

*  修改配置：

vim /etc/logstash/logstash.yml

http.host: "192.168.30.133"
    http.port: 9600
    path.data: /var/lib/logstash
    pipeline.ordered: auto
    path.logs: /var/log/logstash

*  收集nginx日志：

vim /etc/logstash/conf.d/nginx.conf

input { 
        beats { 
            port => 5040
        }
    }
    
    filter { 
        if [type] == "nginx_access" { 
            grok { 
                match => {  "message" => "%{COMBINEDAPACHELOG}" }
            }
        }
    }
    
    output { 
        if [type] == "nginx_access" { 
            elasticsearch { 
                hosts => ["192.168.30.132:9200","192.168.30.133:9200","192.168.30.134:9200"]
                user => "elastic"
                password => "changeme"
                index => "nginx-log"
            }
        }
    }

*  启动logstash：

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf -t              #检查配置文件是否正确
    
    systemctl enable logstash && systemctl start logstash

*  查看索引：

访问nginx页面，以产生日志，

curl '192.168.30.132:9200/_cat/indices?v'
    
    health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   logstash-2020.12.29-000001      18ZOpN6mSiqZYBNzua0rFg   1   1          0            0       416b           208b
    green  open   .apm-custom-link                RJ6l1ijjSt2msGVyKQ0yLA   1   1          0            0       416b           208b
    green  open   nginx-log                       L4MZcw6BSMCZzbItjlElVg   1   1         11            0    248.4kb        124.2kb
    green  open   .kibana_task_manager_1          QrcOB_pNT9Cj_6yGz4-pfg   1   1          5          978    292.4kb        134.9kb
    green  open   .apm-agent-configuration        DCbIKU_SSX-nToV5kSxJpA   1   1          0            0       416b           208b
    green  open   .kibana-event-log-7.10.1-000001 P8WJ1ayXSPmKUbVkXTA6lw   1   1          1            0     11.2kb          5.6kb
    green  open   .kibana_1                       AHRdWUl6QgO6inJ3ypekSg   1   1         45           45      8.5mb          4.2mb

head上查看，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 4]

kibana上 `Stack Management` → `索引模式` → `创建索引模式`，创建索引 `nginx-log`，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 5]

--------------------

### x-pack ###

x-pack 启用工作请在 es 节点上完成。

*  生成证书和私钥：

cd /usr/share/elasticsearch/
    
    /usr/share/elasticsearch/bin/elasticsearch-certutil ca                #直接回车
    
    /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12                #直接回车
    
    cp elastic-certificates.p12 /etc/elasticsearch/
    
    scp elastic-certificates.p12 192.168.30.133:!$ ;scp elastic-certificates.p12 192.168.30.134:!$

如果生成证书设置了密码，需要将密码添加到elasticsearch秘钥库：

/usr/share/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
    
    /usr/share/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

*  修改配置：

vim /etc/elasticsearch/elasticsearch.yml                #追加

http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type             #增加head
    xpack.security.enabled: true #启用x-pack
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

chmod 660 /etc/elasticsearch/elastic-certificates.p12
    
    systemctl restart elasticsearch

*  自定义密码：

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive              #自定义密码，任选一节点执行
    
    Enter password for [elastic]: elk-2021
    Reenter password for [elastic]: elk-2021
    Enter password for [apm_system]: elk-2021
    Reenter password for [apm_system]: elk-2021
    Enter password for [kibana]: elk-2021
    Reenter password for [kibana]: elk-2021
    Enter password for [logstash_system]: elk-2021
    Reenter password for [logstash_system]: elk-2021
    Enter password for [beats_system]: elk-2021
    Reenter password for [beats_system]: elk-2021
    Enter password for [remote_monitoring_user]: elk-2021
    Reenter password for [remote_monitoring_user]: elk-2021

curl -u elastic:elk-2021 -XGET 'http://192.168.30.132:9200/_cluster/health?pretty'

{ 
      "cluster_name" : "elk",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 3,
      "active_primary_shards" : 1,
      "active_shards" : 2,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }

*  修改kibana配置：

vim /etc/kibana/kibana.yml

systemctl restart kibana

刷新kibana网页，可以看到需要登录账号及密码。输入账号：`elastic`、密码：`elk-2021`，登录

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 6]

ELFK 7.x 版本的 `basic` 许可永不过期，因此不需要像 6.x 版本那样破解。

开启x-pack后访问head需要加上账号及密码，

http://192.168.30.132:9100/?auth_user=elastic&auth_password=elk-2021

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 7]

*  修改logstash配置：

vim /etc/logstash/logstash.yml

http.host: "192.168.30.133"
    http.port: 9600
    path.data: /var/lib/logstash
    pipeline.ordered: auto
    path.logs: /var/log/logstash
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: logstash_system
    xpack.monitoring.elasticsearch.password: elk-2021
    xpack.monitoring.elasticsearch.hosts: ["http://192.168.30.132:9200","http://192.168.30.133:9200","http://192.168.30.134:9200"]
    xpack.monitoring.collection.interval: 10s

logstash收集日志时，output部分如果是输出到es，也需要更改账号密码，否则输出会失败。

output { 
        elasticsearch { 
            hosts => ["192.168.30.132:9200","192.168.30.133:9200","192.168.30.134:9200"]
            user => "elastic"
            password => "elk-2021"
        }
    }

systemctl restart logstash

*  修改filebeat配置：

filebeat收集日志时，output部分如果是输出到es，也需要更改账号密码，否则输出会失败。

vim /etc/filebeat/filebeat.yml

output.elasticsearch:
      hosts: ["192.168.30.132:9200","192.168.30.133:9200","192.168.30.134:9200"]
      username: "elastic"
      password: "elk-2021"

systemctl restart filebeat

至此，elfk 7.x 集群部署完成。

--------------------

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center]: /images/20221024/542f2224ffc342ccb957e557f5def73b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221024/cd808e916a7e4ee8938b3beda909c0e1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221024/9d9ca065d8be475d8417c0def0fde83e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221024/e972fce0d12447a5945eff30634ebda0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221024/e8643a4b502840ba85dcbd23993623d0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20221024/4b6f7d5807524ab29270334e01019b98.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 6]: /images/20221024/7d17b96b832a495394b621e050039aca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70_pic_center 7]: /images/20221024/6b88f1e9841845a6b81a6b8e8ed26b4b.png