ELK日志分析平台环境部署

待我称王封你为后i 2022-05-15 05:21 306阅读 0赞

**注释：原理部分粘贴自https://www.cnblogs.com/kevingrace/p/5919021.html**  
**日志概念介绍：**

日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了解服务器软硬件信息、检查配置过程中的错误及
    错误发生的原因。经常分析日志可以了解服务器的负荷，性能安全性，从而及时采取措施纠正错误。
    通常，日志被分散在储存不同的设备上。如果你管理数十上百台服务器，你还在使用依次登录每台机器的传统方法查阅日志。这样是不是
    感觉很繁琐和效率低下。当务之急我们使用集中化的日志管理，例如：开源的syslog，将所有服务器上的日志收集汇总。集中化管理日志
    后，日志的统计和检索又成为一件比较麻烦的事情，一般我们使用grep、awk和wc等Linux命令能实现检索和统计，但是对于要求更高的
    查询、排序和统计等要求和庞大的机器数量依然使用这样的方法难免有点力不从心。

**完整的日志数据体系：**

通过我们需要对日志进行集中化管理，将所有机器上的日志信息收集、汇总到一起。
    完整的日志数据具有非常重要的作用：
    1）信息查找。通过检索日志信息，定位相应的bug，找出解决方案。
    2）服务诊断。通过对日志信息进行统计、分析，了解服务器的负荷和服务运行状态，找出耗时请求进行优化等等。
    3）数据分析。如果是格式化的log，可以做进一步的数据分析，统计、聚合出有意义的信息.

**ELK平台部署：**

开源实时日志分析ELK平台能够完美的解决我们上述的问题，ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成：
    1）ElasticSearch是一个基于Lucene的开源分布式搜索服务器。它的特点有：分布式，零配置，自动发现，索引自动分片，索引
    副本机制，restful风格接口，多数据源自动搜索负载等。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。
    Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是第二流行的企业搜索引擎。设计用于云计算中，
    能够达到实时搜索，稳定，可靠，快速，安装使用方便。在elasticsearch中，所有节点的数据是均等的。
    2）Logstash是一个完全开源的工具，它可以对你的日志进行收集、过滤、分析，支持大量的数据获取方法，并将其存储供以后
    使用（如搜索）。说到搜索，logstash带有一个web界面，搜索和展示所有日志。一般工作方式为c/s架构，client端安装在需要
    收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
    3）Kibana 是一个基于浏览器页面的Elasticsearch前端展示工具，也是一个开源和免费的工具，Kibana可以为 Logstash 和
     ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

**为什么要用到ELK？**

一般我们需要进行日志分析场景是：直接在日志文件中 grep、awk 就可以获得自己想要的信息。但在规模较大的场景中，此方法效率
    低下，面临问题包括日志量太大如何归档、文本搜索太慢怎么办、如何多维度查询。需要集中化的日志管理，所有服务器上的日志收集
    汇总。常见解决思路是建立集中式日志收集系统，将所有节点上的日志统一收集，管理，访问。一般大型系统是一个分布式部署的架构，
    不同的服务模块部署在不同的服务器上，问题出现时，大部分情况需要根据问题暴露的关键信息，定位到具体的服务器和服务模块，
    构建一套集中式日志系统，可以提高定位问题的效率。
    
    一般大型系统是一个分布式部署的架构，不同的服务模块部署在不同的服务器上，问题出现时，大部分情况需要根据问题暴露的关键
    信息，定位到具体的服务器和服务模块，构建一套集中式日志系统，可以提高定位问题的效率。
    一个完整的集中式日志系统，需要包含以下几个主要特点：
    1）收集－能够采集多种来源的日志数据
    2）传输－能够稳定的把日志数据传输到中央系统
    3）存储－如何存储日志数据
    4）分析－可以支持 UI 分析
    5）警告－能够提供错误报告，监控机制
    ELK提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用。目前主流的一种日志系统

**实验环境，必须保证各个主机均有解析：**

server1:master 172.25.38.1
    server2:slave 172.25.38.2
    server3:slave 172.25.38.3

**安装es开源分布式搜索服务器：**

![这里写图片描述][70]  
**进行es的配置：**

[root@server1 ~]# pwd
    /root
    [root@server1 ~]# ls
    bigdesk-master.zip               jemalloc-devel-3.6.0-1.el6.x86_64.rpm
    elasticsearch-2.3.3.rpm          kibana-4.5.1-1.x86_64.rpm
    elasticsearch-head-master.zip    logstash-2.3.3-1.noarch.rpm
    elk日志分析平台.pdf              nginx-1.8.0-1.el6.ngx.x86_64.rpm
    jemalloc-3.6.0-1.el6.x86_64.rpm  redis-3.0.6.tar.gz
    [root@server1 ~]# cd /etc/elasticsearch/
    [root@server1 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server1 elasticsearch]# vim elasticsearch.yml    编辑主配置文件
    [root@server1 elasticsearch]# /etc/init.d/elasticsearch start  无法安装，这是基于java环境下的
    which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
    Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME

**设置cluster和node节点的名字：**  
![这里写图片描述][70 1]  
**设置data和logs路径：**  
![这里写图片描述][70 2]  
**设置主机IP，打开9200端口，打开查询请求：**  
![这里写图片描述][70 3]

[root@server1 elasticsearch]# cd
    [root@server1 ~]# ls
    bigdesk-master.zip               jemalloc-devel-3.6.0-1.el6.x86_64.rpm
    elasticsearch-2.3.3.rpm          kibana-4.5.1-1.x86_64.rpm
    elasticsearch-head-master.zip    logstash-2.3.3-1.noarch.rpm
    elk日志分析平台.pdf              nginx-1.8.0-1.el6.ngx.x86_64.rpm
    jdk-8u121-linux-x64.rpm          redis-3.0.6.tar.gz
    jemalloc-3.6.0-1.el6.x86_64.rpm
    [root@server1 ~]# rpm -ivh jdk-8u121-linux-x64.rpm   安装jdk
    [root@server1 ~]# /etc/init.d/elasticsearch start  开启服务
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 4]  
**可以查看到9200和9300端口：**

[root@server1 ~]# netstat -antlp
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
    tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      908/sshd            
    tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      984/master          
    tcp        0      0 172.25.38.1:22              172.25.38.250:45526         ESTABLISHED 1070/sshd           
    tcp        0      0 ::ffff:172.25.38.1:9200     :::*                        LISTEN      1356/java           
    tcp        0      0 ::ffff:172.25.38.1:9300     :::*                        LISTEN      1356/java           
    tcp        0      0 :::22                       :::*                        LISTEN      908/sshd            
    tcp        0      0 ::1:25                      :::*                        LISTEN      984/master

![这里写图片描述][70 5]  
**在网页访问可以看到集群信息：**  
![这里写图片描述][70 6]  
**调用plugin安装elasticsearch-head-master插件：**

[root@server1 ~]# ls
    bigdesk-master.zip               jemalloc-devel-3.6.0-1.el6.x86_64.rpm
    elasticsearch-2.3.3.rpm          kibana-4.5.1-1.x86_64.rpm
    elasticsearch-head-master.zip    logstash-2.3.3-1.noarch.rpm
    elk日志分析平台.pdf              nginx-1.8.0-1.el6.ngx.x86_64.rpm
    jdk-8u121-linux-x64.rpm          redis-3.0.6.tar.gz
    jemalloc-3.6.0-1.el6.x86_64.rpm
    [root@server1 ~]# /usr/share/elasticsearch/bin/plugin install file:/root/el
    elasticsearch-2.3.3.rpm        elk日志分析平台.pdf
    elasticsearch-head-master.zip  
    [root@server1 ~]# /usr/share/elasticsearch/bin/plugin install file:/root/elasticsearch-head-master.zip   调用plugin安装插件
    [root@server1 ~]# /usr/share/elasticsearch/bin/plugin list  可以列出信息
    Installed plugins in /usr/share/elasticsearch/plugins:
        - head

![这里写图片描述][70 7]

[root@server1 ~]# cd /usr/share/elasticsearch/plugins
    [root@server1 plugins]# ls
    head
    [root@server1 plugins]# cd head/
    [root@server1 head]# ls
    elasticsearch-head.sublime-project  LICENCE                       _site
    Gruntfile.js                        package.json                  src
    grunt_fileSets.js                   plugin-descriptor.properties  test
    index.html                          README.textile

![这里写图片描述][70 8]  
**在网页可以访问查看节点信息：**  
![这里写图片描述][70 9]  
**在复合查询之中可以提交请求：**  
![这里写图片描述][70 10]  
**黄色代表主分片未丢失。红色代表主分片丢失，绿色表示未提交**  
![这里写图片描述][70 11]  
**在server2和server3安装服务：**  
![这里写图片描述][70 12]  
![这里写图片描述][70 13]  
**在server配置集群节点信息：**

[root@server1 ~]# cd /etc/elasticsearch/
    [root@server1 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server1 elasticsearch]# vim elasticsearch.yml   设置集群节点

![这里写图片描述][70 14]

[root@server1 elasticsearch]# scp elasticsearch.yml server2:/etc/elasticsearch/  将文件传递给从节点
    root@server2's password: 
    elasticsearch.yml                             100% 3197     3.1KB/s   00:00    
    [root@server1 elasticsearch]# scp elasticsearch.yml server3:/etc/elasticsearch/
    root@server3's password: 
    elasticsearch.yml                             100% 3197     3.1KB/s   00:00    
    [root@server1 elasticsearch]# /etc/init.d/elasticsearch reload  修改完配置文件记得重载服务
    Stopping elasticsearch:                                    [  OK  ]
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 15]  
**配置server2和server3节点：**

[root@server2 ~]# cd /etc/elasticsearch/
    [root@server2 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server2 elasticsearch]# vim elasticsearch.yml  更改IP和主机名
    [root@server2 elasticsearch]# /etc/init.d/elasticsearch start  开启服务
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 16]  
![这里写图片描述][70 17]

[root@server3 ~]# cd /etc/elasticsearch/
    [root@server3 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server3 elasticsearch]# vim elasticsearch.yml  更改IP和主机名
    [root@server3 elasticsearch]# /etc/init.d/elasticsearch start  开启服务
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 18]  
![这里写图片描述][70 19]  
**在网页访问刷新就可以看到节点分布：**  
![这里写图片描述][70 20]  
**配置节点信息使主从节点分工明确：**

[root@server1 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server1 elasticsearch]# vim elasticsearch.yml  配置master端不收集信息
    [root@server1 elasticsearch]# /etc/init.d/elasticsearch reload  重载服务
    Stopping elasticsearch:                                    [  OK  ]
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 21]  
**配置从端进行接受信息：**

[root@server2 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server2 elasticsearch]# vim elasticsearch.yml 
    [root@server2 elasticsearch]# /etc/init.d/elasticsearch reload
    Stopping elasticsearch:                                    [  OK  ]
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 22]

[root@server3 elasticsearch]# ls
    elasticsearch.yml  logging.yml  scripts
    [root@server3 elasticsearch]# vim elasticsearch.yml 
    [root@server3 elasticsearch]# /etc/init.d/elasticsearch reload
    Stopping elasticsearch:                                    [  OK  ]
    Starting elasticsearch:                                    [  OK  ]

![这里写图片描述][70 23]  
**在网页访问进行刷新可以看到主从节点功能分离：**  
![这里写图片描述][70 24]  
**curl命令测试集群节点信息：**

[root@server1 elasticsearch]# curl -XGET 'http://172.25.38.1:9200/_cluster/health?pretty=true'
    {
      "cluster_name" : "my-es",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 2,
      "active_primary_shards" : 5,
      "active_shards" : 10,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }

![这里写图片描述][70 25]  
**在网页删除索引的方式直接点击节点出来提示根据提示删除即可：**  
![这里写图片描述][70 26]  
**调用命令删除节点：**

[root@server1 elasticsearch]# curl -XDELETE 'http://172.25.38.1:9200/index'
    {"acknowledged":true}[root@server1 elasticsearch]#

![这里写图片描述][70 27]  
**在网页查看节点已经被删除：**  
![这里写图片描述][70 28]  
**配置日志数据处理的工具logstash:**

[root@server1 ~]# ls
    bigdesk-master.zip               jemalloc-devel-3.6.0-1.el6.x86_64.rpm
    elasticsearch-2.3.3.rpm          kibana-4.5.1-1.x86_64.rpm
    elasticsearch-head-master.zip    logstash-2.3.3-1.noarch.rpm
    elk日志分析平台.pdf              nginx-1.8.0-1.el6.ngx.x86_64.rpm
    jdk-8u121-linux-x64.rpm          redis-3.0.6.tar.gz
    jemalloc-3.6.0-1.el6.x86_64.rpm
    [root@server1 ~]# rpm -ivh logstash-2.3.3-1.noarch.rpm 
    Preparing...                ########################################### [100%]
       1:logstash               ########################################### [100%]
    [root@server1 ~]# cd /etc/logstash/
    [root@server1 logstash]# ls
    conf.d
    [root@server1 logstash]# cd conf.d/
    [root@server1 conf.d]# ls
    [root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# cd /opt/logstash/
    [root@server1 logstash]# ls
    bin           CONTRIBUTORS  Gemfile.jruby-1.9.lock  LICENSE     vendor
    CHANGELOG.md  Gemfile       lib                     NOTICE.TXT

![这里写图片描述][70 29]

[root@server1 logstash]# cd bin/
    [root@server1 bin]# ls
    logstash      logstash.lib.sh  logstash-plugin.bat  plugin.bat  rspec.bat
    logstash.bat  logstash-plugin  plugin               rspec       setup.bat
    [root@server1 bin]# /opt/logstash/bin/logstash -e 'input { stdin {} } output{ stdout {} }'
    ^[[ASettings: Default pipeline workers: 1      调用终端录入终端输出的方式来处理数据
    Pipeline main started
    hello
    2018-08-25T02:42:39.801Z server1 hello
    westos
    2018-08-25T02:42:43.917Z server1 westos
    xfl
    2018-08-25T02:42:47.220Z server1 xfl
    ^CSIGINT received. Shutting down the agent. {:level=>:warn}   按ctrl+c可以结束
    stopping pipeline {:id=>"main"}
    Received shutdown signal, but pipeline is still waiting for in-flight events
    to be processed. Sending another ^C will force quit Logstash, but this may cause
    data loss. {:level=>:warn}
    ^CSIGINT received. Terminating immediately.. {:level=>:fatal}

![这里写图片描述][70 30]

[root@server1 bin]# /opt/logstash/bin/logstash -e 'input { stdin {} } output{ stdout { codec => rubydebug } }'   调用转换格式对终端录入内容进行转换输出
    Settings: Default pipeline workers: 1
    Pipeline main started
    hello
    {
           "message" => "hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T02:44:54.249Z",
              "host" => "server1"
    }
    westos
    {
           "message" => "westos",
          "@version" => "1",
        "@timestamp" => "2018-08-25T02:44:57.160Z",
              "host" => "server1"
    }
    ^CSIGINT received. Shutting down the agent. {:level=>:warn}
    stopping pipeline {:id=>"main"}
    ^CSIGINT received. Terminating immediately.. {:level=>:fatal}
    ^CSIGINT received. Terminating immediately.. {:level=>:fatal}

![这里写图片描述][70 31]  
![这里写图片描述][70 32]  
**调用脚本进行转换格式输出：**

[root@server1 bin]# cd /etc/logstash/conf.d/
    [root@server1 conf.d]# ls
    [root@server1 conf.d]# vim es.conf
    [root@server1 conf.d]# cat es.conf 
    input {
    	stdin {}
    }
    
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "logstash-%{+YYYY.MM.dd}"
    	}
    	
    	stdout {
    		codec => rubydebug
    	}
    }

![这里写图片描述][70 33]

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf 
    Settings: Default pipeline workers: 1   每次提交会转换格式输出
    Pipeline main started
    westos
    {
           "message" => "westos",
          "@version" => "1",
        "@timestamp" => "2018-08-25T02:54:13.778Z",
              "host" => "server1"
    }
    linux
    {
           "message" => "linux",
          "@version" => "1",
        "@timestamp" => "2018-08-25T02:54:17.213Z",
              "host" => "server1"
    }
    redhat
    {
           "message" => "redhat",
          "@version" => "1",
        "@timestamp" => "2018-08-25T02:54:21.037Z",
              "host" => "server1"

![这里写图片描述][70 34]  
![这里写图片描述][70 35]  
**将输出结果输出到文件中：**

[root@server1 conf.d]# ls
    es.conf
    [root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# vim es.conf 
    [root@server1 conf.d]# cat es.conf 
    input {
    	stdin {}
    }
    
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "logstash-%{+YYYY.MM.dd}"
    	}
    	
    	stdout {
    		codec => rubydebug
    	}
    	file {
    		path => "/tmp/testfile"
    		codec => line { format => "custom format: %{message}"}
    	}
    }

![这里写图片描述][70 36]  
**调用脚本进行终端输入来查看文件内容：**

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started
    xfl
    {
           "message" => "xfl",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:20:05.713Z",
              "host" => "server1"
    }
    xfl
    {
           "message" => "xfl",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:20:08.839Z",
              "host" => "server1"
    }
    ^CSIGINT received. Shutting down the agent. {:level=>:warn}
    stopping pipeline {:id=>"main"}
    ^CSIGINT received. Terminating immediately.. {:level=>:fatal}
    [root@server1 conf.d]# cd /tmp/
    [root@server1 tmp]# ls
    hsperfdata_elasticsearch  hsperfdata_root  jna--1985354563  testfile  yum.log
    [root@server1 tmp]# cat testfile   查看文件可以看到提交信息
    custom format: xfl
    custom format: xfl

![这里写图片描述][70 37]  
**在网页点击数据浏览可以看到提交信息：**  
![这里写图片描述][70 38]

[root@server1 log]# pwd
    /var/log
    [root@server1 log]# ls
    anaconda.ifcfg.log    btmp           lastlog            secure-20180825
    anaconda.log          btmp-20180825  logstash           spooler
    anaconda.program.log  cron           maillog            spooler-20180825
    anaconda.storage.log  cron-20180825  maillog-20180825   tallylog
    anaconda.syslog       dmesg          messages           wtmp
    anaconda.yum.log      dmesg.old      messages-20180825  yum.log
    audit                 dracut.log     rhsm
    boot.log              elasticsearch  secure
    [root@server1 log]# cd elasticsearch/
    [root@server1 elasticsearch]# ls
    my-es_deprecation.log             my-es_index_search_slowlog.log
    my-es_index_indexing_slowlog.log  my-es.log
    [root@server1 elasticsearch]# ll
    total 560
    -rw-r--r-- 1 elasticsearch elasticsearch      0 Aug 25 09:30 my-es_deprecation.log
    -rw-r--r-- 1 elasticsearch elasticsearch      0 Aug 25 09:30 my-es_index_indexing_slowlog.log
    -rw-r--r-- 1 elasticsearch elasticsearch      0 Aug 25 09:30 my-es_index_search_slowlog.log
    -rw-r--r-- 1 elasticsearch elasticsearch 568080 Aug 25 11:22 my-es.log
    [root@server1 elasticsearch]# cd ..
    [root@server1 log]# ll messages
    -rw------- 1 root root 448 Aug 25 09:25 messages
    [root@server1 log]# ll -d messages
    -rw------- 1 root root 448 Aug 25 09:25 messages

![这里写图片描述][70 39]  
**直接将日志输出到es网页：**

[root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# ls
    es.conf  message.conf
    [root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# vim es.conf 
    [root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# cat message.conf 
    input {
    	file {
    		path => "/var/log/messages"
    		start_position => "beginning"
    	}
    }
    
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "message-%{+YYYY.MM.dd}"
    	}
    	
    }

![这里写图片描述][70 40]  
**进行调用脚本测试不关闭终端：**

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started

![这里写图片描述][70 41]  
**在网页刷新可以看到logstash信息：**  
![这里写图片描述][70 42]  
**查看数据浏览有信息：**  
![这里写图片描述][70 43]  
**重新打开一个终端进行提交日志内容：**

[root@server1 ~]# logger hello
    [root@server1 ~]# logger hello
    [root@server1 ~]# logger hello
    [root@server1 ~]# logger hello
    [root@server1 ~]# logger hello
    [root@server1 ~]# logger hello
    [root@server1 ~]# logger hello

![这里写图片描述][70 44]  
**在网页查看数据浏览会实时同步：**  
![这里写图片描述][70 45]  
**在之前打开的终端可以看到提交的信息：**

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started
    {
           "message" => "Aug 25 11:38:47 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:43.157Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 11:38:48 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:45.004Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 11:38:48 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:45.005Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 11:38:49 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:45.005Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 11:38:49 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:45.005Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 11:38:50 server1 root: hello",
          "@version" => "1",
        "@timestamp" => "2018-08-25T03:41:45.006Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }

![这里写图片描述][70 46]

[root@server1 ~]# l.
    .              .bash_logout   .cshrc                                     .ssh
    ..             .bash_profile  .oracle_jre_usage                          .tcshrc
    .bash_history  .bashrc        .sincedb_452905a167cf4509fd08acb964fdb20c  .viminfo
    [root@server1 ~]# cat .sincedb_452905a167cf4509fd08acb964fdb20c 
    523024 0 64768 774
    [root@server1 ~]# cat .sincedb_452905a167cf4509fd08acb964fdb20c  查看隐藏文件发生变更了才会读取日志文件，不然一直提交不上去
    523024 0 64768 774
    [root@server1 ~]# logger westos  提交日志
    [root@server1 ~]# cat .sincedb_452905a167cf4509fd08acb964fdb20c  查看已经发生改变
    523024 0 64768 922

![这里写图片描述][70 47]  
**会实时出来数据：**  
![这里写图片描述][70 48]  
**搭建syslog日志进行远程传输：**

[root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# cat message.conf 
    input {
    	syslog {
    		port => 514
    	}
    
    }
    
    output {
    #	elasticsearch {
    #		hosts => ["172.25.38.1"]
    #		index => "message-%{+YYYY.MM.dd}"
    #	}
    
    	stdout {
    		codec => rubydebug
    	}
    }
    [root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started

![这里写图片描述][70 49]  
**在另外一台终端查看端口可以看到514端口：**

[root@server1 ~]# netstat -antulp | grep :514
    tcp        0      0 :::514                      :::*                        LISTEN      2104/java           
    udp        0      0 :::514                      :::*                                    2104/java

![这里写图片描述][70 50]  
**设置日志的远程传输：**

[root@server2 elasticsearch]# vim /etc/rsyslog.conf 
    [root@server2 elasticsearch]# cat /etc/rsyslog.conf | tail -n 1
    *.*  @@172.25.38.1:514    指定server1的IP，UDP格式
    [root@server2 elasticsearch]# /etc/init.d/rsyslog restart  重启服务
    Shutting down system logger:                               [  OK  ]
    Starting system logger:                                    [  OK  ]

![这里写图片描述][70 51]  
**在serevr1可以同步server2的信息：**

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started
    {
               "message" => "imklog 5.8.10, log source = /proc/kmsg started.\n",
              "@version" => "1",
            "@timestamp" => "2018-08-25T03:50:24.000Z",
                  "host" => "172.25.38.2",
              "priority" => 6,
             "timestamp" => "Aug 25 11:50:24",
             "logsource" => "server2",
               "program" => "kernel",
              "severity" => 6,
              "facility" => 0,
        "facility_label" => "kernel",
        "severity_label" => "Informational"
    }
    {
               "message" => "[origin software=\"rsyslogd\" swVersion=\"5.8.10\" x-pid=\"1828\" x-info=\"http://www.rsyslog.com\"] start\n",
              "@version" => "1",
            "@timestamp" => !["2018-08-25T03:50:24.000Z",
                  "host" =>](https://img-blog.csdn.net/20180825115110859?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FhYWFhYWJf/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70) "172.25.38.2",
              "priority" => 46,
             "timestamp" => "Aug 25 11:50:24",
             "logsource" => "server2",
               "program" => "rsyslogd",
              "severity" => 6,
              "facility" => 5,
        "facility_label" => "syslogd",
        "severity_label" => "Informational"
    }

![这里写图片描述][70 52]  
**在server2提交日志：**

[root@server2 elasticsearch]# logger hello world

**在server1可以实时同步数据：**  
![这里写图片描述][70 53]  
**在网页可以查看数据浏览信息：**  
![这里写图片描述][70 54]  
**删除messages节点：**  
![这里写图片描述][70 55]  
**日志递送原理：**

[root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# ls
    es.conf  message.conf
    [root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# cat message.conf 
    input {
    	file {
    		path => "/var/log/messages"
    		start_position => "beginning"
    	}
    
    }
    
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "message-%{+YYYY.MM.dd}"
    	}
    
    	stdout {
    		codec => rubydebug
    	}
    }
    [root@server1 conf.d]# cd

![这里写图片描述][70 56]

[root@server1 ~]# logger helloworld  必须提交日志使得.sincedb_452905a167cf4509fd08acb964fdb20c数值变化才会重新读取
    [root@server1 ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started
    {
           "message" => "Aug 25 13:38:02 server1 kernel: Clocksource tsc unstable (delta = -1101891417301 ns)",
          "@version" => "1",
        "@timestamp" => "2018-08-25T05:59:45.358Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }
    {
           "message" => "Aug 25 13:58:16 server1 root: helloworld",
          "@version" => "1",
        "@timestamp" => "2018-08-25T05:59:51.707Z",
              "path" => "/var/log/messages",
              "host" => "server1"
    }

![这里写图片描述][70 57]  
**messages节点会重新出现：**  
![这里写图片描述][70 58]  
**日志的多行合并：**

[root@server1 ~]# /var/log/elasticsearch/my-es.log 查看日志格式特别乱

![这里写图片描述][70 59]

[root@server1 ~]# vim /etc/logstash/conf.d/message.conf 
    [root@server1 ~]# cat /etc/logstash/conf.d/message.conf
    input {
    	file {
    		path => "/var/log/elasticsearch/my-es.log"  将日志整合
    		start_position => "beginning"  从开头整合
    	}
    
    }
    
    filter {
      multiline {
    #    type => "type"
        pattern => "^\["   将[开头的日志进行合并
        negate => true
        what => "previous"
    	}
    }
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "es-%{+YYYY.MM.dd}"
    	}
    
    	stdout {
    		codec => rubydebug
    	}
    }

**执行命令查看日志的合并：**

[root@server1 ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    
    Settings: Default pipeline workers: 1
    Pipeline main started

![这里写图片描述][70 60]  
**httpd数据采集：**

[root@server1 ~]# yum install httpd -y  安装http服务
    [root@server1 ~]# cd /var/www/html/
    [root@server1 html]# ls
    [root@server1 html]# vim index.html
    [root@server1 html]# cat index.html   编写测试页
    <h1>hello xfl</h1>
    [root@server1 html]# /etc/init.d/httpd start  开启服务
    Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.38.1 for ServerName
                                                               [  OK  ]

![这里写图片描述][70 61]  
**在网页测试阿帕奇访问：**  
![这里写图片描述][70 62]

[root@server1 html]# cd /var/log/httpd/
    [root@server1 httpd]# ls
    access_log  error_log
    [root@server1 httpd]# ll
    total 8
    -rw-r--r-- 1 root root 462 Aug 25 14:37 access_log
    -rw-r--r-- 1 root root 544 Aug 25 14:37 error_log
    [root@server1 httpd]# cd /etc/logstash/
    [root@server1 logstash]# ls
    conf.d
    [root@server1 logstash]# cd conf.d/
    [root@server1 conf.d]# ls
    es.conf  message.conf

![这里写图片描述][70 63]  
**编写测试阿帕其脚本：**

[root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# cat message.conf 
    input {
    	file {
    		path => ["/var/log/httpd/access_log","/var/log/httpd/error_log"]
    		start_position => "beginning"
    	}
    
    }
    
    #filter {
    #  multiline {
    ##    type => "type"
    #    pattern => "^\["
    #    negate => true
    #    what => "previous"
    #	}
    #}
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "apache-%{+YYYY.MM.dd}"
    	}
    
    	stdout {
    		codec => rubydebug
    	}
    }
    [root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started

![这里写图片描述][70 64]  
**在网页可以看到阿帕其的信息：**  
![这里写图片描述][70 65]  
**进行日志的切片：**

[root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# ls
    es.conf  message.conf  test.conf
    [root@server1 conf.d]# vim test.conf 
    [root@server1 conf.d]# cat test.conf 
    input {
    	stdin {}
    }
    filter {
      grok {
        match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }   定义格式
      }
    }
    
    output {
    	stdout {
    		codec => rubydebug
    	}
    }
    [root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started
    55.3.244.1 GET /index.html 15824 0.043   输入指定格式
    {
           "message" => "55.3.244.1 GET /index.html 15824 0.043",
          "@version" => "1",
        "@timestamp" => "2018-08-25T07:34:15.257Z",   会自动按照切片形式输出
              "host" => "server1",     
            "client" => "55.3.244.1",
            "method" => "GET",
           "request" => "/index.html",
             "bytes" => "15824",
          "duration" => "0.043"
    }

![这里写图片描述][70 66]  
**在网页删除阿帕其节点进行再次推送：**

[root@server1 conf.d]# ls
    es.conf  message.conf  test.conf
    [root@server1 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server1 conf.d]# vim message.conf 
    [root@server1 conf.d]# cat message.conf 
    input {
    	file {
    		path => ["/var/log/httpd/access_log","/var/log/httpd/error_log"]
    		start_position => "beginning"
    	}
    
    }
    
    filter {
    	grok {
    		match => { "message" => "%{COMBINEDAPACHELOG}" }
    	}
    }
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "apache-%{+YYYY.MM.dd}"
    	}
    
    	stdout {
    		codec => rubydebug
    	}
    }

[root@server1 conf.d]# ls -i /var/log/httpd/error_log 
    523011 /var/log/httpd/error_log
    [root@server1 conf.d]# ls -i /var/log/httpd/access_log 
    527862 /var/log/httpd/access_log
    [root@server1 conf.d]# cd
    [root@server1 ~]# cat .sincedb_ef0edb00900aaa8dcb520b280cb2fb7d 
    527862 0 64768 462
    523011 0 64768 544
    [root@server1 ~]# rm -f .sincedb_ef0edb00900aaa8dcb520b280cb2fb7d  要不删除要不提交日志使数值改变不然没办法推送
    [root@server1 ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/message.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started

![这里写图片描述][70 67]  
**在网页可以看到httpd日志信息：**  
![这里写图片描述][70 68]  
**数据展示：**  
**在sever3安装kibana:**

[root@server3 ~]# ls
    elasticsearch-2.3.3.rpm  jdk-8u121-linux-x64.rpm  kibana-4.5.1-1.x86_64.rpm
    [root@server3 ~]# rpm -ivh kibana-4.5.1-1.x86_64.rpm 
    Preparing...                ########################################### [100%]
       1:kibana                 ########################################### [100%]
    [root@server3 ~]# cd /opt/
    [root@server3 opt]# ls
    kibana
    [root@server3 opt]# cd kibana/
    [root@server3 kibana]# ls
    bin     installedPlugins  node          optimize      README.txt  webpackShims
    config  LICENSE.txt       node_modules  package.json  src
    [root@server3 kibana]# cd config/
    [root@server3 config]# ls
    kibana.yml
    [root@server3 config]# vim kibana.yml 写入master的IP打开kibanna索引
    [root@server3 config]# /etc/init.d/kibana start  开启服务
    kibana started
    [root@server3 config]# netstat -antlp | grep :5601   查看监听端口5601的开启
    tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1921/node

![这里写图片描述][70 69]  
**在网页输入172.25.38.3:5601来加载kabana：**  
![这里写图片描述][70 70]  
**拉到最下方点击create:**  
![这里写图片描述][70 71]  
![这里写图片描述][70 72]  
**设置时间：**  
![这里写图片描述][70 73]  
**搭建redis消息队列进行联动：**

[root@server2 ~]# ls
    elasticsearch-2.3.3.rpm  jdk-8u121-linux-x64.rpm  redis-3.0.6.tar.gz
    [root@server2 ~]# tar zxf redis-3.0.6.tar.gz 
    [root@server2 ~]# cd redis-3.0.6
    [root@server2 redis-3.0.6]# ls
    00-RELEASENOTES  COPYING  Makefile   redis.conf       runtest-sentinel  tests
    BUGS             deps     MANIFESTO  runtest          sentinel.conf     utils
    CONTRIBUTING     INSTALL  README     runtest-cluster  src
    [root@server2 redis-3.0.6]# yum install -y gcc  安装gcc编译工具
    [root@server2 redis-3.0.6]# make && make install  源码编译

![这里写图片描述][70 74]

[root@server2 redis-3.0.6]# ls
    00-RELEASENOTES  COPYING  Makefile   redis.conf       runtest-sentinel  tests
    BUGS             deps     MANIFESTO  runtest          sentinel.conf     utils
    CONTRIBUTING     INSTALL  README     runtest-cluster  src
    [root@server2 redis-3.0.6]# cd utils/
    [root@server2 utils]# ./install_server.sh   进行初始化安装
    Welcome to the redis service installer
    This script will help you easily set up a running redis server
    
    Please select the redis port for this instance: [6379] 
    Selecting default: 6379
    Please select the redis config file name [/etc/redis/6379.conf] 
    Selected default - /etc/redis/6379.conf
    Please select the redis log file name [/var/log/redis_6379.log] 
    Selected default - /var/log/redis_6379.log
    Please select the data directory for this instance [/var/lib/redis/6379] 
    Selected default - /var/lib/redis/6379
    Please select the redis executable path [/usr/local/bin/redis-server] 
    Selected config:
    Port           : 6379
    Config file    : /etc/redis/6379.conf
    Log file       : /var/log/redis_6379.log
    Data dir       : /var/lib/redis/6379
    Executable     : /usr/local/bin/redis-server
    Cli Executable : /usr/local/bin/redis-cli
    Is this ok? Then press ENTER to go on or Ctrl-C to abort.
    Copied /tmp/6379.conf => /etc/init.d/redis_6379
    Installing service...
    Successfully added to chkconfig!
    Successfully added to runlevels 345!
    Starting Redis server...
    Installation successful!

![这里写图片描述][70 75]

[root@server2 utils]# netstat -antlp|grep :6379  查看redis监听端口
    tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      4860/redis-server * 
    tcp        0      0 :::6379                     :::*                        LISTEN      4860/redis-server *

![这里写图片描述][70 76]

[root@server1 ~]# /etc/init.d/httpd stop  停止阿帕其
    Stopping httpd:                                            [  OK  ]
    [root@server1 ~]# rpm -ivh nginx-1.8.0-1.el6.ngx.x86_64.rpm  安装nginx服务
    warning: nginx-1.8.0-1.el6.ngx.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 7bd9bf62: NOKEY
    Preparing...                ########################################### [100%]
       1:nginx                  ########################################### [100%]
    ----------------------------------------------------------------------
    
    Thanks for using nginx!
    
    Please find the official documentation for nginx here:
    * http://nginx.org/en/docs/
    
    Commercial subscriptions for nginx are available on:
    * http://nginx.com/products/
    
    ----------------------------------------------------------------------
    [root@server1 ~]# cd /etc/nginx/
    [root@server1 nginx]# ls
    conf.d          koi-utf  mime.types  scgi_params   win-utf
    fastcgi_params  koi-win  nginx.conf  uwsgi_params
    [root@server1 nginx]# cd /etc/logstash/conf.d/
    [root@server1 conf.d]# ls
    es.conf  message.conf  test.conf
    [root@server1 conf.d]# cp message.conf nginx.cof

![这里写图片描述][70 77]  
**编写nginx测试脚本：**

[root@server1 conf.d]# ls
    es.conf  message.conf  nginx.conf  test.conf
    [root@server1 conf.d]# cat nginx.conf 
    input {
    	file {
    		path => "/var/log/nginx/access_log"
    		start_position => "beginning"
    	}
    
    }
    
    filter {
    	grok {
    		match => { "message" => "%{COMBINEDAPACHELOG}" }
    	}
    }
    output {
    	redis {
    		host => ["172.25.38.2"]   写入redis监听端口IP
    		port => 6379  默认端口
    		data_type => "list"
    		key => "logstash:redis"
    	}	
    
    	stdout {
    		codec => rubydebug
    	}
    }

**进行调用脚本测试：**

[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf 
    在真机进行压测：
    [kiosk@foundation38 Desktop]$ ab -c 1 -n 10 http://172.25.38.1/index.html  写入master端的IP
    This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    
    Benchmarking 172.25.38.1 (be patient).....done
    
    
    Server Software:        nginx/1.8.0
    Server Hostname:        172.25.38.1
    Server Port:            80
    
    Document Path:          /index.html
    Document Length:        612 bytes
    
    Concurrency Level:      1
    Time taken for tests:   0.027 seconds
    Complete requests:      10
    Failed requests:        0
    Write errors:           0
    Total transferred:      8440 bytes
    HTML transferred:       6120 bytes
    Requests per second:    363.99 [#/sec] (mean)
    Time per request:       2.747 [ms] (mean)
    Time per request:       2.747 [ms] (mean, across all concurrent requests)
    Transfer rate:          300.01 [Kbytes/sec] received
    
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.1      0       1
    Processing:     0    2   5.3      1      17
    Waiting:        0    2   5.3      0      17
    Total:          1    3   5.4      1      18
    
    Percentage of the requests served within a certain time (ms)
      50%      1
      66%      1
      75%      1
      80%      2
      90%     18
      95%     18
      98%     18
      99%     18
     100%     18 (longest request)

![这里写图片描述][70 78]  
**在serevr2安装logstash:**

[root@server2 ~]# ls
    elasticsearch-2.3.3.rpm  logstash-2.3.3-1.noarch.rpm  redis-3.0.6.tar.gz
    jdk-8u121-linux-x64.rpm  redis-3.0.6
    [root@server2 ~]# rpm -ivh logstash-2.3.3-1.noarch.rpm 
    Preparing...                ########################################### [100%]
       1:logstash               ########################################### [100%]%))
    [root@server2 ~]# cd /etc/logstash/
    [root@server2 logstash]# ls
    conf.d
    [root@server2 logstash]# cd conf.d/
    [root@server2 conf.d]# ls
    [root@server2 conf.d]# pwd
    /etc/logstash/conf.d
    [root@server2 conf.d]# ls
    es.conf

![这里写图片描述][70 79]

[root@server2 conf.d]# vim es.conf 
    [root@server2 conf.d]# cat es.conf 
    input {
    	redis {
    		host => "172.25.38.2"  redis的IP
    		port => 6379   端口号
    		data_type => "list"
    		key => "logstash:redis"
    }
    }
    output {
    	elasticsearch {
    		hosts => ["172.25.38.1"]
    		index => "nginx-%{+YYYY.MM.dd}"
    	}
    }

![这里写图片描述][70 80]  
**调用脚本挂在后台即可：**

[root@server2 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf 
    Settings: Default pipeline workers: 1
    Pipeline main started

**在网页可以看到nginx和kibana的节点信息：**  
![这里写图片描述][70 81]  
**进去搭建图形来展示日志信息：**  
![这里写图片描述][70 82]  
![这里写图片描述][70 83]  
![这里写图片描述][70 84]  
![这里写图片描述][70 85]  
![这里写图片描述][70 86]  
![这里写图片描述][70 87]  
**两个图表搭建成功：**  
![这里写图片描述][70 88]  
**当有人访问时会有数据变化：**  
![这里写图片描述][70 89]  
**在真机做压侧：**

[kiosk@foundation38 Desktop]$ ab -c 1 -n 10 http://172.25.38.1/index.html
    This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    
    Benchmarking 172.25.38.1 (be patient).....done
    
    
    Server Software:        nginx/1.8.0
    Server Hostname:        172.25.38.1
    Server Port:            80
    
    Document Path:          /index.html
    Document Length:        612 bytes
    
    Concurrency Level:      1
    Time taken for tests:   0.013 seconds
    Complete requests:      10
    Failed requests:        0
    Write errors:           0
    Total transferred:      8440 bytes
    HTML transferred:       6120 bytes
    Requests per second:    769.53 [#/sec] (mean)
    Time per request:       1.299 [ms] (mean)
    Time per request:       1.299 [ms] (mean, across all concurrent requests)
    Transfer rate:          634.26 [Kbytes/sec] received
    
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    1   0.3      1       1
    Processing:     1    1   0.2      1       1
    Waiting:        0    1   0.2      0       1
    Total:          1    1   0.3      1       2
    ERROR: The median and mean for the waiting time are more than twice the standard
           deviation apart. These results are NOT reliable.
    
    Percentage of the requests served within a certain time (ms)
      50%      1
      66%      1
      75%      1
      80%      1
      90%      2
      95%      2
      98%      2
      99%      2
     100%      2 (longest request)

**配置柱形图进行直观的观察访问情况：**  
![这里写图片描述][70 90]  
![这里写图片描述][70 91]  
![这里写图片描述][70 92]

[70]: /images/20220515/98d06b11c8694fa7bac0c09d8e799aa5.png
[70 1]: /images/20220515/d8a3cb00df8d42d3b76e8ac71a92612f.png
[70 2]: /images/20220515/b7c68e7830d84c6d9ef015aac7f1d6f2.png
[70 3]: /images/20220515/a0a577551b0e43e2a5366cb5946f8ce7.png
[70 4]: /images/20220515/7319bcff3c304483a92f3507558c3813.png
[70 5]: /images/20220515/ffe8ae647be44d879f30e6ba03221fe2.png
[70 6]: /images/20220515/55b82234d53a467b9752051f4171edae.png
[70 7]: /images/20220515/1d2f87c72f42429f827c36700e5d0c6f.png
[70 8]: /images/20220515/f7cedcfd569e48259a3b4a9115736e05.png
[70 9]: /images/20220515/14f8fec8a5d34608a94d0f147a3a477a.png
[70 10]: /images/20220515/6f9f541395a94209bcfaf3e940412881.png
[70 11]: /images/20220515/d68e97285c484430a2c82649226a16fc.png
[70 12]: /images/20220515/02f33ee497b14e938049aa5324039688.png
[70 13]: /images/20220515/8b14bd114adc4e89949146d3a0b1a75c.png
[70 14]: /images/20220515/97d8c73d8e324e61afcb549a02b4ded1.png
[70 15]: /images/20220515/cb24e463feb84331adbe650d4b8bd6fd.png
[70 16]: /images/20220515/7b3cb804cab74e49a058b3546aa58f90.png
[70 17]: /images/20220515/a32919a17a684585bf29c36a3400a08e.png
[70 18]: /images/20220515/cf7a712402ae4c26aaa993a8e4c0de14.png
[70 19]: /images/20220515/4b81ef581bbc410aa7d86b7acdfcfd96.png
[70 20]: /images/20220515/68f3d0821ef14fe4970a32f28ebf918d.png
[70 21]: /images/20220515/499ae62e340a4afab820228834a63994.png
[70 22]: /images/20220515/36d89340fe9f4055a281be531e35ae9c.png
[70 23]: /images/20220515/924db211436d42e6bf0441e390c893a7.png
[70 24]: /images/20220515/4d231110382f42a1aab5f8a0d60545bc.png
[70 25]: /images/20220515/ecb544a927f4427bbe210477db92354c.png
[70 26]: /images/20220515/ceefb15373f04683a53673520bc7ff20.png
[70 27]: /images/20220515/16a12448d4624750bd5a09fabe7095dd.png
[70 28]: /images/20220515/7f96e5c970164060adf4d2e0006debb9.png
[70 29]: /images/20220515/81925068467646f7999fbb39423afba6.png
[70 30]: /images/20220515/de09e96c78e34e419fe0973263237859.png
[70 31]: /images/20220515/3b73373b62dc49e8a28cdbaa170c4ac4.png
[70 32]: /images/20220515/703dc6f4f2994776986bf24b1c7c7c21.png
[70 33]: /images/20220515/3a7cffc75b3b4c169adc67c8416c022e.png
[70 34]: /images/20220515/e183df70daf64bf3a66b767c321c197f.png
[70 35]: /images/20220515/bab2a8eb39284a67abed934999640726.png
[70 36]: /images/20220515/c9388c751e654565911caf1e2cd469c8.png
[70 37]: /images/20220515/0ef3c33182cb4066af232809870c63da.png
[70 38]: /images/20220515/76fc4d2fdbfe44bc93a6e04c5190c589.png
[70 39]: /images/20220515/5d30c57819e3487f9d551bc26eb368a3.png
[70 40]: /images/20220515/8331ca29fc234d5daa467f6103f3cdaf.png
[70 41]: /images/20220515/819252ea1f32451fbb539dc411535af7.png
[70 42]: /images/20220515/51fc388995714ba1a9a96a93c1cfa310.png
[70 43]: /images/20220515/dceebe151a604a50ac96795ff9fa9ab9.png
[70 44]: /images/20220515/c5c9cf62e8de4b8391c4431d80420898.png
[70 45]: /images/20220515/b095600501154adbb7633561cc0934fa.png
[70 46]: /images/20220515/3fb6c1a42015465b9d4ac4d631de55fa.png
[70 47]: /images/20220515/8d13c50085e0468c986b56fa84af6c5c.png
[70 48]: /images/20220515/c73f7406aa644c3d8351f4cc08e1ec3a.png
[70 49]: /images/20220515/439a83781093453aaaca970f7d356919.png
[70 50]: /images/20220515/37113c7d8b76479baab9c0e2066a8409.png
[70 51]: /images/20220515/98cd7438ee09402fa1edbde53b043dcf.png
[70 52]: /images/20220515/dcb0a399e57448f5ada97a3d878da997.png
[70 53]: /images/20220515/0695146cbce24d95befe459a40f7d71c.png
[70 54]: /images/20220515/c44f9d3d14994f78ad940380ab0323b1.png
[70 55]: /images/20220515/4f3d52644abf49518acc713bd69f3dd9.png
[70 56]: /images/20220515/d5faa32d64504c2ea7fb591aecc0cde1.png
[70 57]: /images/20220515/6c94974e75c34875b7caaee4f13fb572.png
[70 58]: /images/20220515/b9a9a986644f427e9f79ac42ea58e789.png
[70 59]: /images/20220515/fa196f9c72704fb9a990695c7252acaa.png
[70 60]: /images/20220515/34a91db56d8545b0ab69f970a5d363c5.png
[70 61]: /images/20220515/319a66f5485a4eec98f5c76d943d5568.png
[70 62]: /images/20220515/2c3bfde0315648fe8f2c86187b9bce77.png
[70 63]: /images/20220515/72f3f6c9f58848829a205715eea243ae.png
[70 64]: /images/20220515/37631db72394455b9a0fa06fe4bdc352.png
[70 65]: /images/20220515/0c7659b538454d439abc774927ab934a.png
[70 66]: /images/20220515/f26fd0176b8943e580e58342a0bc232c.png
[70 67]: /images/20220515/efadebf301594f309d0e89f50a6a018c.png
[70 68]: /images/20220515/58bd8f95b7cd4dedb3bd6fe28b0b4f61.png
[70 69]: /images/20220515/1ec0a9e01f05409db9f3744601889336.png
[70 70]: /images/20220515/70531126717846379f6c77814a2a5675.png
[70 71]: /images/20220515/8e2bd7a0e8a04351b782a4626021e1f7.png
[70 72]: /images/20220515/3fdaa3ecf8844c809b5d3fda32376db3.png
[70 73]: /images/20220515/889d4ba1f83443679406985f4eb49da7.png
[70 74]: /images/20220515/5a3041836fdf416399d2ce6d7d23bec0.png
[70 75]: /images/20220515/b082aa9b58834ee8aeb8ab91f5bada77.png
[70 76]: /images/20220515/9f33b4603acf44f899d8c386833b8bc5.png
[70 77]: /images/20220515/aeeeb8bbafdb4c03bc76c9f081c32a29.png
[70 78]: /images/20220515/42e797aa1bdc45b68b3dfcc1d93db1b4.png
[70 79]: /images/20220515/d318a918d48b4489a5673108a405a234.png
[70 80]: /images/20220515/ba4f2658e043475c8343d3dd13a24059.png
[70 81]: /images/20220515/90507038b333465db5015d6052a13a2f.png
[70 82]: /images/20220515/e84291d31e7840ffae246938574d14a6.png
[70 83]: /images/20220515/13831921b1c847ac96edeb3355b497fe.png
[70 84]: /images/20220515/6c772b994e954d30922720b8b7e10769.png
[70 85]: /images/20220515/86098a1b54a84ba9b5ef77afbb4f4fc3.png
[70 86]: /images/20220515/2f733de4a3f34dd883dcd195f3828b23.png
[70 87]: /images/20220515/8d87e031353c419582732ecdb4fb9efb.png
[70 88]: /images/20220515/1e96beb7223f44b18616b26a8af81b9f.png
[70 89]: /images/20220515/f703158603ed4da28bd69b54886508e6.png
[70 90]: /images/20220515/afd036fe552e4003ac423dcbea279104.png
[70 91]: /images/20220515/f0de704b2e1a428992ba66e6600d8d95.png
[70 92]: /images/20220515/68090e85b8b1448baf8dec9c2b4a5a48.png