ELK日志分析平台.1-搭建

╰半夏微凉° 2022-04-24 04:56 259阅读 0赞

# [ELK日志分析平台.1-搭建][ELK_.1-] #

# ELK日志分析平台.1-搭建 #

2017-12-28 |  [admin][]

**一、简介**  
**1、核心组成**  
    ELK由 **Elasticsearch、Logstash**和 **Kibana**三部分组件组成；  
     **Elasticsearch**是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。  
     **Logstash**是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用  
     **kibana** 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。  
**2、四大组件**  
    Logstash: logstash server 端用来搜集日志；  
    Elasticsearch: 存储各类日志；  
    Kibana: web化接口用作查寻和可视化日志；  
    Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；

**3、ELK工作流程**  
    在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。

简单的流程示意图如下：

![elk01][]

**二、开始部署**

系统环境：CentOS 7.1

服务器IP：172.16.16.201

建立安装目录：/elk

新建测试用户：useradd elk

给予权限：chown -R elk:elk /elk

**关于ELK三者的版本问题：**最好统一在同一修订版本，否则会在最后出现版本错误而不能抓取到日志数据

如本次安装版本全部为6.1.1版本

**1、安装JDK环境**

Elasticsearch、Logstash、Kibana均需要JDK1.8及以上环境，所以需要先安装JDK

安装JDK过程略，参考笔记：JDK1.8安装笔记：http://www.chengzi520.com/?p=1512

**2、安装Elasticsearch**

**安装版本：**elasticsearch-6.1.1

**安装目录：**/elk/elasticsearch-6.1.1，安装过程略，参考笔记：elasticsearch-6.0.1安装：http://www.chengzi520.com/?p=1519

**3、安装Logstash**

**  安装版本：**logstash-6.1.1

** 安装目录：**/elk/logstash-6.1.1

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span># 下载</span> 
     </div> 
     <div> 
      <span>cd<span> <span>/<span>elk</span></span></span></span> 
     </div> 
     <div> 
      <span>wget <span>https<span>:<span>/<span>/<span>artifacts<span>.<span>elastic<span>.<span>co<span>/<span>downloads<span>/<span>logstash<span>/<span>logstash<span>-<span>6.1.1.tar.gz</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span># 解压、进入目录</span> 
     </div> 
     <div> 
      <span>tar <span>zxvf <span>logstash<span>-<span>6.1.1.tar.gz</span></span></span></span></span> 
     </div> 
     <div> 
      <span>cd<span> <span>logstash<span>-<span>6.1.1</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**添加系统变量**

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>vim<span> <span>/<span>etc<span>/<span>profile</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>LOGSTASH_HOME<span>=<span>/<span>elk<span>/<span>logstash<span>-<span>6.1.1</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>export <span>PATH<span>=<span>$<span>PATH<span>:<span>$<span>LOGSTASH_HOME<span>/<span>bin</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>source<span> <span>/<span>etc<span>/<span>profile</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**logstash常用参数**  
**    -e :**指定logstash的配置信息，可以用于快速测试;  
**    -f :**指定logstash的配置文件；可以用于生产环境;

**启动Logstash测试**

**测试1：**通过-e参数指定logstash的配置信息，用于快速测试，直接输出到屏幕。

**logstash -e “input \{stdin\{\}\} output \{stdout\{\}\}”**

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
     <div>
       12 
     </div> 
     <div>
       13 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>[<span>elk<span>@<span>localhost<span> <span>~<span>]<span>$<span> <span>logstash<span> <span>-<span>e<span> <span>"input {stdin{}} output {stdout{}}"</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>Sending <span>Logstash<span>'s logs to /elk/logstash-6.1.1/logs which is now configured via log4j2.properties</span></span></span> 
     </div> 
     <div> 
      <span>[2017-12-28T16:05:36,161][INFO ][logstash.modules.scaffold] Initializing module {:module_name=&gt;"netflow", :directory=&gt;"/elk/logstash-6.1.1/modules/netflow/configuration"}</span> 
     </div> 
     <div> 
      <span>[2017-12-28T16:05:36,212][INFO ][logstash.modules.scaffold] Initializing module {:module_name=&gt;"fb_apache", :directory=&gt;"/elk/logstash-6.1.1/modules/fb_apache/configuration"}</span> 
     </div> 
     <div> 
      <span>[2017-12-28T16:05:37,574][WARN ][logstash.config.source.multilocal] Ignoring the '<span>pipelines<span>.<span>yml'<span> <span>file<span> <span>because <span>modules <span>or<span> <span>command <span>line <span>options <span>are <span>specified</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>[<span>2017<span>-<span>12<span>-<span>28T16<span>:<span>05<span>:<span>39<span>,<span>184<span>]<span>[<span>INFO<span> <span>]<span>[<span>logstash<span>.<span>runner<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>]<span> <span>Starting<span> <span>Logstash<span> <span>{ <span>"logstash.version"<span>=<span>&gt;<span>"6.1.1"<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>[<span>2017<span>-<span>12<span>-<span>28T16<span>:<span>05<span>:<span>40<span>,<span>558<span>]<span>[<span>INFO<span> <span>]<span>[<span>logstash<span>.<span>agent<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>]<span> <span>Successfully<span> <span>started<span> <span>Logstash<span> <span>API<span> <span>endpoint<span> <span>{ <span>:<span>port<span>=<span>&gt;<span>9600<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>[<span>2017<span>-<span>12<span>-<span>28T16<span>:<span>05<span>:<span>44<span>,<span>576<span>]<span>[<span>INFO<span> <span>]<span>[<span>logstash<span>.<span>pipeline<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>]<span> <span>Starting<span> <span>pipeline<span> <span>{ <span>:<span>pipeline_id<span>=<span>&gt;<span>"main"<span>,<span> <span>"pipeline.workers"<span>=<span>&gt;<span>1<span>,<span> <span>"pipeline.batch.size"<span>=<span>&gt;<span>125<span>,<span> <span>"pipeline.batch.delay"<span>=<span>&gt;<span>5<span>,<span> <span>"pipeline.max_inflight"<span>=<span>&gt;<span>125<span>,<span> <span>:<span>thread<span>=<span>&gt;<span>"#&lt;Thread:0x449f3ddb run&gt;"<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>[<span>2017<span>-<span>12<span>-<span>28T16<span>:<span>05<span>:<span>44<span>,<span>792<span>]<span>[<span>INFO<span> <span>]<span>[<span>logstash<span>.<span>pipeline<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>]<span> <span>Pipeline<span> <span>started<span> <span>{ <span>"pipeline.id"<span>=<span>&gt;<span>"main"<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>The <span>stdin <span>plugin <span>is<span> <span>now <span>waiting <span>for<span> <span>input<span>:<span>&nbsp;&nbsp;&nbsp;&nbsp;<span>#等待手动输入</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>[<span>2017<span>-<span>12<span>-<span>28T16<span>:<span>05<span>:<span>44<span>,<span>968<span>]<span>[<span>INFO<span> <span>]<span>[<span>logstash<span>.<span>agent<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>]<span> <span>Pipelines<span> <span>running<span> <span>{ <span>:<span>count<span>=<span>&gt;<span>1<span>,<span> <span>:<span>pipelines<span>=<span>&gt;<span>[<span>"main"<span>]<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>HelloWorld<span>&nbsp;&nbsp;&nbsp;&nbsp;<span># 手动输入</span></span></span> 
     </div> 
     <div> 
      <span>2017<span>-<span>12<span>-<span>28T08<span>:<span>05<span>:<span>58.028Z<span> <span>localhost<span>.<span>localdomain <span>HelloWorld<span>&nbsp;&nbsp;<span>#这种输入直接原封不动返回</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**    测试2：**通过-e参数指定logstash的配置信息，用于快速测试，以json格式输出到屏幕。

**  logstash -e ‘input\{stdin\{\}\}output\{stdout\{codec=>rubydebug\}\}’**

![elk02][]

**4、安装Kibana**

** 安装版本：**kibana-6.1.1

**安装目录：**/elk/kibana-6.1.1

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span># 下载、解压</span> 
     </div> 
     <div> 
      <span>wget <span>https<span>:<span>/<span>/<span>artifacts<span>.<span>elastic<span>.<span>co<span>/<span>downloads<span>/<span>kibana<span>/<span>kibana<span>-<span>6.1.1<span>-<span>linux<span>-<span>x86_64<span>.<span>tar<span>.<span>gz</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>tar <span>zxvf <span>kibana<span>-<span>6.1.1<span>-<span>linux<span>-<span>x86_64<span>.<span>tar<span>.<span>gz</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>mv <span>kibana<span>-<span>6.1.1<span>-<span>linux<span>-<span>x86<span>_64&nbsp;<span> <span>.<span>/<span>kibana<span>-<span>6.1.1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>cd<span> <span>kibana<span>-<span>6.1.1<span>/</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**修改配置文件 **

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>vim <span>config<span>/<span>kibana<span>.<span>yml</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>server<span>.<span>port<span>:<span> <span>5601<span>&nbsp;&nbsp;<span># 端口号</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>server<span>.<span>host<span>:<span> <span>"172.16.16.201"<span>&nbsp;&nbsp;<span># 本机IP</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>elasticsearch<span>.<span>url<span>:<span> <span>"http://172.16.16.201:9200"<span>&nbsp;&nbsp;<span># Elasticsearch url地址</span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**启动**

![elk03][]

**通过浏览器访问**

**    http://172.16.16.201:5601**

![elk04][]

posted @ 2018-03-22 18:10 [疯狂的米粒儿][Link 1] 阅读( ...) 评论( ...) [编辑][Link 2] 收藏

[ELK_.1-]: https://www.cnblogs.com/Su-per-man/p/8625470.html
[admin]: https://www.chengzi520.com/?author=1
[elk01]: http://www.chengzi520.com/wp-content/uploads/2017/12/elk01.jpg
[elk02]: http://www.chengzi520.com/wp-content/uploads/2017/12/elk02.png
[elk03]: http://www.chengzi520.com/wp-content/uploads/2017/12/elk03.png
[elk04]: http://www.chengzi520.com/wp-content/uploads/2017/12/elk04.png
[Link 1]: https://www.cnblogs.com/Su-per-man/
[Link 2]: https://i.cnblogs.com/EditPosts.aspx?postid=8625470