6.6 工具-ELK安装

浅浅的花香味﹌ 2024-03-30 17:00 73阅读 0赞

**目录**

6.6.1 Elasticsearch安装

6.6.1.1 安装

6.6.1.1.1 window

6.6.1.1.2 Linux

6.6.1.2 问题

6.6.1.2.1 问题一

6.6.1.2.2 问题二

6.6.2 Logstash安装

6.6.2.1 安装

6.6.2.1.1 window

6.6.2.1.2 Linux

6.6.2.2 问题

6.6.2.2.1 问题一

6.6.3 Kibana

6.6.3.1 安装

--------------------

## 6.6.1 Elasticsearch安装 ##

### 6.6.1.1 安装 ###

#### 6.6.1.1.1 window ####

下载地址：[下载 Elastic 产品 | Elastic][Elastic _ _ Elastic]，在下载页面我们需要下载 Elasticsearch和Kibana

下载后解压进入到bin目录下，双击elasticsearch.bat，即可启动。

![5d642010ad8d4382a2cedc90373a88cd.png][]

在浏览器输入**http://localhost:9200/** (如果是Linux环境安装，则访问路径**http://Linux的IP地址:9200**)

![24d5ec6ae12f483e92f6ed1c85c560f4.png][]

#### 6.6.1.1.2 Linux ####

在opt目录下解压elasticsearch-7.4.2

进入elasticsearch-7.4.2目录下的bin目录

./elasticsearch

注意这里可能会出现用户问题,解决办法看《6.6.1.2.2 问题二》

### 6.6.1.2 问题 ###

#### 6.6.1.2.1 问题一 ####

max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]

elasticsearch用户拥有的可创建文件描述的权限太低，至少需要65536

切换到root用户修改

su root

vim /etc/security/limits.conf

在最后面追加下面内容

SHIFT+G 可以直接定位到最后

# * ：表示用户
    * hard nofile 65536
    
    * soft nofile 65536

重新su以下，才能使配置生效

su root
    或者
    # su其他用户是因为es启动不能以root，所以需要添加新的用户来启动es
    su 其他用户

查看硬限制

[sandwich@centos-elk ~]$ ulimit -Hn
    65536
    [sandwich@centos-elk ~]$ ulimit -Sn
    65536

#### 6.6.1.2.2 问题二 ####

3405 -- main class information unavailable
    [root@master bin]# [2019-06-20T00:14:04,792][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es1] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
    	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
    	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
    	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
    Caused by: java.lang.RuntimeException: can not run elasticsearch as root

不能用root用户启动elasticsearch

创建elsearch用户组及elsearch用户：

groupadd elsearch
    useradd elsearch -g elsearch
    passwd elsearch

更改elasticsearch文件夹及内部文件的所属用户及组为elsearch:elsearch

cd /opt
    chown -R elsearch:elsearch  elasticsearch-6.8.0

切换到elsearch用户再启动

su elsearch 
    cd /opt/elasticsearch-6.8.0/bin
    ./elasticsearch

## 6.6.2 Logstash安装 ##

### 6.6.2.1 安装 ###

### 6.6.2.1.1 window ###

Logstash是一个开源的服务器端数据处理管道，可以同时从多个数据源获取数据，并对其进行转换，然后将其发送到你最喜欢的“存储”。

> Logstash的下载地址：[Download Logstash Free | Get Started Now | Elastic][Download Logstash Free _ Get Started Now _ Elastic]
> 
> 下载测试数据集，下载地址：[Index of /datasets/movielens][Index of _datasets_movielens]，在该网页中下载[ml-latest.zip][]

**movies.csv**

![8e5b649b7b124ac394686269eab3350c.png][]

![5249a34d0d054d65a60cfdae6f0c84b4.png][]

下载后解压目录，将movies.csv文件拷贝到指定的目录(随便创建一个文件夹如：D:/logstash-datas/)下。

进入到Logstash的解压目录，进入到config目录下，新建logstash.conf，文件内容如下：

input {
      file {
    # 注意路径要修改为自己的路径
        path => "D:/logstash-datas/movies.csv"
        start_position => "beginning"
        sincedb_path => "D:/Develop/ELK/logstash-7.4.2/db_path.log"
      }
    }
    filter {
    # 将数据分为三列id，content，genre
      csv {
        separator => ","
        columns => ["id","content","genre"]
      }
    
      mutate {
    # 之后将genre用|去分割，并将"path","host"等删除
        split => { "genre" => "|" }
        remove_field => ["path", "host","@timestamp","message"]
      }
    
      mutate {
    # content 用(分割 前面为title,后面为year
        split => ["content", "("]
        add_field => { "title" => "%{[content][0]}"}
        add_field => { "year" => "%{[content][1]}"}
      }
    
      mutate {
    # year类型转为integer
        convert => {
          "year" => "integer"
        }
        strip => ["title"]
        remove_field => ["path", "host","@timestamp","message","content"]
      }
    
    }
    # 将数据存储到指定es中
    output {
       elasticsearch {
         hosts => "http://localhost:9200"
    # 索引叫movies
         index => "movies"
    # 文档id
         document_id => "%{id}"
       }
      stdout {}
    }

**注意修改路径**

> input \{  
> file \{  
> path => "D:/logstash-datas/movies.csv" **数据集路径**  
> start\_position => "beginning"  
> sincedb\_path => "D:/Develop/ELK/logstash-7.4.2/db\_path.log" **logstash的路径**  
> \}  
> \}

Logstash会转换数据，上面配置的logstash.conf就是转换规则

在命令行进入到logstash的bin目录下，输入如下命令启动Logstash

logstash.bat -f D:\Develop\ELK\logstash-7.4.2\config\logstash.conf

### 6.6.2.1.2 Linux ###

将logstash7.4.2解压到opt目录下

进入到Logstash的解压目录，进入到config目录下，新建logstash.conf

**logstash.conf**

input {
      file {
        path => "/opt/logstash-datas/movies.csv" # 这里指定数据集路径
        start_position => "beginning"
    # 这里指定db_path.log路径，起初elasticsearch-7.4.2/目录下是没有db_path.log的，跑完数据集会自动生成db_path.log
        sincedb_path => "/opt/elasticsearch-7.4.2/db_path.log"
      }
    }
    filter {
      csv {
        separator => ","
        columns => ["id","content","genre"]
      }
    
      mutate {
        split => { "genre" => "|" }
        remove_field => ["path", "host","@timestamp","message"]
      }
    
      mutate {
    
        split => ["content", "("]
        add_field => { "title" => "%{[content][0]}"}
        add_field => { "year" => "%{[content][1]}"}
      }
    
      mutate {
        convert => {
          "year" => "integer"
        }
        strip => ["title"]
        remove_field => ["path", "host","@timestamp","message","content"]
      }
    
    }
    output {
       elasticsearch {
    # es的地址
         hosts => "http://localhost:9200"
         index => "movies"
         document_id => "%{id}"
       }
      stdout {}
    }

配置完成后进入bin目录,启动logstash，并指定配置文件

./logstash -f config/logstash.conf

![e11b7ed93ad34b179e4b81a01d348b99.png][] 开始转换数据![78a1a91f64534914a9c4296c8f57a199.png][]

kibana中输入，查看所有索引

GET _cat/indices

![234102d64dda437c8b3f4ef46bc0d26e.png][]

### 6.6.2.2 问题 ###

#### 6.6.2.2.1 问题一 ####

Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

删除 logstash-7.4.2/data中的.lock 文件

删除命令
    rm  -rf  .lock

## 6.6.3 Kibana ##

### 6.6.3.1 安装 ###

Kibana *\[kɪbana\]* 是世界上最受欢迎的开源日志分析平台它为用户提供了一个工具，用于在存储于Elasticsearch集群中的日志数据进行检索，可视化和构建仪表板。

Kibana的核心功能是数据查询和分析

图形化界面，方便操作

> 如果在Linux环境中
> 
> 下载后进入到家目录下的bin目录中，首先修改config目录下的kibana.yml文件，修改如下内容：
> 
> ![ba3a3d7a29484328ba2cfb9dbdb6d22e.png][]
> 
>     elasticsearch.hosts: ["http://192.168.27.129:9200"]

![2f2e50a9f63d4ab5afc2da76ed3c6898.png][]

进入bin目录下，双加 kibana.bat文件，在浏览器地址栏访问：**http://localhost:5601**![b13c8f93deee407da1a381d2a29aaaed.png][]

打开开发工具

![bfa4ac64fedd4f55ac9a72cbbaf1d594.png][]

类似于在Navicat操作mysql数据库

[Elastic _ _ Elastic]: https://www.elastic.co/cn/start
[5d642010ad8d4382a2cedc90373a88cd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/5e8d240c6a124a008a732d9b844cb7e1.png
[24d5ec6ae12f483e92f6ed1c85c560f4.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/9a121a41d0504cdfa9bca9f14ca53274.png
[Download Logstash Free _ Get Started Now _ Elastic]: https://www.elastic.co/cn/downloads/logstash
[Index of _datasets_movielens]: http://files.grouplens.org/datasets/movielens/
[ml-latest.zip]: http://files.grouplens.org/datasets/movielens/ml-latest.zip
[8e5b649b7b124ac394686269eab3350c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/2aa2481cab7f4b1a851cc9f4a41ca018.png
[5249a34d0d054d65a60cfdae6f0c84b4.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/83ac88acbd8a45abb9ca5ff2731443cc.png
[e11b7ed93ad34b179e4b81a01d348b99.png]: https://img-blog.csdnimg.cn/e11b7ed93ad34b179e4b81a01d348b99.png
[78a1a91f64534914a9c4296c8f57a199.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/a2f1b3ac9a844163aff4ac65f236570f.png
[234102d64dda437c8b3f4ef46bc0d26e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/0813b90c5b92487296baa36b07d54596.png
[ba3a3d7a29484328ba2cfb9dbdb6d22e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/d0020f7130cf407d9789d206344f82ec.png
[2f2e50a9f63d4ab5afc2da76ed3c6898.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/16ec28dd760b4b948ad3870766fe6613.png
[b13c8f93deee407da1a381d2a29aaaed.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/2d32bd467bd74127a738fbcc9a877ce8.png
[bfa4ac64fedd4f55ac9a72cbbaf1d594.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/e125a438472d4881b994de9ab9a6fa57.png