Logstash安装及数据导入

客官°小女子只卖身不卖艺 2023-07-04 06:11 24阅读 0赞

### 一、安装前准备 ###

**1）下载与ES相同版本号的logstash，（7.1.0），并解压到相应目录**  
官网下载地址：https://www.elastic.co/cn/downloads/logstash  
华为过年镜像地址：https://mirrors.huaweicloud.com/logstash/7.1.0/

**2）下载最MovieLens最小测试数据集**  
地址：https://grouplens.org/datasets/movielens/  
对应文件：Small: 100,000 ratings and 3,600 tag applications applied to 9,000 movies by 600 users. Last updated 9/2018.

**3）运行环境需要提前安装jdk**  
**4）启动Kibana、Elasticsearch**  
启动安装说明前面章节已经描述  
启动Kibana: https://blog.csdn.net/qq\_36918149/article/details/104224625  
启动Elasticsearch :https://blog.csdn.net/qq\_36918149/article/details/104221934

### 二、配置 ###

**1）在Logstash目录下创建配置文件目录**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70]  
**2）将MovieLens下载的测试数据拷贝到movielens\_dome\_data目录**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 1]  
**3）创建配置文件logstash.conf**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 2]  
配置文件内容

input { 
      file { 
        # 这是在加载demo数据，请改为你本地数据路径,注意路径用的是‘/’而非‘\’,否则会导致无法读入数据
        path => "D:/step/logstash-7.1.0/movielens_dome_data/movies.csv"
        start_position => "beginning"
        sincedb_path => "nul"
      }
    }
    filter { 
      csv { 
        separator => ","
        columns => ["id","content","genre"]
      }
    
      mutate { 
        split => {  "genre" => "|" }
        remove_field => ["path", "host","@timestamp","message"]
      }
    
      mutate { 
    
        split => ["content", "("]
        add_field => {  "title" => "%{[content][0]}"}
        add_field => {  "year" => "%{[content][1]}"}
      }
    
      mutate { 
        convert => { 
          "year" => "integer"
        }
        strip => ["title"]
        remove_field => ["path", "host","@timestamp","message","content"]
      }
    
    }
    output { 
       elasticsearch { 
         # 连接elasticserach 并导入数据
         hosts => "http://localhost:9200"
         index => "movies"
         document_id => "%{id}"
       }
      stdout { }
    }

**3）启动 logstash 并开始导入数据**  
在Logstash安装包下执行命令

![在这里插入图片描述][20200208200802275.png]

bin\logstash -f D:\step\logstash-7.1.0\movielens_dome_data\logstash.conf

如图已经启动成功并开始导入数据  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 3]  
**4）kibana 验证数据是否已经导入**  
http://localhost:5601/app/kibana\#/management/elasticsearch/index\_management/indices?\_g=()  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 4]  
备注：如上图，我们看到Elasticsearch的索引已经新增

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200208192547268.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200208192644732.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200208192815501.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[20200208200802275.png]: https://img-blog.csdnimg.cn/20200208200802275.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2020020820465861.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/2020020820484291.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2OTE4MTQ5,size_16,color_FFFFFF,t_70