Logstash多表同步MySQL数据到ES

柔情只为你懂 2021-10-19 01:34 550阅读 0赞

### 1、Logstash下载 ###

前往官网下载logstash，下载地址[https://www.elastic.co/downloads/logstash][https_www.elastic.co_downloads_logstash]

解压

### 2、下载mysql-connector-java jar包 ###

在logstash目录（\\bin同级）新建mysql目录，将下载好的mysql-connector-java-5.1.34.jar放入此目录；

在mysql目录新建jdbc.conf文件，此文件将配置数据库连接信息、查询数据sql、分页信息、同步频率等核心信息。

### 3、配置文件 ###

多表同步时，多个jdbc\{\}，output中多个判断。下例为全量更新

注意：在elasticsearch6.0中，一个索引下只能有一个类型。

input {
        stdin {}
        jdbc {
            jdbc_driver_library => "mysql/mysql-connector-java-6.0.6.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_connection_string => "jdbc:mysql://sandboxo2o.mysql.rds.aliyuncs.com:3306/food"
            jdbc_user => "petkitfood"
            jdbc_password => "petkit123"
            jdbc_paging_enabled => "true"
            jdbc_page_size => "100000"
            clean_run => true
            use_column_value => true
            tracking_column => "id"
            last_run_metadata_path => "/Users/menglinjie/ES-node/testdata.text"
            statement => "select * from user"
            schedule => "0 1 * * * *"
            type => "food_user"
       }
       jdbc {
            jdbc_driver_library => "mysql/mysql-connector-java-6.0.6.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_connection_string => "jdbc:mysql://sandboxo2o.mysql.rds.aliyuncs.com:3306/food"
            jdbc_user => "petkitfood"
            jdbc_password => "petkit123"
            jdbc_paging_enabled => "true"
            jdbc_page_size => "100000"
            clean_run => true
            use_column_value => true
            tracking_column => "id"
            last_run_metadata_path => "/Users/menglinjie/ES-node/testdata.text"
            statement => "select * from store_vip_pet"
            schedule => "0 1 * * * *"
            type => "food_pet"
       }
    }
    output {
       stdout {
            codec => json_lines
       }
       if[type] == "food_user"{
            elasticsearch {
            hosts => ["192.168.7.95:9200"]
            index => "food_user_index"
            document_id => "%{id}"
            document_type => "user"
       }
        }
       if[type] == "food_pet"{
            elasticsearch {
            hosts => ["192.168.7.95:9200"]
            index => "food_pet_index"
            document_id => "%{id}"
            document_type => "pet"
       }
        }
    }

### 4、启动Logstash ###

进入 Logstash的bin目录下：

./logstash -f mysql/jdbc.conf

### 5、查看kibana中索引和文档 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21lbmdsaW5qaWU_size_16_color_FFFFFF_t_70][]

[Kibana安装与汉化][Kibana]

[https_www.elastic.co_downloads_logstash]: https://www.elastic.co/downloads/logstash
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21lbmdsaW5qaWU_size_16_color_FFFFFF_t_70]: /images/20211018/0b1b509ea66d4afc9ef3f2432289926b.png
[Kibana]: https://blog.csdn.net/Gekkoou/article/details/80956191