mysql8与elasticsearch7.3.2数据同步，使用docker容器

墨蓝 2023-07-03 10:51 1阅读 0赞

一、需求软件及相关说明

需求软件  
来源容器镜像  
是否改造  
改造后镜像  
支持特别说明  
备注  
Mysql8.0  
mysql:8.0  
否  
   
   
   
elasticsearch-head  
mobz/elasticsearch-head:5-alpine  
是  
registry.cn-shanghai.aliyuncs.com/qlongit/elasticsearch-head:v1.0  
   
   
拉取最近代码修改而成，原有的代码无法显示数据  
Elasticsearch7.3.2  
Elasticsearch:7.3.2  
否  
   
   
安全性考虑升级到较高版本7.3.2  
但是此版本不支持一个索引下多个类型，支持一个索引下单个类型  
go-mysql-elasticsearch  
v2.1  
mysql:8.0  
是  
registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v2.1  
支持mysql8.0版本  
基于mysql8.0做mysqldump的客户端，使用mariadb最新的客户端版本是不兼容mysql8.0服务端的  
go-mysql-elasticsearch  
v1  
https://github.com/siddontang/go-mysql-elasticsearch  
是  
registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v1  
支持mysql5.7以下版本  
因为Dockerfile文件构建镜像时go的模块文件下载一直报错，所以改用国内的下载源  
kibana  
   
   
   
   
待有时间再上线这个  
Java服务端调用  
   
   
   
   
使用jest方式调用

二、自己制作的容器镜像上传地址

目前都在阿里云上

三、docker-compose的编排脚本

3.1、编排脚本

version: '2.2'

services:

es01:

image: elasticsearch:7.3.2

container\_name: es01

environment:

- node.name=es01

- discovery.seed\_hosts=es02

- cluster.initial\_master\_nodes=es01,es02

- cluster.name=docker-cluster

- bootstrap.memory\_lock=true

- "ES\_JAVA\_OPTS=-Xms512m -Xmx512m"

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- esdata01:/usr/share/elasticsearch/data

- ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

ports:

- 9200:9200

networks:

- esnet

es02:

image: elasticsearch:7.3.2

container\_name: es02

environment:

- node.name=es02

- discovery.seed\_hosts=es01

- cluster.initial\_master\_nodes=es01,es02

- cluster.name=docker-cluster

- bootstrap.memory\_lock=true

- "ES\_JAVA\_OPTS=-Xms512m -Xmx512m"

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- esdata02:/usr/share/elasticsearch/data

- ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

networks:

- esnet

eshead:

image: elasticsearch-head:v1.0

container\_name: elasticsearch-head

ports:

- 9100:9100

links:

- es01:es

command:

- sh

- -c

- npm run start

networks:

- esnet

mysql8:

image: mysql:8.0

container\_name: mysql8

ports:

- 13306:3306

environment:

- MYSQL\_ROOT\_PASSWORD=\*\*\*\*\*

- MYSQL\_DATABASE=\*\*\*\*

- MYSQL\_USER=\*\*\*\*\*

- MYSQL\_PASSWORD=\*\*\*\*\*\*

volumes:

- ~/elastic/mysql8/data:/var/lib/mysql

networks:

- esnet

gomysqles:

image: go-mysql-es:v2.1

container\_name: go-mysql-es

volumes:

- ~/elastic/go-mysql-elasticsearch/var:/goworkerspace/bin/var

- ~/elastic/go-mysql-elasticsearch/etc/river.toml:/goworkerspace/bin/etc/river.toml

networks:

- esnet

links:

- es01:es

- mysql8:mysql

depends\_on:

- es01

- es02

- mysql8

volumes:

esdata01:

driver: local

esdata02:

driver: local

networks:

esnet:

其中一些敏感的数据库账号与密码需要自行调整，还要镜像调整为上面阿里云镜像

关于一些外挂的一些配置文件看下面几章节

3.2、elasticsearch配置文件如下：

~/elastic/elastic/elasticsearch.yml

配置文件位置视系统环境调整

cluster.name: "docker-cluster"

network.host: 0.0.0.0

http.cors.enabled: true

http.cors.allow-origin: "\*"

3.3、go-mysql-elasticsearch的配置文件

~/elastic/go-mysql-elasticsearch/etc/river.toml

\# MySQL address, user and password

\# user must have replication privilege in MySQL.

my\_addr = "mysql:3306"

my\_user = "\*\*\*"

my\_pass = "\*\*\*\*\*"

my\_charset = "utf8"

\# Set true when elasticsearch use https

\#es\_https = false

\# Elasticsearch address

es\_addr = "es:9200"

\# Elasticsearch user and password, maybe set by shield, nginx, or x-pack

es\_user = ""

es\_pass = ""

\# Path to store data, like master.info, if not set or empty,

\# we must use this to support breakpoint resume syncing.

\# TODO: support other storage, like etcd.

data\_dir = "./var"

\# Inner Http status address

stat\_addr = "127.0.0.1:12800"

\# pseudo server id like a slave

server\_id = 1001

\# mysql or mariadb

flavor = "mysql"

\#flavor = "mariadb"

\# mysqldump execution path

\# if not set or empty, ignore mysqldump.

mysqldump = "mysqldump"

\# if we have no privilege to use mysqldump with --master-data,

\# we must skip it.

\#skip\_master\_data = false

\# minimal items to be inserted in one bulk

bulk\_size = 128

\# force flush the pending requests if we don't have enough items >= bulk\_size

flush\_bulk\_time = "200ms"

\# Ignore table without primary key

skip\_no\_pk\_table = false

\# MySQL data source

\[\[source\]\]

schema = "sakila"

\# Only below tables will be synced into Elasticsearch.

\# "t\_\[0-9\]\{4\}" is a wildcard table format, you can use it if you have many sub tables, like table\_0000 - table\_1023

\# I don't think it is necessary to sync all tables in a database.

tables = \["sougoulog","film"\]

\# Below is for special rule mapping

\# Very simple example

\# desc t;

\# +-------+--------------+------+-----+---------+-------+

\# +-------+--------------+------+-----+---------+-------+

\# | id    | int(11)      | NO   | PRI | NULL    |       |

\# +-------+--------------+------+-----+---------+-------+

\# The table \`t\` will be synced to ES index \`test\` and type \`t\`.

\[\[rule\]\]

schema = "sakila"

table = "sougoulog"

index = "sakila\_sougoulog"

type = "sougoulog"

\[\[rule\]\]

schema = "sakila"

table = "film"

index = "sakila\_film"

type = "film"

如果是es6以下版本支持一个index下多个type，

在es7.3.2版本是不支持一个index多个type的，只能是一个index一个type。

这个视个人情况来配置

四、关于go-mysql-elasticsearch不支持DDL操作同步解决方案

4.1、删除做了ddl操作的index

4.2、新开一个docker镜像

新开的只有原来的表，重新mysqldump操作，需要调整配置文件与删除日志存储点文件实现

4.3、销毁新开的，原来的就会重新按binlog来同步

五、关于go-mysql-elasticsearch运行过久不同步的问题

这个大家都遇到过，建立一个定时任务重启docker即可。

真正解决，目前要看作者有无时间研究处理。本人对于go也才刚刚接触

mysql8与elasticsearch7.3.2数据同步，使用docker容器

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关在docker容器中安装Elasticsearch分词器并同步数据

相关 elasticsearch与mysql数据同步

相关使用logstash同步mysql数据到elasticsearch

相关 mysql8与elasticsearch7.3.2数据同步，使用docker容器

相关 docker部署logstash实现mysql数据同步至elasticsearch

相关 mysql8与elasticsearch7.3.2数据同步,使用docker容器

相关 ElasticSearch--与MySQL同步

相关 elasticsearch使用river同步mysql数据

相关 elasticsearch与MySQL数据同步

相关 docker容器与系统时间同步