IK Analyzer 热更新自定义词典

女爷i 2023-06-13 09:29 10阅读 0赞

IK Analyzer的GitHub地址:

https://github.com/medcl/elasticsearch-analysis-ik

IK Analyzer 自己有一个热更新方案, 但不推荐使用, 我们可以修改下源码, 从mysql读取中读取自定义的词典.

> 本文Elasticsearch安装的是5.2.0版本的, 所以对应的IK Analyzer也是安装的5.2.0版本.

**自定义热更新方案**

elasticsearch-analysis-ik源码的代码结构图如下所示, 我们只需要修改一个Dictionary类, 新增一个CustomLoadExtDict类和一个jdbc.properties配置文件即可.

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]

(1) 数据库表结构

首先在数据库新建两张表, 一张主扩展词典表, 一张停用词扩展词典表.

CREATE TABLE `t_hot_words` (
      `id` bigint(20) NOT NULL AUTO_INCREMENT,
      `word` varchar(255) NOT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
    
    
    CREATE TABLE `t_hot_stop_words` (
      `id` bigint(20) NOT NULL AUTO_INCREMENT,
      `word` varchar(255) NOT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;

(2) jdbc.properties

我们在配置文件中新增一个数据库配置文件.

jdbc.url=jdbc:mysql://127.0.0.1:3306/tyshawn_test?useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT
    jdbc.user=root
    jdbc.password=123
    jdbc.words.sql=select word from t_hot_words
    jdbc.stopwords.sql=select word from t_hot_stop_words
    jdbc.interval=1000

(3) 从MySQL加载词典的类

该类有两个方法, 一个加载主扩展词典表, 一个加载停用词扩展词典表

public class CustomLoadExtDict {
        private static Logger logger = ESLoggerFactory.getLogger(CustomLoadExtDict.class);
        private static Properties prop = new Properties();
    
        static {
            try {
                Class.forName("com.mysql.jdbc.Driver");
            } catch (ClassNotFoundException e) {
                logger.error("class.forName error", e);
            }
        }
    
    
        /**
         * 从mysql加载主扩展词典
         */
        public static void loadMySQLExtDict(String dictRoot, DictSegment mainDict) {
            Connection conn = null;
            Statement stmt = null;
            ResultSet rs = null;
    
            try {
                Path file = PathUtils.get(dictRoot, "jdbc.properties");
                prop.load(new FileInputStream(file.toFile()));
    
                logger.info("query hot dict from mysql: " + prop.getProperty("jdbc.words.sql") + "......");
    
                conn = DriverManager.getConnection(
                        prop.getProperty("jdbc.url"),
                        prop.getProperty("jdbc.user"),
                        prop.getProperty("jdbc.password"));
                stmt = conn.createStatement();
                rs = stmt.executeQuery(prop.getProperty("jdbc.words.sql"));
    
                while(rs.next()) {
                    String theWord = rs.getString("word");
    
                    logger.info("load hot word from mysql: " + theWord);
                    mainDict.fillSegment(theWord.trim().toCharArray());
                }
    
                Thread.sleep(Integer.valueOf(String.valueOf(prop.get("jdbc.interval"))));
            } catch (Exception e) {
                logger.error("erorr", e);
            } finally {
                if(rs != null) {
                    try {
                        rs.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
                if(stmt != null) {
                    try {
                        stmt.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
                if(conn != null) {
                    try {
                        conn.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
            }
        }
    
        /**
         * 从mysql加载停用词扩展词典
         */
        public static void loadMySQLStopwordDict(String dictRoot, DictSegment stopWords) {
            Connection conn = null;
            Statement stmt = null;
            ResultSet rs = null;
    
            try {
                Path file = PathUtils.get(dictRoot, "jdbc.properties");
                prop.load(new FileInputStream(file.toFile()));
    
                logger.info("query hot stopword dict from mysql, " + prop.getProperty("jdbc.stopwords.sql") + "......");
    
                conn = DriverManager.getConnection(
                        prop.getProperty("jdbc.url"),
                        prop.getProperty("jdbc.user"),
                        prop.getProperty("jdbc.password"));
                stmt = conn.createStatement();
                rs = stmt.executeQuery(prop.getProperty("jdbc.stopwords.sql"));
    
                while(rs.next()) {
                    String theWord = rs.getString("word");
    
                    logger.info("load hot stopword from mysql: " + theWord);
                    stopWords.fillSegment(theWord.trim().toCharArray());
                }
    
                Thread.sleep(Integer.valueOf(String.valueOf(prop.get("jdbc.interval"))));
            } catch (Exception e) {
                logger.error("erorr", e);
            } finally {
                if(rs != null) {
                    try {
                        rs.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
                if(stmt != null) {
                    try {
                        stmt.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
                if(conn != null) {
                    try {
                        conn.close();
                    } catch (SQLException e) {
                        logger.error("error", e);
                    }
                }
            }
        }
    }

(4) 修改CustomLoadExtDict类

a. 词典初始化方法

在词典初始化方法中创建了一个线程去不断的执行reLoadMainDict()方法.

public static synchronized Dictionary initial(Configuration cfg) {
    	if (singleton == null) {
    		synchronized (Dictionary.class) {
    			if (singleton == null) {
    
    				singleton = new Dictionary(cfg);
    				singleton.loadMainDict();
    				singleton.loadSurnameDict();
    				singleton.loadQuantifierDict();
    				singleton.loadSuffixDict();
    				singleton.loadPrepDict();
    				singleton.loadStopWordDict();
    
    				if(cfg.isEnableRemoteDict()){
    					// 建立监控线程
    					for (String location : singleton.getRemoteExtDictionarys()) {
    						// 10 秒是初始延迟可以修改的 60是间隔时间 单位秒
    						pool.scheduleAtFixedRate(new Monitor(location), 10, 60, TimeUnit.SECONDS);
    					}
    					for (String location : singleton.getRemoteExtStopWordDictionarys()) {
    						pool.scheduleAtFixedRate(new Monitor(location), 10, 60, TimeUnit.SECONDS);
    					}
    				}
    
    				//自定义热更新方案: 定时从mysql加载自定义词库
    				Executors.newFixedThreadPool(1).submit(new Runnable() {
    					@Override
    					public void run() {
    						while (true) {
    							Dictionary.getSingleton().reLoadMainDict();
    						}
    					}
    				});
    
    				return singleton;
    			}
    		}
    	}
    	return singleton;
    }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 1]

b. 重新加载词典方法

这个方法不用修改. 这里调用了 tmpDict.loadMainDict() 和 tmpDict.loadStopWordDict() 两个方法来加载主扩展词典和停用词扩展词典.

public void reLoadMainDict() {
    	logger.info("重新加载词典...");
    	// 新开一个实例加载词典，减少加载过程对当前词典使用的影响
    	Dictionary tmpDict = new Dictionary(configuration);
    	tmpDict.configuration = getSingleton().configuration;
    	tmpDict.loadMainDict();
    	tmpDict.loadStopWordDict();
    	_MainDict = tmpDict._MainDict;
    	_StopWords = tmpDict._StopWords;
    	logger.info("重新加载词典完毕...");
    }

c. 加载主扩展词典方法

在这个方法最后添加从mysql加载主扩展词典的方法.

/**
     * 加载主词典及扩展词典
     */
    private void loadMainDict() {
    	// 建立一个主词典实例
    	_MainDict = new DictSegment((char) 0);
    
    	// 读取主词典文件
    	Path file = PathUtils.get(getDictRoot(), Dictionary.PATH_DIC_MAIN);
    
    	InputStream is = null;
    	try {
    		is = new FileInputStream(file.toFile());
    	} catch (FileNotFoundException e) {
    		logger.error(e.getMessage(), e);
    	}
    
    	try {
    		BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"), 512);
    		String theWord = null;
    		do {
    			theWord = br.readLine();
    			if (theWord != null && !"".equals(theWord.trim())) {
    				_MainDict.fillSegment(theWord.trim().toCharArray());
    			}
    		} while (theWord != null);
    
    	} catch (IOException e) {
    		logger.error("ik-analyzer", e);
    
    	} finally {
    		try {
    			if (is != null) {
    				is.close();
    				is = null;
    			}
    		} catch (IOException e) {
    			logger.error("ik-analyzer", e);
    		}
    	}
    	// 加载扩展词典
    	this.loadExtDict();
    	// 加载远程自定义词库
    	this.loadRemoteExtDict();
    
    	// 从mysql加载主扩展词典
    	CustomLoadExtDict.loadMySQLExtDict(getDictRoot(), _MainDict);
    }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 2]

d. 加载停用词扩展词典方法

在这个方法最后添加从mysql加载停用词扩展词典的方法.

/**
     * 加载用户扩展的停止词词典
     */
    private void loadStopWordDict() {
    	// 建立主词典实例
    	_StopWords = new DictSegment((char) 0);
    
    	// 读取主词典文件
    	Path file = PathUtils.get(getDictRoot(), Dictionary.PATH_DIC_STOP);
    
    	InputStream is = null;
    	try {
    		is = new FileInputStream(file.toFile());
    	} catch (FileNotFoundException e) {
    		logger.error(e.getMessage(), e);
    	}
    
    	try {
    		BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"), 512);
    		String theWord = null;
    		do {
    			theWord = br.readLine();
    			if (theWord != null && !"".equals(theWord.trim())) {
    				_StopWords.fillSegment(theWord.trim().toCharArray());
    			}
    		} while (theWord != null);
    
    	} catch (IOException e) {
    		logger.error("ik-analyzer", e);
    
    	} finally {
    		try {
    			if (is != null) {
    				is.close();
    				is = null;
    			}
    		} catch (IOException e) {
    			logger.error("ik-analyzer", e);
    		}
    	}
    
    	// 加载扩展停止词典
    	List<String> extStopWordDictFiles = getExtStopWordDictionarys();
    	if (extStopWordDictFiles != null) {
    		is = null;
    		for (String extStopWordDictName : extStopWordDictFiles) {
    			logger.info("[Dict Loading] " + extStopWordDictName);
    
    			// 读取扩展词典文件
    			file = PathUtils.get(getDictRoot(), extStopWordDictName);
    			try {
    				is = new FileInputStream(file.toFile());
    			} catch (FileNotFoundException e) {
    				logger.error("ik-analyzer", e);
    			}
    			// 如果找不到扩展的字典，则忽略
    			if (is == null) {
    				continue;
    			}
    			try {
    				BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"), 512);
    				String theWord = null;
    				do {
    					theWord = br.readLine();
    					if (theWord != null && !"".equals(theWord.trim())) {
    						// 加载扩展停止词典数据到内存中
    						_StopWords.fillSegment(theWord.trim().toCharArray());
    					}
    				} while (theWord != null);
    
    			} catch (IOException e) {
    				logger.error("ik-analyzer", e);
    
    			} finally {
    				try {
    					if (is != null) {
    						is.close();
    						is = null;
    					}
    				} catch (IOException e) {
    					logger.error("ik-analyzer", e);
    				}
    			}
    		}
    	}
    
    	// 加载远程停用词典
    	List<String> remoteExtStopWordDictFiles = getRemoteExtStopWordDictionarys();
    	for (String location : remoteExtStopWordDictFiles) {
    		logger.info("[Dict Loading] " + location);
    		List<String> lists = getRemoteWords(location);
    		// 如果找不到扩展的字典，则忽略
    		if (lists == null) {
    			logger.error("[Dict Loading] " + location + "加载失败");
    			continue;
    		}
    		for (String theWord : lists) {
    			if (theWord != null && !"".equals(theWord.trim())) {
    				// 加载远程词典数据到主内存中
    				logger.info(theWord);
    				_StopWords.fillSegment(theWord.trim().toLowerCase().toCharArray());
    			}
    		}
    	}
    
    	// 从mysql加载停用词扩展词典
    	CustomLoadExtDict.loadMySQLStopwordDict(getDictRoot(), _StopWords);
    }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 3]

到此为止, 我们的自定义热更新方案就已经完成了, 下面我们来看下效果.

**安装IK Analyzer**

这里我的Elasticsearch是安装在Windows上的, 如果是在CentOS上操作也一样.

(1) 将我们修改后的源码打包, 解压后放在Elasticsearch安装目录的plugins目录下. 注意必须在plugins下创建一个目录来安装IK Analyzer插件.

(2) 添加mysql-connector-java-x.x.x.jar

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 4]

**实例**

(1) 使用IK默认词库来分析文本"我是真的蓝瘦香菇!".

GET /_analyze
    {
      "analyzer": "ik_max_word",
      "text": "我是真的蓝瘦香菇!"
    }

结果:

[ 我, 真的, 蓝, 瘦, 香菇, 香, 菇 ]

(2) 添加自定义词库

在t\_hot\_words表中添加"蓝瘦香菇", 在t\_hot\_stop\_words表中添加"真的".

GET /_analyze
    {
      "analyzer": "ik_max_word",
      "text": "我是真的蓝瘦香菇!"
    }

结果:

[ 我, 蓝瘦香菇, 蓝, 瘦, 香菇, 香, 菇 ]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191115144400274.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191115145730635.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191115150509396.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191115150659356.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191115151540622.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70