ElasticSearch学习 - （八）安装中文分词器IK和拼音分词器

男娘i 2022-05-29 00:10 344阅读 0赞

## IK分词器 ##

**下载地址**:[https://github.com/medcl/elasticsearch-analysis-ik][https_github.com_medcl_elasticsearch-analysis-ik]

也可以在这个地址选择:[https://github.com/medcl/elasticsearch-analysis-ik/releases][https_github.com_medcl_elasticsearch-analysis-ik_releases]  
这个下载下来了可以直接使用, 所以推荐下载这个

选择elasticsearch对应版本的分词器进行下载  
![这里写图片描述][70]  
进入到对应页面下载  
![这里写图片描述][70 1]  
找到下载好的文件，右键，解压到当前文件夹  
![这里写图片描述][70 2]  
进入文件夹，cmd进入dos窗口，使用maven打包  
![这里写图片描述][70 3]  
输入命令，打包，前提是安装好了maven  
![这里写图片描述][70 4]  
命令：

mvn package

打包好了过后，当前目录多了一个target文件夹，点击进入  
![这里写图片描述][70 5]  
点击进入releases文件夹  
![这里写图片描述][70 6]  
右键，解压到当前文件夹  
![这里写图片描述][70 7]  
进入解压后的文件夹，复制所有文件  
![这里写图片描述][70 8]  
找到elasticsearch安装目录，在plugins文件夹下面新建ik（任意取名，方便记忆）文件夹，把刚才复制的文件粘贴到ik文件夹下面  
![这里写图片描述][70 9]  
![这里写图片描述][70 10]

## 拼音分词器 ##

**下载地址**：[https://github.com/medcl/elasticsearch-analysis-pinyin][https_github.com_medcl_elasticsearch-analysis-pinyin]

也可以在这个地址选择:[https://github.com/medcl/elasticsearch-analysis-pinyin/releases][https_github.com_medcl_elasticsearch-analysis-pinyin_releases]

下载，安装过程和ik分词器一模一样，参考上面步骤  
最终结果  
![这里写图片描述][70 11]

## 测试分词效果 ##

## elasticsearch自带分词器效果 ##

GET http://localhost:9200/_analyze?pretty=true
    {
      "analyzer" : "standard",
      "text" : "我是一名java程序员"
    }

分词效果如下:

{
    "tokens": [ { "token": "我", "start_offset": 0, "end_offset": 1, "type": "<IDEOGRAPHIC>", "position": 0 } , { "token": "是", "start_offset": 1, "end_offset": 2, "type": "<IDEOGRAPHIC>", "position": 1 } , { "token": "一", "start_offset": 2, "end_offset": 3, "type": "<IDEOGRAPHIC>", "position": 2 } , { "token": "名", "start_offset": 3, "end_offset": 4, "type": "<IDEOGRAPHIC>", "position": 3 } , { "token": "java", "start_offset": 4, "end_offset": 8, "type": "<ALPHANUM>", "position": 4 } , { "token": "程", "start_offset": 8, "end_offset": 9, "type": "<IDEOGRAPHIC>", "position": 5 } , { "token": "序", "start_offset": 9, "end_offset": 10, "type": "<IDEOGRAPHIC>", "position": 6 } , { "token": "员", "start_offset": 10, "end_offset": 11, "type": "<IDEOGRAPHIC>", "position": 7 } ] }

## 使用ik\_max\_word分词 ##

**ik\_max\_word ：会将文本做最细粒度的拆分；尽可能多的拆分出词语**

GET http://localhost:9200/_analyze?pretty=true
    {
      "analyzer" : "ik_max_word",
      "text" : "我是一名java程序员"
    }

效果如下：

{
    "tokens": [ { "token": "我", "start_offset": 0, "end_offset": 1, "type": "CN_CHAR", "position": 0 } , { "token": "是", "start_offset": 1, "end_offset": 2, "type": "CN_CHAR", "position": 1 } , { "token": "一名", "start_offset": 2, "end_offset": 4, "type": "CN_WORD", "position": 2 } , { "token": "一", "start_offset": 2, "end_offset": 3, "type": "TYPE_CNUM", "position": 3 } , { "token": "名", "start_offset": 3, "end_offset": 4, "type": "COUNT", "position": 4 } , { "token": "java", "start_offset": 4, "end_offset": 8, "type": "ENGLISH", "position": 5 } , { "token": "程序员", "start_offset": 8, "end_offset": 11, "type": "CN_WORD", "position": 6 } , { "token": "程序", "start_offset": 8, "end_offset": 10, "type": "CN_WORD", "position": 7 } , { "token": "员", "start_offset": 10, "end_offset": 11, "type": "CN_CHAR", "position": 8 } ] }

## 使用ik\_smart分词 ##

**ik\_smart：会做最粗粒度的拆分；已被分出的词语将不会再次被其它词语占有**

GET http://localhost:9200/_analyze?pretty=true
    {
      "analyzer" : "ik_smart",
      "text" : "我是一名java程序员"
    }

分词效果如下:

## 使用pinyin分词 ##

http://localhost:9200/_analyze?pretty=true
    {
      "analyzer" : "pinyin",
      "text" : "我是一名java程序员"
    }

效果如下：

{
    "tokens": [ { "token": "wo", "start_offset": 0, "end_offset": 1, "type": "word", "position": 0 } , { "token": "wsymjavacxy", "start_offset": 0, "end_offset": 11, "type": "word", "position": 0 } , { "token": "shi", "start_offset": 1, "end_offset": 2, "type": "word", "position": 1 } , { "token": "yi", "start_offset": 2, "end_offset": 3, "type": "word", "position": 2 } , { "token": "ming", "start_offset": 3, "end_offset": 4, "type": "word", "position": 3 } , { "token": "ja", "start_offset": 4, "end_offset": 6, "type": "word", "position": 4 } , { "token": "v", "start_offset": 6, "end_offset": 7, "type": "word", "position": 5 } , { "token": "a", "start_offset": 7, "end_offset": 8, "type": "word", "position": 6 } , { "token": "cheng", "start_offset": 8, "end_offset": 9, "type": "word", "position": 7 } , { "token": "xu", "start_offset": 9, "end_offset": 10, "type": "word", "position": 8 } , { "token": "yuan", "start_offset": 10, "end_offset": 11, "type": "word", "position": 9 } ] }

## IK+pinyin分词配置 ##

**创建索引和类型**  
具体含义参考这篇博客：[http://blog.csdn.net/napoay/article/details/73100110][http_blog.csdn.net_napoay_article_details_73100110]

-put http://localhost:9200/demo
    
    {
      "settings": {
        "analysis": {
            "analyzer": {
                "ik_pinyin_analyzer": {//分词器名称，自定义
                    "type": "custom",//custom表示自己定制
                    "tokenizer": "ik_max_word",//分词的策略
                    "filter":["my_pinyin", "word_delimiter"]// 对拼音和分隔的词源做处理
                }
            },
            "filter":{
                "my_pinyin":{
                    "type":"pinyin",
                    "first_letter":"prefix",
                    "padding_char":" "
                }
            }
        }
      },
      "mappings": {
        "article": {
          "properties": {
            "subject": {
              "type": "keyword",
              "fields": {
                  "pinyin": {
                      "type": "text",
                      "store": "no",
                      "term_vector": "with_positions_offsets",
                      "analyzer": "ik_pinyin_analyzer",
                      "boost": 10
                   }
               }
            }
          }
        }
      }
    }

**索引一个文档**

-post http://localhost:9200/demo/article
    
    {
      "subject": "我是一名java程序员"
    }

**中文查询**

-post http://localhost:9200/demo/article/_search
    
    {
      "query": {
        "match": {
          "subject.pinyin": "程序员"
        }
      }
    }

结果如下：

{
    "took": 7,
    "timed_out": false,
    "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 },
    "hits": { "total": 1, "max_score": 14.584841, "hits": [ { "_index": "demo", "_type": "article", "_id": "AWIeeeTJ2JGj7w9eQwEK", "_score": 14.584841, "_source": { "subject": "我是一名java程序员" } } ] } }

**拼音查询**

-post http://localhost:9200/demo/article/_search
    
    {
      "query": {
        "match": {
          "subject.pinyin": "chengxuyuan"
        }
      }
    }

查询结果：

{
    "took": 6,
    "timed_out": false,
    "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 },
    "hits": { "total": 1, "max_score": 4.3648314, "hits": [ { "_index": "demo", "_type": "article", "_id": "AWIeeeTJ2JGj7w9eQwEK", "_score": 4.3648314, "_source": { "subject": "我是一名java程序员" } } ] } }

**注意：使用pinyin分词以后，原始的字段搜索要加上.pinyin后缀，搜索原始字段没有返回结果**  
![这里写图片描述][Image 1]

[https_github.com_medcl_elasticsearch-analysis-ik]: https://github.com/medcl/elasticsearch-analysis-ik
[https_github.com_medcl_elasticsearch-analysis-ik_releases]: https://github.com/medcl/elasticsearch-analysis-ik/releases
[70]: /images/20220529/be11c236e1f24b47a903962870b7630d.png
[70 1]: /images/20220529/9229de69583740e7947eb79e7552c8e8.png
[70 2]: /images/20220529/7cdec374008f49959735f91f20fe888e.png
[70 3]: /images/20220529/766718b921b74308a6c7f83914b87b11.png
[70 4]: /images/20220529/e624c45677b546768ad703145a1f9375.png
[70 5]: /images/20220529/c89dff06c0804e569419e84eea8d6c7c.png
[70 6]: /images/20220529/d50af7d55972497088f0e8f17a67ef63.png
[70 7]: /images/20220529/c6385171cb4c4ce0ab38ae084bb5d9f0.png
[70 8]: /images/20220529/93dbc4c65f754908a101a0541249f7dd.png
[70 9]: /images/20220529/6fb40caac21244a49874876b6f787434.png
[70 10]: /images/20220529/d9f048cfcaf948eea7fd0cdbb1266370.png
[https_github.com_medcl_elasticsearch-analysis-pinyin]: https://github.com/medcl/elasticsearch-analysis-pinyin
[https_github.com_medcl_elasticsearch-analysis-pinyin_releases]: https://github.com/medcl/elasticsearch-analysis-pinyin/releases
[70 11]: /images/20220529/6aef91fbcd2241f0a90f5c8746659eec.png
[http_blog.csdn.net_napoay_article_details_73100110]: http://blog.csdn.net/napoay/article/details/73100110
[Image 1]: