自定义分词器

￡神魔★判官ぃ 2022-11-22 00:28 255阅读 0赞

在Es中，一个字段可以定义多个字段类型，并设置分词器。

Es分词器由三部分组成：character filter:分词前过滤；tokenizer:按照一定逻辑规则分词；token filter:对分词结果处理。

1 用keyword分词（输入即输出），并在分词前过滤掉html

post \_analyze  
\{  
  "tokenizer":"keyword",  
  "char\_filter":\["html\_strip"\],  
  "text":"<b>hello word</b>"  
\}

post \_analyze  
\{  
  "tokenizer":"path\_hierarchy",  
  "text":"/usr/local/java/elasticsearch"  
\}

3 按词拆分，分词前把 - 转换成 \_

post \_analyze  
\{  
  "tokenizer":"standard",  
  "char\_filter":\[  
      \{  
        "type":"mapping",  
        "mappings":\["- => \_"\]  
      \}  
    \],  
  "text":"123-456, I-test, test-990 650-555-1234"  
\}

4 把表情符号替换成单词

post \_analyze  
\{  
  "tokenizer":"standard",  
  "char\_filter":\[  
      \{  
        "type":"mapping",  
        "mappings":\[":) => happy",":( => sad"\]  
      \}  
    \],  
    "text":\["I am feeling :)","feeling :("\]  
\}

5 分词后停用词过滤掉 近义词转换  
post \_analyze  
\{  
  "tokenizer":"standard",  
  "filter":\["stop","snowball"\],  
  "text":\["The girls in China are playing this games!"\]  
\}

6 分词后停用词过滤掉 近义词转换

post \_analyze  
\{  
  "tokenizer":"whitespace",  
  "filter":\["stop","snowball"\],  
  "text":\["The rain in Spain falls mainly on the plain."\]  
\}

7 The 小写后 被当做停用词删除  
post \_analyze  
\{  
  "tokenizer":"whitespace",  
  "filter":\["lowercase","stop","snowball"\],  
  "text":\["The girls in China are playing this games!"\]  
\}

8 正则表达式  
post \_analyze  
\{  
  "tokenizer":"standard",  
  "char\_filter":\[  
      \{  
        "type":"pattern\_replace",  
        "pattern":"http://(.\*)",  
        "replacement":"$0"  
      \}  
    \],  
  "text":"http://www.elastic.co"  
\}

9 自定义分词器

\# 自定义分词器  
设置多个自定义分词器 在analyzer节点下，增加多个下级节点，一个下级节点定义一个分词器  
delete my\_index  
put my\_index  
\{  
  "settings":\{  
    "analysis":\{  
      "analyzer":\{  
        "my\_custom\_analyzer":\{  
          "type":"custom",  
          "char\_filter":\["emoticons"\],  
          "tokenizer":"punctuation",  
          "filter":\["lowercase","english\_stop"\]  
        \}  
      \},  
      "tokenizer":\{  
        "punctuation":\{  
          "type":"pattern",  
          "pattern":"\[.,!?\]"  
        \}  
      \},  
      "char\_filter":\{  
        "emoticons":\{  
          "type":"mapping",  
          "mappings":\[":) => happy"\]  
        \}  
      \},  
      "filter":\{  
        "english\_stop":\{  
          "type":"stop",  
          "stopwords":"\_english\_"  
        \}  
      \}  
    \}  
  \}  
\}

测试自定义分词器  
post my\_index/\_analyze  
\{  
  "analyzer":"my\_custom\_analyzer",  
  "text":"i am a :) person , and you ?"  
\}