elasticsearch搜索中文分词理解、类似SQL语句中的"LIKE”条件的模糊搜索以及忽略大小写搜索

我就是我 2022-05-24 08:06 587阅读 0赞

### elasticsearch作为一款搜索引擎，应用于数据库无法承受前端的搜索压力时，采用其进行数据的搜索。可以大并发架构设计中一种选择，以下是elasticsearch搜索引擎的部分规则，在实际应用中可以让我们快速熟悉和帮助解决一些问题。    ###

### **01**》不进行分词的索引建立 ###

URL： **es\_index\_test**

\{  
  "settings": \{  
    "index": \{  
      "number\_of\_shards": "4",  
      "number\_of\_replicas": "1"  
    \}  
  \},  
  "mappings": \{  
    "es\_index\_type\_test": \{  
      "properties": \{  
        "productId": \{  
          "type": "text"  
        \},  
        "productName": \{  
          "type": "keyword",  
          "index": "true"  
        \}  
      \}  
    \}  
  \}

说明：“productName”属性建立索引时，将其设置为不进行分词设置。利用wildcard搜索方式，可以实现MYSQL中的LIKE效果。例如：文档\{"productId":10001,"productName":"山鸡图"\}，可以用\{"query":\{"wildcard":\{"productName":"\*鸡\*"\}\}\}，搜索出来。

### **02**》需要进行分词的索引建立 ###

URL： **es\_index\_test**

\{  
  "settings": \{  
    "index": \{  
      "number\_of\_shards": "4",  
      "number\_of\_replicas": "1"  
    \}  
  \},  
  "mappings": \{  
    "es\_index\_type\_test": \{  
      "properties": \{  
        "productId": \{  
        "type": "text"  
        \},  
        "productName": \{  
          "type": "text",  
          "analyzer": "ik\_max\_word",  
          "search\_analyzer": "ik\_max\_word"  
        \}  
      \}  
    \}  
  \}

说明：“productName”属性建立索引时，将其设置为进行分词设置。elasticsearch默认针对中文的分词是按照一个中文字符，就是一个分词。例如：文档\{"productId":10001,"productName":"山鸡图"\}中,会拆分为“山”、“鸡”和“图”三个分词。中文分词的拆分，可以安装ik分词器进行分词拆分。例如：文档\{"productId":10001,"productName":"山鸡图"\}中，会拆分为“山鸡”和“图”两个分词。中文短句具体拆分成哪些分词是ik分词器的字典来识别的，此字典可以根据实际情况进行调整。

### **03**》忽略大小写的索引建立 ###

URL： **es\_index\_test**

\{  
  "settings": \{  
    "index": \{  
      "number\_of\_shards": "10",  
      "number\_of\_replicas": "3"  
    \},  
    "analysis": \{  
      "normalizer": \{  
        "es\_normalizer": \{  
          "filter": \[  
            "lowercase",  
            "asciifolding"  
          \],  
          "type": "custom"  
        \}  
      \}  
    \}  
  \},  
  "mappings": \{  
    "es\_index\_test": \{  
      "properties": \{  
        "productId": \{  
          "type": "text"  
        \},  
        "productName": \{  
          "type": "keyword",  
          "normalizer": "es\_normalizer",  
          "index": "true"  
        \}  
      \}  
    \}  
  \}

说明：“productName”属性建立索引时，将其设置为忽略大小写。

### 04》分词查询 ###

URL：**es\_index\_test/es\_index\_type\_test/\_analyze**

*  ik分词器以“ik\_max\_word”方式拆分

> > > > > > > \{  
> > > > > > >   "analyzer":"ik\_max\_word",  
> > > > > > >   "text":"中华人民共和国"  
> > > > > > > 
> > > > > > > 
> > > > > > > \}

*  结果

\{  
  "tokens": \[  
    \{  
      "end\_offset": 7,  
      "start\_offset": 0,  
      "position": 0,  
      "type": "CN\_WORD",  
      "token": "中华人民共和国"  
    \},  
    \{  
      "end\_offset": 4,  
      "start\_offset": 0,  
      "position": 1,  
      "type": "CN\_WORD",  
      "token": "中华人民"  
    \},  
    \{  
      "end\_offset": 2,  
      "start\_offset": 0,  
      "position": 2,  
      "type": "CN\_WORD",  
      "token": "中华"  
    \},  
    \{  
      "end\_offset": 3,  
      "start\_offset": 1,  
      "position": 3,  
      "type": "CN\_WORD",  
      "token": "华人"  
    \},  
    \{  
      "end\_offset": 7,  
      "start\_offset": 2,  
      "position": 4,  
      "type": "CN\_WORD",  
      "token": "人民共和国"  
    \},  
    \{  
      "end\_offset": 4,  
      "start\_offset": 2,  
      "position": 5,  
      "type": "CN\_WORD",  
      "token": "人民"  
    \},  
    \{  
      "end\_offset": 7,  
      "start\_offset": 4,  
      "position": 6,  
      "type": "CN\_WORD",  
      "token": "共和国"  
    \},  
    \{  
      "end\_offset": 6,  
      "start\_offset": 4,  
      "position": 7,  
      "type": "CN\_WORD",  
      "token": "共和"  
    \},  
    \{  
      "end\_offset": 7,  
      "start\_offset": 6,  
      "position": 8,  
      "type": "CN\_CHAR",  
      "token": "国"  
    \}  
  \]

*  ik分词器以“ik\_smart”方式拆分

\{  
  "analyzer":"ik\_smart",  
  "text":"中华人民共和国"

*  结果

"tokens": \[

\{  
      "end\_offset": 7,  
      "start\_offset": 0,  
      "position": 0,  
      "type": "CN\_WORD",  
      "token": "中华人民共和国"  
    \}  
  \]

*  ES默认

> \{  
>   "text":"中华人民共和国"  
> 
> 
> \}
> 
>  *  结果  
>     
> 
> \{
> 
>   "tokens": \[  
>     \{  
>       "end\_offset": 1,  
>       "start\_offset": 0,  
>       "position": 0,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "中"  
>     \},  
>     \{  
>       "end\_offset": 2,  
>       "start\_offset": 1,  
>       "position": 1,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "华"  
>     \},  
>     \{  
>       "end\_offset": 3,  
>       "start\_offset": 2,  
>       "position": 2,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "人"  
>     \},  
>     \{  
>       "end\_offset": 4,  
>       "start\_offset": 3,  
>       "position": 3,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "民"  
>     \},  
>     \{  
>       "end\_offset": 5,  
>       "start\_offset": 4,  
>       "position": 4,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "共"  
>     \},  
>     \{  
>       "end\_offset": 6,  
>       "start\_offset": 5,  
>       "position": 5,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "和"  
>     \},  
>     \{  
>       "end\_offset": 7,  
>       "start\_offset": 6,  
>       "position": 6,  
>       "type": "<IDEOGRAPHIC>",  
>       "token": "国"  
>     \}  
>   \]  
> 
> 
> \}

说明：以上三种分词拆分的方式不一样，最终产生分词的结果不相同。

### 05》数据查询-wildcard ###

URL： **es\_index\_test/es\_index\_type\_test/\_search**  
\{  
  "query":\{"wildcard":\{"productName": "山鸡图" \}\}

说明：wildcard种查询方式需要结合方法支持匹配符合，例如：\*鸡\*,ES会去匹配，在JAVA程序中构建采用。JAVA程序中采用QueryBuilders类的wildcardQuery(String name, Object text)方法。

### 06》数据查询-match    ###

URL：**es\_index\_test/es\_index\_type\_test/\_search**  
\{  
  "query":\{"match":\{"productName": "山鸡图" \}\}

说明：查询时会根据分词进行匹配，例如：“山鸡图”ES拆分为“山鸡”和“图”两个分词到ES搜索引擎内筛选出记录，最后将符合记录的数据返回。返回的记录可能包含，山鸡汤（包含“山鸡”）和山虎图（包含“图”分词）。JAVA程序中采用QueryBuilders类的matchQuery(String name, Object text)方法。

### 07》数据查询-term ###

URL： **es\_index\_test/es\_index\_type\_test/\_search**

\{  
  "query":\{  
    "term":\{  
      "productName":"山鸡图"  
    \}  
  \}  
\}

说明：只有分词完全匹配“山鸡图”这三个字后，才可以返回数据。JAVA程序中采用QueryBuilders类的termQuery(String name, Object value)方法。

### 08》数据查询-terms ###

URL： **es\_index\_test/es\_index\_type\_test/\_search**

\{  
  "query":\{  
    "terms":\{  
      "productName":\["山鸡图","山虎图"\]  
    \}  
  \}

说明：分词匹配“山鸡图”和“山虎图”返回记录。JAVA程序中采用QueryBuilders类的termsQuery(String name, String... values)方法。

### 09》删除查询出来的结果集 ###

URL：**es\_index\_test/es\_index\_type\_test/\_delete\_by\_query**

\{  
  "query":\{"wildcard":\{"productName": "\*鸡\*" \}\}

说明：删除产品名称包含“鸡”字文档。

### 10》elasticsearch中JAVA实例 ###

1、ElasticSearchProperties

package com.jd.ccc.sys.biz.yb.op.notice.config;
    
    import org.springframework.boot.context.properties.ConfigurationProperties;
    import org.springframework.stereotype.Component;
    
    import lombok.Data;
    
    /**
     * ElasticSearch搜索引擎配置参数 
     * 具体参数的配置信息在yml文件内
     * 
     * @create 2018-5-10
     * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
     *
     */
    @Data
    @Component
    @ConfigurationProperties(prefix = "elasticsearch")
    public class ElasticSearchProperties {
       /**
        * 集群名
        */
       private String clusterName;
       /**
        * 索引名称
        */
       private String indexName;
    
       /**
        * 类型名称
        */
       private String typeName;
    
       /**
        * 主节点
        */
       private String masterNode;
    
       /**
        * 从节点
        */
       private String slaveNodes;
    
    }

2、ElasticSearchConfig

package com.jd.ccc.sys.biz.yb.op.notice.config;
    
    import java.net.InetAddress;
    import java.net.UnknownHostException;
    
    import org.elasticsearch.client.Client;
    import org.elasticsearch.client.transport.TransportClient;
    import org.elasticsearch.common.settings.Settings;
    import org.elasticsearch.common.transport.InetSocketTransportAddress;
    import org.elasticsearch.transport.client.PreBuiltTransportClient;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.boot.context.properties.EnableConfigurationProperties;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    
    /**
     * 
     * 初始化一个ES搜索引擎配置
     * 
     * @create 2018-5-10
     * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
     *
     */
    @Configuration
    @EnableConfigurationProperties(ElasticSearchProperties.class)
    public class ElasticSearchConfig {
       
        private static final Logger LOGGER = LoggerFactory.getLogger(ElasticSearchConfig.class);
       
       @Autowired
       private ElasticSearchProperties elasticSearchProperties;
       
       private static final String SYS_PROPERTY="es.set.netty.runtime.available.processors";
       
       private static final String CLUSTER_NAME="cluster.name";
       
       private static final String CLIENT_SNIFF="client.transport.sniff";
    
       @Bean(name="elasticSearchCluster")
       public Client getClient() {
    
          System.setProperty(SYS_PROPERTY, "false");
          Settings settings = Settings.builder().put(CLUSTER_NAME, elasticSearchProperties.getClusterName())
                .put(CLIENT_SNIFF, false).build();
          
          TransportClient transportClient = null;
          try {
    
             String[] masters = elasticSearchProperties.getMasterNode().split(":");
             transportClient = new PreBuiltTransportClient(settings).addTransportAddress(
                   new InetSocketTransportAddress(InetAddress.getByName(masters[0]), Integer.parseInt(masters[1])));
    
             String[] slaveNodes = elasticSearchProperties.getSlaveNodes().split(",");// 逗号分隔
    
             //遍历从库信息
             for (String node : slaveNodes) {
                String[] ipPort = node.split(":");// 冒号分隔
                if (ipPort.length == 2) {
                   transportClient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(ipPort[0]),
                         Integer.parseInt(ipPort[1])));
                }
             }
             return transportClient;
          } catch (UnknownHostException e) {
             LOGGER.error("ES 客户端连接失败.{}",e);
             return null;
          }
       }
    
    }

3、服务层操作

/**
     * 查询模糊搜索产品列表的总记录数
     *
     * @param likeProductName
     *            模糊搜索产品名称的关键字
     * @param type
     *            产品类型
     * @return 总记录数
     *
     * @create 2018-5-9
     * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
     */
    private Integer queryCount(String likeProductName, String type) {
        BoolQueryBuilder builder=this.builderQueryData(likeProductName, type);
        try {
            SearchResponse searchResponse = elasticSearchCluster.prepareSearch(elasticSearchProperties.getIndexName())
                    .setTypes(elasticSearchProperties.getTypeName()).setQuery(builder)
                    .setSearchType(SearchType.DEFAULT).get();
            SearchHits hits = searchResponse.getHits();
            return (int)hits.getTotalHits();
        }catch(Exception e) {
            LOGGER.error("Server access failure,{}",e);
            return 0;
        }
    }
    
    /**
     * 拼接模糊查询筛选条件
     *
     * @param likeProductName
     *            模糊搜索产品名称的关键字
     * @param type
     *            产品类型
     * @return 筛选条件字符串
     *
     * @create 2018-5-9
     * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
     */
    private BoolQueryBuilder builderQueryData(String likeProductName, String type) {
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(QueryBuilders.matchQuery(PRODUCT_STATUS, "03"));
        if(StringUtils.isNotBlank(likeProductName)) {
            boolQueryBuilder.must(QueryBuilders.wildcardQuery(PRODUCT_NAME,"*"+likeProductName+"*"));
        }
        // 类型不为空
        if (StringUtils.isNotBlank(type)) {
            String[] types = type.split(",");
            if (types.length == 1) {
                boolQueryBuilder.must(QueryBuilders.matchQuery(INST_TYPE,type));
            } else {
                boolQueryBuilder.must(QueryBuilders.termsQuery(INST_TYPE, types));
            }
        }
        LOGGER.debug("wild card query-->{}",boolQueryBuilder.toString());
        return boolQueryBuilder;
    }
    
    /**
     * 模糊查询商品列表数据
     * @param likeProductName 模糊搜索产品名称的关键字
     * @param type 产品类型
     * @param startIndex 开始索引
     * @param pageSize 每页大小
     * @returnW
     *
     * @create 2018-5-9
     * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
     */
    private List<String> queryData(String likeProductName, String type, int startIndex, int pageSize) {
        List<String> resultList = new ArrayList<>();
        BoolQueryBuilder builder=this.builderQueryData(likeProductName, type);
        try {
    
            SearchResponse searchResponse = elasticSearchCluster.prepareSearch(elasticSearchProperties.getIndexName())
                    .setTypes(elasticSearchProperties.getTypeName()).setQuery(builder)
                    .setSearchType(SearchType.DEFAULT).setFrom(startIndex).setSize(pageSize).get();
            SearchHit[] hits = searchResponse.getHits().getHits();
    
    
            for (SearchHit hit : hits) {
                resultList.add(hit.getSourceAsString());
            }
        }catch(Exception e) {
            LOGGER.error("Server access failure,{}",e);
        }
    
        return resultList;
    }