elasticsearch bucket 之rare terms聚合

╰+哭是因爲堅強的太久メ 2024-04-01 18:19 76阅读 0赞

#### 文章目录 ####

*  1、背景
 *  2、需求
 *  3、前置准备
 *   *  3.1 准备mapping
     *  3.2 准备数据
 *  4、实现需求
 *   *  4.1 dsl
     *  4.2 java代码
     *  4.3 运行结果
 *  5、max\_doc\_count 和 search.max\_buckets
 *  6、注意事项
 *  7、完整代码
 *  8、参考文档

## 1、背景 ##

我们知道当我们使用 `terms`聚合时，当修改默认顺序为`_count asc`时，统计的结果是不准备的，而且官方也不推荐我们这样做，而是推荐使用`rare terms`聚合。`rare terms`是一个`稀少`的term聚合，可以一定程度的解决升序问题。

## 2、需求 ##

统计`province`字段中包含`上和湖`的term数据，并且最多只能出现2次。获取到聚合后的结果。

## 3、前置准备 ##

### 3.1 准备mapping ###

PUT /index_person
    {
      "settings": {
        "number_of_shards": 1
      },
      "mappings": {
        "properties": {
          "id": {
            "type": "long"
          },
          "name": {
            "type": "keyword"
          },
          "province": {
            "type": "keyword"
          },
          "sex": {
            "type": "keyword"
          },
          "age": {
            "type": "integer"
          },
          "pipeline_province_sex":{
            "type": "keyword"
          },
          "address": {
            "type": "text",
            "analyzer": "ik_max_word",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }

### 3.2 准备数据 ###

PUT /_bulk
    {"create":{"_index":"index_person","_id":1}}
    {"id":1,"name":"张三","sex":"男","age":20,"province":"湖北","address":"湖北省黄冈市罗田县匡河镇"}
    {"create":{"_index":"index_person","_id":2}}
    {"id":2,"name":"李四","sex":"男","age":19,"province":"江苏","address":"江苏省南京市"}
    {"create":{"_index":"index_person","_id":3}}
    {"id":3,"name":"王武","sex":"女","age":25,"province":"湖北","address":"湖北省武汉市江汉区"}
    {"create":{"_index":"index_person","_id":4}}
    {"id":4,"name":"赵六","sex":"女","age":30,"province":"北京","address":"北京市东城区"}
    {"create":{"_index":"index_person","_id":5}}
    {"id":5,"name":"钱七","sex":"女","age":16,"province":"北京","address":"北京市西城区"}
    {"create":{"_index":"index_person","_id":6}}
    {"id":6,"name":"王八","sex":"女","age":45,"province":"北京","address":"北京市朝阳区"}
    {"create":{"_index":"index_person","_id":7}}
    {"id":7,"name":"九哥","sex":"男","age":25,"province":"上海市","address":"上海市嘉定区"}

## 4、实现需求 ##

### 4.1 dsl ###

GET /index_person/_search
    {
      "size": 0,
      "aggs": {
        "agg_province": {
          "rare_terms": {
            "field": "province",
            "max_doc_count": 2,
            "precision": 0.01,
            "include": "(.*上.*|.*湖.*|.*江.*)",
            "exclude": ["江苏"],
            "missing": "default省"
          }
        }
      }
    }

### 4.2 java代码 ###

@Test
    @DisplayName("稀少的term聚合，类似按照 _count asc 排序的terms聚合，但是terms聚合中按照_count asc的结果是不准的，需要使用 rare terms 聚合")
    public void agg01() throws IOException {
        
    
        SearchRequest searchRequest = new SearchRequest.Builder()
                .size(0)
                .index("index_person")
                .aggregations("agg_province", agg ->
                        agg.rareTerms(rare ->
                                // 稀有词 的字段
                                rare.field("province")
                                        // 该稀有词最多可以出现在几个文档中，最大值为100，如果要调整，需要修改search.max_buckets参数的值(尝试修改这个值，不生效)
                                        // 在该例子中，只要是出现的次数<=2的聚合都会返回
                                        .maxDocCount(2L)
                                        // 内部布谷鸟过滤器的精度，精度越小越准，但是相应的消耗内存也越多，最小值为 0.00001，默认值为 0.01
                                        .precision(0.01)
                                        // 应该包含在聚合的term, 当是单个字段是，可以写正则表达式
                                        .include(include -> include.regexp("(.*上.*|.*湖.*|.*江.*)"))
                                        // 排出在聚合中的term，当是集合时，需要写准确的值
                                        .exclude(exclude -> exclude.terms(Collections.singletonList("江苏")))
                                        // 当文档中缺失province字段时，给默认值
                                        .missing("default省")
                        )
                )
                .build();
        System.out.println(searchRequest);
        SearchResponse<Object> response = client.search(searchRequest, Object.class);
        System.out.println(response);
    }

**一些注意事项都在注释中。**

### 4.3 运行结果 ###

![运行结果][32a1f753d76243db9a99643784172bdb.png]

## 5、max\_doc\_count 和 search.max\_buckets ##

![max\_doc\_count 和 search.max\_buckets][max_doc_count _ search.max_buckets]

## 6、注意事项 ##

1.  `rare terms`统计返回的数据没有`大小`限制，而且受`max_doc_count`参数的限制，比如：如果复合 max\_doc\_count 的分组有60个，那么这60个分组会直接返回。
2.  `max_doc_count`的值最大为`100`，貌似不能修改。
3.  如果一台节点聚合收集的结果过多，那么很容易超过 `search.max_buckets`的值，此时就需要修改这个值。

# 临时修改
    PUT /_cluster/settings
    {"transient": {"search.max_buckets": 65536}}
    
    # 永久修改
    PUT /_cluster/settings
    {"persistent": {"search.max_buckets": 65536}}

## 7、完整代码 ##

[https://gitee.com/huan1993/spring-cloud-parent/blob/master/es/es8-api/src/main/java/com/huan/es8/aggregations/bucket/RareTermsAggs.java][https_gitee.com_huan1993_spring-cloud-parent_blob_master_es_es8-api_src_main_java_com_huan_es8_aggregations_bucket_RareTermsAggs.java]

## 8、参考文档 ##

1.  [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-rare-terms-aggregation.html][https_www.elastic.co_guide_en_elasticsearch_reference_current_search-aggregations-bucket-rare-terms-aggregation.html]

[32a1f753d76243db9a99643784172bdb.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/954c04af53ec41338a3b6729579ffba8.png
[max_doc_count _ search.max_buckets]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/1010b0d3f6d54fc882595c3ae6ac7c8a.png
[https_gitee.com_huan1993_spring-cloud-parent_blob_master_es_es8-api_src_main_java_com_huan_es8_aggregations_bucket_RareTermsAggs.java]: https://gitee.com/huan1993/spring-cloud-parent/blob/master/es/es8-api/src/main/java/com/huan/es8/aggregations/bucket/RareTermsAggs.java
[https_www.elastic.co_guide_en_elasticsearch_reference_current_search-aggregations-bucket-rare-terms-aggregation.html]: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-rare-terms-aggregation.html