Elasticsearch分页API

Bertha 。 2023-06-27 03:27 59阅读 0赞

> 本文基于Elasticsearch7.x版本.

在学习本篇博客前先了解下[Elasticsearch全文搜索之基础语法API][Elasticsearch_API]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]

### Rest API ###

Elasticsearch分页api有三种:

*  from/size
 *  search after
 *  scroll

search after 和 scroll用来解决深度分页时的性能问题.

**分布式系统中深度分页的问题**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 1]

Elasticsearch是分布式系统, 数据保存在不同机器的分片上, 分页查询数据时默认按照相关度分数排序(降序). 当我们进行一个分页查询from=990, size=10时:

*  首先在每个分片上先都获取 1000 个文档
 *  通过 Coordinating Node 聚合所有结果
 *  再通过排序选取前 1000 个文档

页数越深，占用内存就越多, 为了避免深度分页带来的内存开销, Elasticsearch默认分页搜索的最大文档数为10000(Index.max\_result\_window)

**添加搜索实例数据**

POST /blogs/_bulk
    {"index": {}}
    {"post_date": "2020-01-01", "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen.", "author_id": 11401}
    {"index": {}}
    {"post_date": "2020-01-02", "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis.", "author_id": 11402}
    {"index": {}}
    {"post_date": "2020-01-03", "title": "My dog barks", "content": "I see a lot of barking dogs on the road.", "author_id": 11403}

**from/size**

from+size必须小于10000.

GET /blogs/_search
    {
      "query": {
        "match": {
          "content": "rabbits"
        }
      },
      "sort": [
        {
          "author_id": {
            "order": "desc"
          }
        }
      ], 
      "from": 0, 
      "size": 2
    }

**search after**

search after用来实时的获取下一页文档信息, 它不支持指定页数(from), 只能往下翻.

(1) 语法

*  第⼀步搜索需要指定 sort，并且保证值是唯⼀的（可以通过加⼊ \_id 保证唯⼀性）
 *  然后使用上⼀次搜索结果中最后⼀个文档的 sort 值进行查询.

(2) 实例

a. 第一步不指定search\_after

GET /blogs/_search
    {
      "size": 1, 
      "query": {
        "match": {
          "content": "rabbits"
        }
      },
      "sort": [
        {
          "post_date": {
            "order": "desc"
          },
          "_id": {
            "order": "asc"
          }
        }
      ]
    }

结果:

"hits" : [
      {
        "_index" : "blogs",
        "_type" : "_doc",
        "_id" : "hF_5QG8Bqop3iLDDiYtZ",
        "_score" : null,
        "_source" : {
          "post_date" : "2020-01-02",
          "title" : "Keeping pets healthy",
          "content" : "My quick brown fox eats rabbits on a regular basis.",
          "author_id" : 11402
        },
        "sort" : [
          1577923200000,
          "hF_5QG8Bqop3iLDDiYtZ"
        ]
      }
    ]

b. 第二步指定search\_after, 值为上一步搜索结果里的sort值.

GET /blogs/_search
    {
      "size": 1, 
      "query": {
        "match": {
          "content": "rabbits"
        }
      },
      "search_after": [1577923200000, "hF_5QG8Bqop3iLDDiYtZ"],
      "sort": [
        {
          "post_date": {
            "order": "desc"
          },
          "_id": {
            "order": "asc"
          }
        }
      ]
    }

**scroll**

scroll用来处理大数据量的分页搜索, 不适用于实时的分页搜索. scroll分页搜索每次都会创建一个快照, 新数据写入只能影响到后续的分页搜索.

(1) 语法

*  第⼀步搜索需要指定scroll, scroll值为快照的保留时间, 超过时间就不能再使用了.
 *  第二步需要指定scroll 和 scroll\_id. scroll\_id来自于上一步的搜索结果.

(2) 实例

a. 第一步指定scroll

GET /blogs/_search?scroll=1m
    {
      "size": 1,
      "query": {
        "match": {
          "content": "rabbits"
        }
      }
    }

结果:

{
      "_scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAvnAWcFRZdW9oa2JTQzZVRENYcXVGQjdTZw==",
      "took" : 0,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 2,
          "relation" : "eq"
        },
        "max_score" : 0.5619609,
        "hits" : [
          {
            "_index" : "blogs",
            "_type" : "_doc",
            "_id" : "g1_5QG8Bqop3iLDDiYtZ",
            "_score" : 0.5619609,
            "_source" : {
              "post_date" : "2020-01-01",
              "title" : "Quick brown rabbits",
              "content" : "Brown rabbits are commonly seen.",
              "author_id" : 11401
            }
          }
        ]
      }
    }

b. 第二步指定scroll和scroll\_id

GET /_search/scroll
    {
      "scroll": "1m",
      "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAvnAWcFRZdW9oa2JTQzZVRENYcXVGQjdTZw=="
    }

### Java API ###

下面介绍Elasticsearch Java Client 的使用, 我们来将上文的实例转化为 Java Client.

(1) main方法

public static void main(String[] args) throws IOException {
        RestHighLevelClient client = new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost("localhost", 9200, "http")));
    
        bulkIndex(client);
    
        basePagination(client);
    
        searchAfter(client);
    
        scroll(client);
    
        client.close();
    }

新增文档和查询文档请求不要一起执行, 这样会查不到文档, 因为新增文档后需要1s时间进行倒排索引创建, 这也是ES被称为近实时的原因.

(2) 添加搜索数据

private static void bulkIndex(RestHighLevelClient client) throws IOException {
        BulkRequest bulkRequest = new BulkRequest();
    
        bulkRequest.add(new IndexRequest("blogs").id("1")
                .source(XContentType.JSON, "post_date", "2020-01-01", "title", "Quick brown rabbits", "content", "Brown rabbits are commonly seen.", "author_id", 11401));
        bulkRequest.add(new IndexRequest("blogs").id("2")
                .source(XContentType.JSON, "post_date", "2020-01-02", "title", "Keeping pets healthy", "content", "My quick brown fox eats rabbits on a regular basis.", "author_id", 11402));
        bulkRequest.add(new IndexRequest("blogs").id("3")
                .source(XContentType.JSON, "post_date", "2020-01-03", "title", "My dog barks", "content", "I see a lot of barking dogs on the road.", "author_id", 11403));
    
        client.bulk(bulkRequest, RequestOptions.DEFAULT);
    }

(3) from/size

private static void basePagination(RestHighLevelClient client) throws IOException {
        SearchRequest searchRequest = new SearchRequest("blogs");
    
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("content", "rabbits");
        searchSourceBuilder.query(matchQueryBuilder);
    
        searchSourceBuilder.sort("author_id", SortOrder.DESC);
        searchSourceBuilder.from(0);
        searchSourceBuilder.size(2);
    
        searchRequest.source(searchSourceBuilder);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHit[] hits = searchResponse.getHits().getHits();
        for (SearchHit hit : hits) {
            System.out.println(hit.getSourceAsString());
        }
    }

(4) search after

private static void searchAfter(RestHighLevelClient client) throws IOException {
        SearchRequest searchRequest = new SearchRequest("blogs");
    
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("content", "rabbits");
        searchSourceBuilder.query(matchQueryBuilder);
    
        searchSourceBuilder.sort("post_date", SortOrder.DESC).sort("_id", SortOrder.ASC);
        searchSourceBuilder.size(1);
    
        searchRequest.source(searchSourceBuilder);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHit[] hits = searchResponse.getHits().getHits();
        while (hits != null && hits.length > 0) {
            Object[] sortValues = null;
            for (SearchHit hit : hits) {
                System.out.println(hit.getSourceAsString());
                sortValues = hit.getSortValues();
            }
    
            //第二步: 下一页
            searchSourceBuilder.searchAfter(sortValues);
            searchRequest.source(searchSourceBuilder);
            searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
            hits = searchResponse.getHits().getHits();
        }
    }

(5) scroll

private static void scroll(RestHighLevelClient client) throws IOException {
        final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
    
        SearchRequest searchRequest = new SearchRequest("blogs");
        searchRequest.scroll(scroll);
    
        MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("content", "rabbits");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(matchQueryBuilder);
        searchSourceBuilder.size(1);
        searchRequest.source(searchSourceBuilder);
    
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        String scrollId = searchResponse.getScrollId();
        SearchHit[] hits = searchResponse.getHits().getHits();
        while (hits != null && hits.length > 0) {
            for (SearchHit hit : hits) {
                System.out.println(hit.getSourceAsString());
            }
    
            //第二步: 下一页
            SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
            searchScrollRequest.scroll(scroll);
            searchResponse = client.scroll(searchScrollRequest, RequestOptions.DEFAULT);
            scrollId = searchResponse.getScrollId();
            hits = searchResponse.getHits().getHits();
        }
    
        //清除scroll快照
        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
    }

[Elasticsearch_API]: https://blog.csdn.net/litianxiang_kaola/article/details/103804382
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200102145725993.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191230172216765.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70