from-size VS scroll-scan

拼搏现实的明天。 2022-07-24 04:17 37阅读 0赞

# 1.from-size #

ElasticSearch可以用一种分页的形式来查询数据——from-size，[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html][https_www.elastic.co_guide_en_elasticsearch_reference_current_search-request-from-size.html]。

1.  from：定义从哪里开始拿数据。
2.  size：定义一共拿多少条数据。

from-size的工作原理是：如size=10&from=100，那么ElasticSearch会从每个Shard里取出100条数据，然后再排序，取出前10条。由此观之，from-size的效率必然不会很高，特别是分页越深，需要排序的数据越多，其效率就越低。  
另外，ElasticSearch对于from-size的默认分页深度的10000，如果超过10000就会报错如下：

$ curl -XGET 'http://127.0.01:9200/test-index/test-type/_search?size=2&from=10000&pretty=true'
    {
      "error" : {
        "root_cause" : [ {
          "type" : "query_phase_execution_exception",
          "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [10002]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."
        } ],
        "type" : "search_phase_execution_exception",
        "reason" : "all shards failed",
        "phase" : "query",
        "grouped" : true,
        "failed_shards" : [ {
          "shard" : 0,
          "index" : "passenger0510",
          "node" : "KQcH1OgUQDGAJX9ojz2OqQ",
          "reason" : {
            "type" : "query_phase_execution_exception",
            "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [10002]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."
          }
        } ]
      },
      "status" : 500
    }

但是，由我的工作经验即使在from=2000000的时候，响应时间也不是太慢，5秒左右。同时，ElasticSearch也提供了修改默认分页深度的参数：

curl -XPUT "http://127.0.0.1:9200/test-index/_settings" -d '{
        "index": {
            "max_result_window": 10000000
        }
    }'

# 2.scroll-scan #

## 2.1scroll ##

ElasticSearch还提供了另一种高校的分页查询方式——scroll-scan， [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html][https_www.elastic.co_guide_en_elasticsearch_reference_current_search-request-scroll.html]。  
scroll类似于传统的cursor方式，我的理解是JDBC的ResultSet。scroll才是ElasticSearch提供的用于查询大数据的神器，但是不是为了解决实时的查询。  
注意：从 scroll 请求返回的结果反映了 search 发生时刻的索引状态，就像一个快照。后续的对文档的改动（索引、更新或者删除）都只会影响后面的搜索请求。  
但是，为了使用scroll，你需要首先获取\_scroll\_id，同时可以为这个\_scroll\_id设置保存时长。 **注意：\_scroll\_id的长度与Shards的数目有关。**：

$curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?scroll=1m' -d '{"query":{"match_all":{}}}'
    {"_scroll_id":"cXVlcnlBbmRGZXRjaDsxOzYzOmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MDs=","took":37,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2068128,"max_score":1.0,"hits":[{"_index":"test-index","_type":"test-index","_id":"10781773","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1947331466223","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2069595890757","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1544758231677","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2069367750853","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1947537248223","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2069233668997","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"8572333","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1672543738239","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2419961040135","_score":1.0}]}}

然后，使用上面得到的\_scroll\_id获取下一批次的结果：

$curl -XGET 'http://127.0.0.1:9200/_search/scroll?scroll=1m' -d 'cXVlcnlBbmRGZXRjaDsxOzYzOmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MDs='
    {"_scroll_id":"cXVlcnlBbmRGZXRjaDsxOzYzOmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MDs=","took":20,"timed_out":false,"terminated_early":true,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2068128,"max_score":1.0,"hits":[{"_index":"test-index","_type":"test-index","_id":"2068822106901","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2467124554218","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"14649373","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"6412093","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1493598471617","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1477453","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1949882913855","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2419962095719","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"1484893","_score":1.0},{"_index":"test-index","_type":"test-index","_id":"2065404665444","_score":1.0}]}}

## 2.2scroll-scan ##

scroll保持了哪些结果已经返回的记录，所以能更加高效地返回排序的结果。但是，按照默认设定排序结果仍然需要代价。  
一般来说，你仅仅想要找到结果，不关心顺序。你可以通过组合scroll和scan来关闭任何打分或者排序，以最高效的方式返回结果。你需要做的就是将search\_type=scan 加入到查询的字符串中：

curl -XGET 'http://127.0.0.1:9200/test-index/test-type/_search?pretty=true&search_type=scan&scroll=10m&size=10' -d '{"query":{"match_all":{}}}'
    {"_scroll_id":"c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7","took":9,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2068128,"max_score":0.0,"hits":[]}}

同样，然后用得到的\_scroll\_id去查询下一批次的结果。

curl -XGET 'http://127.0.0.1:9200/_search/scroll?scroll=10m' -d 'c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7'
    {"_scroll_id":"c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7","took":12,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2068128,"max_score":0.0,"hits":[{"_index":"test-index","_type":"test-type","_id":"10781773","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"1947331466223","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"2069595890757","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"1544758231677","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"2069367750853","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"1947537248223","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"2069233668997","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"8572333","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"1672543738239","_score":0.0},{"_index":"test-index","_type":"test-type","_id":"2419961040135","_score":0.0}]}}

注意：在查询阶段，每个Shard都会把ID保存到memory中直到timeout。  
不难看出scroll-scan的查询分两个阶段：

1.  第一步执行一个query，并返回一个\_scroll\_id；
2.  第二步滚动document。迭代第二步，得到新的\_scroll\_id，然后获取其他的documents。

注意：

1.  初始查询请求和每次连续的scroll请求，都会返回一个新的\_scroll\_id，并且只有最新的\_scroll\_id可用；
2.  如果scroll大数据集，必须使用scan。否则，可能有重复结果；
3.  size是对每个Shard指定的。如果有10个Shards且size=5，那么会返回50个documents。

## 2.3清除scroll id ##

scroll id会在timeout时自动清除，但是如果保存scroll id的时间较长，且Shards较多（scroll id越大），而且很多次回滚，这样会占用不少额外的memory。 ElasticSearch也允许你显示的清除scroll id：

curl -XDELETE "http://127.0.0.1:9200/_search/scroll" -d '{
        "scroll_id" : ["c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7"]
    }'

或

curl -XDELETE "http://127.0.0.1:9200/_search/scroll" -d 'c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7'

多个scroll id放在数组里一起删除：

curl -XDELETE "http://127.0.0.1:9200/_search/scroll" -d '{
        "scroll_id" : ["c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7","c2NhbjsxOzY0OmRVRUN4TG5TUXYyaXRIT0Q5SUxiR0E7MTt0b3RhbF9oaXRzOjIwNjgxMjg7"]
    }'

当然，也可以不用记住scroll id的同时把所有scroll id删除：

curl -XDELETE "http://127.0.0.1:9200/_search/scroll/_all"

[https_www.elastic.co_guide_en_elasticsearch_reference_current_search-request-from-size.html]: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html
[https_www.elastic.co_guide_en_elasticsearch_reference_current_search-request-scroll.html]: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html