测试es from size error，from + size must be less than or equal to: [10000]异常问题

清疚 2023-02-12 04:28 3阅读 0赞

### 测试案例 ###

1，查询第三页每页3334

\{  
    "page":3,  
    "size":3334  
\}

2，查询第一页，每页100001

\{  
    "page":1,  
    "size":100001  
\}

3，查询第26页，每页200条

\{  
    "page":26,  
    "size":200  
\}

以上查询都会报异常。

### 控制方案 ###

如何控制发生异常，常见如

控制每页数量不大于10000，但是如情况1，3还是会发生异常。

1，可能会问，es默认最大查询量10000，每页小于10000啊。但是es分页，会首先取到所有数据，再分页

总量3\*3334和26\*200，总量都大于10000，所以异常。

2，虽然实际情况每页不会3334这么大，会像第3种，每页会查询200，然后前端对20条数据再次分页，当翻页总量大于200，后台es才会查询下一页。实际的操作也不会翻这么多页（前端分页200/size\*26）。但是还是存在异常的可能性。以上两种只是控制

每页数据量10000以内，不能避免发生异常的可能性。

3，控制查询总量，有没有意义，比如我只限制查询总量5000条或者10000条

我觉得没意义，首先比如查询到100万条数据，返回总量肯定100万，你告诉查询5000条或者10000条肯定是错的；其次你也没法保证查询的总量是5000或1万。

推荐：使用search\_after进行分页，参考：

[https://mp.csdn.net/console/editor/html/106314361][https_mp.csdn.net_console_editor_html_106314361]

[https://blog.csdn.net/zzh920625/article/details/84593590][https_blog.csdn.net_zzh920625_article_details_84593590]

### 代码如下 ###

@Override
      public Rpage<ResShopProductExtensionInfoDataVO> selectList(ReqShopProductExtensionInfoSearchVO req) {
        try {
          SearchRequest searchRequest = new SearchRequest(EsOperateTables.SHOP_PRODUCT_EXTENSION_INFO.getAlias());
          SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
          List<String> fields = Arrays.stream(ResShopProductExtensionInfoVO.class.getDeclaredFields())
              /*将类属性名转为es中带下划线的字段名*/
              .map(field -> BizStringUtils.humpToLine(field.getName()))
              .collect(Collectors.toList());
          sourceBuilder.fetchSource(fields.toArray(new String[fields.size()]), null);
          BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
          ShopProductExtensionInfoConverter.setSomeCondition(req, sourceBuilder, boolQueryBuilder);
          sourceBuilder.size(req.getSize() > 10000 ? 10000 : req.getSize());
          sourceBuilder.from((req.getPage() - 1) * req.getSize());
          searchRequest.source(sourceBuilder);
          log.info("terms查询DSL语句：" + searchRequest.toString());
          SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
          long totalHits = 0L;
          if (searchResponse != null) {
            totalHits = searchResponse.getHits().getTotalHits();
          }
          long maxCount = new Long(ConstantsUtils.QueryPartams.MAX_RESULT_COUNT).intValue();//100000
          long totalCount = totalHits;
          totalHits = totalHits <= maxCount ? totalHits : maxCount;
          List<Map<String, Object>> mapList = Lists.newArrayListWithCapacity(searchResponse.getHits().getHits().length);//每页200实际查询的量
          Arrays.stream(searchResponse.getHits().getHits()).forEach(hit -> mapList.add(hit.getSourceAsMap()));
          YhPageBase page = new YhPageBase(mapList, totalHits, req.getSize().longValue(), req.getPage().longValue());
          List<ResShopProductExtensionInfoVO> resList = new ArrayList<>();
          if (CollectionUtils.isNotEmpty(page.getList())) {
            resList = JSONObject.parseArray(JSONObject.toJSONString(page.getList()), ResShopProductExtensionInfoVO.class);
          }
          Page rpage = new Page();
          rpage.setTotalNum((int) totalCount);//总量
          rpage.setPageSize(req.getSize());
          rpage.setPageNo(req.getPage());
          rpage.setResult(adapter(resList, req));//每页实际查询量
          return Rpage.success(rpage);
        } catch (Exception e) {
          log.error("查询出单方式配置es信息出错", e);
        }
        return Rpage.success(new Page<>());
      }

[https_mp.csdn.net_console_editor_html_106314361]: https://mp.csdn.net/console/editor/html/106314361
[https_blog.csdn.net_zzh920625_article_details_84593590]: https://blog.csdn.net/zzh920625/article/details/84593590