Elasticsearch聚合分析Rest API

川长思鸟来 2023-06-28 06:26 192阅读 0赞

> 本文基于Elasticsearch7.x

### 什么是聚合(Aggregation) ###

Elasticsearch除了数据搜索之外, 提供的针对数据的统计分析功能.

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]

**聚合的分类**

*  Bucket Aggregation  
    一系列满足特定条件的文档的集合.
 *  Metric Aggregation  
    一系列数学运算, 可以对文档字段进行统计分析.
 *  Pipeline Aggregation  
    对其他聚合结果进行二次聚合.
 *  Matrix Aggregation  
    支持对多个字段进行统计分析, 并为每个字段提供一个结果矩阵.

下文会详细介绍这四类聚合.

**添加聚合实例数据**

新增电视机销售记录, 用于接下来的聚合分析.

(1) 添加数据

POST /sales/_bulk
    { "index": {}}
    { "price" : 1000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-10-28" }
    { "index": {}}
    { "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-11-05" }
    { "index": {}}
    { "price" : 3000, "color" : "绿色", "brand" : "小米", "sold_date" : "2019-05-18" }
    { "index": {}}
    { "price" : 1500, "color" : "蓝色", "brand" : "TCL", "sold_date" : "2019-07-02" }
    { "index": {}}
    { "price" : 1200, "color" : "绿色", "brand" : "TCL", "sold_date" : "2019-08-19" }
    { "index": {}}
    { "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2019-11-05" }
    { "index": {}}
    { "price" : 8000, "color" : "红色", "brand" : "三星", "sold_date" : "2020-01-01" }
    { "index": {}}
    { "price" : 2500, "color" : "蓝色", "brand" : "小米", "sold_date" : "2020-02-12" }

(2) 查看Mapping

GET /sales/_mapping

结果:

{
      "sales" : {
        "mappings" : {
          "properties" : {
            "brand" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "color" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "price" : {
              "type" : "long"
            },
            "sold_date" : {
              "type" : "date"
            }
          }
        }
      }
    }

text类型的字段不能用于聚合分析, 有三种解决方法:

*  使用text的多字段类型keyword进行搜索.
 *  创建Mapping时将字段类型设置为keyword.
 *  创建Mapping时将fielddata参数设置为true. (这种办法不可取, 虽然能进行聚合了, 但结果不准确, 因为聚合的字段不能被分词.)

> 详细讲解: https://blog.csdn.net/litianxiang\_kaola/article/details/103344432

**语法**

GET(POST) /index_name/_search
    {
      size: 0,
      "aggs": {
        "NAME": {
          "AGG_TYPE": {
    		 "field": "field_name"
    	   }
        }
      }
    }

NAME为聚合操作的名称, 可以取一个有参考意义的名称. AGG\_TYPE为分组或统计分析操作, 当进行分组操作时, 会自动生成一个doc\_count, 统计了组内数据的数量. 默认按照doc\_count降序排列.  
size=0的原因是不需要搜索结果, 如果需要搜索结果, 则去除size=0.

### Bucket Aggregation ###

Bucket Aggregation类似于SQL语句里的分组功能, 这里介绍几种常用的Bucket Aggregation.

*  terms
 *  range
 *  date\_range
 *  histogram
 *  date\_histogram
 *  filter

**terms**

按某个字段进行分组, 比如按品牌进行分组.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "term_aggs": {
          "terms": {
            "field": "brand.keyword"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"term_aggs" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**range**

按字段的值范围进行分组, 比如按电视价格范围分组. range只能作用于数值类型.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "price_range_aggs": {
          "range": {
            "field": "price",
            "ranges": [
              {
                "to": 1000
              },
              {
                "from": 1000,
                "to": 3000
              },
              {
                "from": 3000
              }
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"price_range_aggs" : {
    	  "buckets" : [
    	    {
    	      "key" : "*-1000.0",
    	      "to" : 1000.0,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key" : "1000.0-3000.0",
    	      "from" : 1000.0,
    	      "to" : 3000.0,
    	      "doc_count" : 6
    	    },
    	    {
    	      "key" : "3000.0-*",
    	      "from" : 3000.0,
    	      "doc_count" : 2
    	    }
    	  ]
    	}
    }

**date\_range**

按字段的值范围进行分组, 比如按sold\_date范围分组. date\_range只能作用于date类型.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "date_range_aggs": {
          "date_range": {
            "field": "sold_date",
            "ranges": [
              {
                "key": "start", 
                "to": "2019-10-31"
              },
              {
                "key": "middle", 
                "from": "2019-10-31",
                "to": "2020-01-01"
              },
              {
                "key": "end", 
                "to": "now"
              }
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"date_range_aggs" : {
    	  "buckets" : [
    	    {
    	      "key" : "start",
    	      "to" : 1.57248E12,
    	      "to_as_string" : "2019-10-31T00:00:00.000Z",
    	      "doc_count" : 4
    	    },
    	    {
    	      "key" : "end",
    	      "to" : 1.57803479079E12,
    	      "to_as_string" : "2020-01-03T06:59:50.790Z",
    	      "doc_count" : 7
    	    },
    	    {
    	      "key" : "middle",
    	      "from" : 1.57248E12,
    	      "from_as_string" : "2019-10-31T00:00:00.000Z",
    	      "to" : 1.5778368E12,
    	      "to_as_string" : "2020-01-01T00:00:00.000Z",
    	      "doc_count" : 2
    	    }
    	  ]
    	}
    }

为返回结果里的key命名, 不命名就同上面的range实例一样. 同样range也可以为key命名.

**histogram**

构建一个直方图, 如按照价格区间分组. histogram只能作用于数值类型.

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "histogram_aggs": {
          "histogram": {
            "field": "price",
            "interval": 2000
          }
        }
      }
    }

结果:

"aggregations" : {
    	"histogram_aggs" : {
    	  "buckets" : [
    	    {
    	      "key" : 0.0,
    	      "doc_count" : 3
    	    },
    	    {
    	      "key" : 2000.0,
    	      "doc_count" : 4
    	    },
    	    {
    	      "key" : 4000.0,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key" : 6000.0,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key" : 8000.0,
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**date\_histogram**

构建一个直方图, 如按照sold\_date区间分组. date\_histogram只能作用于date类型.

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "date_histogram_aggs": {
          "date_histogram": {
            "field": "sold_date",
            "calendar_interval": "month",
            "format": "yyyy-MM-dd",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": "2019-05-01",
              "max": "2020-02-01"
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"date_histogram_aggs" : {
    	  "buckets" : [
    	    {
    	      "key_as_string" : "2019-05-01",
    	      "key" : 1556668800000,
    	      "doc_count" : 1
    	    },
    	    {
    	      "key_as_string" : "2019-06-01",
    	      "key" : 1559347200000,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key_as_string" : "2019-07-01",
    	      "key" : 1561939200000,
    	      "doc_count" : 1
    	    },
    	    {
    	      "key_as_string" : "2019-08-01",
    	      "key" : 1564617600000,
    	      "doc_count" : 1
    	    },
    	    {
    	      "key_as_string" : "2019-09-01",
    	      "key" : 1567296000000,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key_as_string" : "2019-10-01",
    	      "key" : 1569888000000,
    	      "doc_count" : 1
    	    },
    	    {
    	      "key_as_string" : "2019-11-01",
    	      "key" : 1572566400000,
    	      "doc_count" : 2
    	    },
    	    {
    	      "key_as_string" : "2019-12-01",
    	      "key" : 1575158400000,
    	      "doc_count" : 0
    	    },
    	    {
    	      "key_as_string" : "2020-01-01",
    	      "key" : 1577836800000,
    	      "doc_count" : 1
    	    },
    	    {
    	      "key_as_string" : "2020-02-01",
    	      "key" : 1580515200000,
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**filter**

过滤分组, 将满足条件的数据分为一组. 比如分析最近6个月电视销售的平均价格.

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "filter_aggs": {
          "filter": {
            "range": {
              "sold_date": {
                "gte": "now-6M"
              }
            }
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"filter_aggs" : {
    	  "meta" : { },
    	  "doc_count" : 6,
    	  "avg_price" : {
    	    "value" : 2783.3333333333335
    	  }
    	}
    }

### Metric Aggregation ###

Metric Aggregation类似于SQL语句分组后的统计功能, 这里介绍几种常用的Metric Aggregation.

*  count
 *  min
 *  max
 *  sum
 *  avg
 *  cardinality
 *  stats
 *  top\_hits
 *  percentiles
 *  percentile\_ranks

**count/min/max/sum/avg**

从上文的Bucket Aggregation实例中我们知道, 当进行Bucket Aggregation时, 默认会生成一个doc\_count, 这个就是Count Aggregation.

min/max/sum/avg实例:

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "min_price": {
          "min": {
            "field": "price"
          }
        },
        "max_price": {
          "max": {
            "field": "price"
          }
        },
        "total_sales": {
          "sum": {
            "field": "price"
          }
        },
        "avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"max_price" : {
    	  "value" : 8000.0
    	},
    	"min_price" : {
    	  "value" : 1000.0
    	},
    	"avg_price" : {
    	  "value" : 2650.0
    	},
    	"total_sales" : {
    	  "value" : 21200.0
    	}
    }

**cardinality**

类似于与SQL里的distinct Count. 比如统计售出的电视品牌数.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "cardinality_aggs": {
          "cardinality": {
            "field": "brand.keyword"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"cardinality_aggs" : {
    	  "value" : 4
    	}
    }

**stats**

统计count, min, max, sum, avg.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "stat_price_aggs": {
          "stats": {
            "field": "price"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"stat_price_aggs" : {
    	  "count" : 8,
    	  "min" : 1000.0,
    	  "max" : 8000.0,
    	  "avg" : 2650.0,
    	  "sum" : 21200.0
    	}
    }

**top\_hits**

返回排在前面的结果, 与sort联用. 比如统计前三销售价格.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "top_hits_aggs": {
          "top_hits": {
            "size": 3,
            "sort": [
              {
                "price": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"top_hits_aggs" : {
    	  "hits" : {
    	    "total" : {
    	      "value" : 8,
    	      "relation" : "eq"
    	    },
    	    "max_score" : null,
    	    "hits" : [
    	      {
    	        "_index" : "sales",
    	        "_type" : "_doc",
    	        "_id" : "8hFjaW8BA-9v09_GWmPr",
    	        "_score" : null,
    	        "_source" : {
    	          "price" : 8000,
    	          "color" : "红色",
    	          "brand" : "三星",
    	          "sold_date" : "2020-01-01"
    	        },
    	        "sort" : [
    	          8000
    	        ]
    	      },
    	      {
    	        "_index" : "sales",
    	        "_type" : "_doc",
    	        "_id" : "7hFjaW8BA-9v09_GWmPr",
    	        "_score" : null,
    	        "_source" : {
    	          "price" : 3000,
    	          "color" : "绿色",
    	          "brand" : "小米",
    	          "sold_date" : "2019-05-18"
    	        },
    	        "sort" : [
    	          3000
    	        ]
    	      },
    	      {
    	        "_index" : "sales",
    	        "_type" : "_doc",
    	        "_id" : "8xFjaW8BA-9v09_GWmPr",
    	        "_score" : null,
    	        "_source" : {
    	          "price" : 2500,
    	          "color" : "蓝色",
    	          "brand" : "小米",
    	          "sold_date" : "2020-02-12"
    	        },
    	        "sort" : [
    	          2500
    	        ]
    	      }
    	    ]
    	  }
    	}
    }

**percentiles**

percentiles表示观察值在某个百分比的最大值. 比如统计50%, 90% 和 99%的电视的最大价格.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "percentile_aggs": {
          "percentiles": {
            "field": "price",
            "percents": [
              50,
              90,
              99
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"percentile_aggs" : {
    	  "values" : {
    	    "50.0" : 2000.0,
    	    "90.0" : 6500.000000000001,
    	    "99.0" : 8000.0
    	  }
    	}
    }

**percentile\_ranks**

percentile\_ranks表示观察值低于一定值的百分比. 比如统计价格在2000以内, 5000以内, 10000以内的电视所占比例.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "percentile_ranks_aggs": {
          "percentile_ranks": {
            "field": "price",
            "values": [
              2000,
              5000,
              10000
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"percentile_ranks_aggs" : {
    	  "values" : {
    	    "2000.0" : 50.0,
    	    "5000.0" : 85.22727272727273,
    	    "10000.0" : 100.0
    	  }
    	}
    }

### Bucket + Metric Aggregation ###

Bucket 聚合分析允许通过添加子聚合分析来进⼀步分析, 子聚合分析可以是Bucket, 也可以是Metric.

(1) 一层嵌套, 如按品牌分组, 统计价格信息

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "stats_price": {
              "stats": {
                "field": "price"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "stats_price" : {
    	        "count" : 3,
    	        "min" : 1000.0,
    	        "max" : 2000.0,
    	        "avg" : 1666.6666666666667,
    	        "sum" : 5000.0
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "stats_price" : {
    	        "count" : 2,
    	        "min" : 1200.0,
    	        "max" : 1500.0,
    	        "avg" : 1350.0,
    	        "sum" : 2700.0
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "stats_price" : {
    	        "count" : 2,
    	        "min" : 2500.0,
    	        "max" : 3000.0,
    	        "avg" : 2750.0,
    	        "sum" : 5500.0
    	      }
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "stats_price" : {
    	        "count" : 1,
    	        "min" : 8000.0,
    	        "max" : 8000.0,
    	        "avg" : 8000.0,
    	        "sum" : 8000.0
    	      }
    	    }
    	  ]
    	}
    }

(2) 多层嵌套, 如先按品牌分组, 然后按颜色分组, 统计价格信息

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "group_by_color": {
              "terms": {
                "field": "color.keyword"
              },
              "aggs": {
                "stats_price": {
                  "stats": {
                    "field": "price"
                  }
                }
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "group_by_color" : {
    	        "doc_count_error_upper_bound" : 0,
    	        "sum_other_doc_count" : 0,
    	        "buckets" : [
    	          {
    	            "key" : "红色",
    	            "doc_count" : 3,
    	            "stats_price" : {
    	              "count" : 3,
    	              "min" : 1000.0,
    	              "max" : 2000.0,
    	              "avg" : 1666.6666666666667,
    	              "sum" : 5000.0
    	            }
    	          }
    	        ]
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "group_by_color" : {
    	        "doc_count_error_upper_bound" : 0,
    	        "sum_other_doc_count" : 0,
    	        "buckets" : [
    	          {
    	            "key" : "绿色",
    	            "doc_count" : 1,
    	            "stats_price" : {
    	              "count" : 1,
    	              "min" : 1200.0,
    	              "max" : 1200.0,
    	              "avg" : 1200.0,
    	              "sum" : 1200.0
    	            }
    	          },
    	          {
    	            "key" : "蓝色",
    	            "doc_count" : 1,
    	            "stats_price" : {
    	              "count" : 1,
    	              "min" : 1500.0,
    	              "max" : 1500.0,
    	              "avg" : 1500.0,
    	              "sum" : 1500.0
    	            }
    	          }
    	        ]
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "group_by_color" : {
    	        "doc_count_error_upper_bound" : 0,
    	        "sum_other_doc_count" : 0,
    	        "buckets" : [
    	          {
    	            "key" : "绿色",
    	            "doc_count" : 1,
    	            "stats_price" : {
    	              "count" : 1,
    	              "min" : 3000.0,
    	              "max" : 3000.0,
    	              "avg" : 3000.0,
    	              "sum" : 3000.0
    	            }
    	          },
    	          {
    	            "key" : "蓝色",
    	            "doc_count" : 1,
    	            "stats_price" : {
    	              "count" : 1,
    	              "min" : 2500.0,
    	              "max" : 2500.0,
    	              "avg" : 2500.0,
    	              "sum" : 2500.0
    	            }
    	          }
    	        ]
    	      }
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "group_by_color" : {
    	        "doc_count_error_upper_bound" : 0,
    	        "sum_other_doc_count" : 0,
    	        "buckets" : [
    	          {
    	            "key" : "红色",
    	            "doc_count" : 1,
    	            "stats_price" : {
    	              "count" : 1,
    	              "min" : 8000.0,
    	              "max" : 8000.0,
    	              "avg" : 8000.0,
    	              "sum" : 8000.0
    	            }
    	          }
    	        ]
    	      }
    	    }
    	  ]
    	}
    }

### Pipeline Aggregation ###

对聚合分析的结果再次进行聚合分析.

Pipeline Aggregation的分析结果会输出到原结果中, 根据不同的位置可以将其分为两类:

*  Sibling  
    结果和现有分析结果同级
 *  Parent  
    结果内嵌到现有的聚合分析结果之中

**Sibling Pipeline**

常用的Sibling Pipeline有:

*  min\_bucket
 *  max\_bucket
 *  sum\_bucket
 *  avg\_bucket
 *  stats\_bucket
 *  percentiles\_bucket

(1) min\_bucket

平均价格最低的品牌

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        },
        "min_price_by_brand": {
          "min_bucket": {
            "buckets_path": "group_by_brand>avg_price"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "avg_price" : {
    	        "value" : 1666.6666666666667
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 1350.0
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 2750.0
    	      }
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "avg_price" : {
    	        "value" : 8000.0
    	      }
    	    }
    	  ]
    	},
    	"min_price_by_brand" : {
    	  "value" : 1350.0,
    	  "keys" : [
    	    "TCL"
    	  ]
    	}
    }

(2) stats\_bucket

所有品牌电视的平均价格的统计分析

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        },
        "stats_price_by_brand": {
          "stats_bucket": {
            "buckets_path": "group_by_brand>avg_price"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "avg_price" : {
    	        "value" : 1666.6666666666667
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 1350.0
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 2750.0
    	      }
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "avg_price" : {
    	        "value" : 8000.0
    	      }
    	    }
    	  ]
    	},
    	"stats_price_by_brand" : {
    	  "count" : 4,
    	  "min" : 1350.0,
    	  "max" : 8000.0,
    	  "avg" : 3441.666666666667,
    	  "sum" : 13766.666666666668
    	}
    }

(3) percentiles\_bucket

所有品牌电视的平均价格的百分比统计

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        },
        "percentiles_price_by_brand": {
          "percentiles_bucket": {
            "percents": [50,90,99], 
            "buckets_path": "group_by_brand>avg_price"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "avg_price" : {
    	        "value" : 1666.6666666666667
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 1350.0
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 2750.0
    	      }
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "avg_price" : {
    	        "value" : 8000.0
    	      }
    	    }
    	  ]
    	},
    	"percentiles_price_by_brand" : {
    	  "values" : {
    	    "50.0" : 2750.0,
    	    "90.0" : 8000.0,
    	    "99.0" : 8000.0
    	  }
    	}
    }

**Parent Pipeline**

常用的Parent Pipeline有:

*  derivative(求导)
 *  cumulative\_sum(累计求和)
 *  moving\_fn(滑动窗口)

Parent Pipeline只适用于 histogram 或 date\_histogram 分组.

(1) derivative

对按照sold\_date分组的平均价格求导

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "sold_date_aggs": {
          "date_histogram": {
            "field": "sold_date",
            "calendar_interval": "month",
            "format": "yyyy-MM-dd"
          },
          "aggs": {
            "avg_price_aggs": {
              "avg": {
                "field": "price"
              }
            },
            "derivative_avg_price_aggs": {
              "derivative": {
                "buckets_path": "avg_price_aggs"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"sold_date_aggs" : {
    	  "buckets" : [
    	    {
    	      "key_as_string" : "2019-05-01",
    	      "key" : 1556668800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 3000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-06-01",
    	      "key" : 1559347200000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-07-01",
    	      "key" : 1561939200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1500.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-08-01",
    	      "key" : 1564617600000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1200.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : -300.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-09-01",
    	      "key" : 1567296000000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-10-01",
    	      "key" : 1569888000000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1000.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-11-01",
    	      "key" : 1572566400000,
    	      "doc_count" : 2,
    	      "avg_price_aggs" : {
    	        "value" : 2000.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : 1000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-12-01",
    	      "key" : 1575158400000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-01-01",
    	      "key" : 1577836800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 8000.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-02-01",
    	      "key" : 1580515200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 2500.0
    	      },
    	      "derivative_avg_price_aggs" : {
    	        "value" : -5500.0
    	      }
    	    }
    	  ]
    	}
    }

(2) cumulative\_sum

对按照sold\_date分组的平均价格累计求和.

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "sold_date_aggs": {
          "date_histogram": {
            "field": "sold_date",
            "calendar_interval": "month",
            "format": "yyyy-MM-dd"
          },
          "aggs": {
            "avg_price_aggs": {
              "avg": {
                "field": "price"
              }
            },
            "cumulative_sum_avg_price_aggs": {
              "cumulative_sum": {
                "buckets_path": "avg_price_aggs"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"sold_date_aggs" : {
    	  "buckets" : [
    	    {
    	      "key_as_string" : "2019-05-01",
    	      "key" : 1556668800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 3000.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 3000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-06-01",
    	      "key" : 1559347200000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 3000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-07-01",
    	      "key" : 1561939200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1500.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 4500.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-08-01",
    	      "key" : 1564617600000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1200.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 5700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-09-01",
    	      "key" : 1567296000000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 5700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-10-01",
    	      "key" : 1569888000000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1000.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 6700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-11-01",
    	      "key" : 1572566400000,
    	      "doc_count" : 2,
    	      "avg_price_aggs" : {
    	        "value" : 2000.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 8700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-12-01",
    	      "key" : 1575158400000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 8700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-01-01",
    	      "key" : 1577836800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 8000.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 16700.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-02-01",
    	      "key" : 1580515200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 2500.0
    	      },
    	      "cumulative_sum_avg_price_aggs" : {
    	        "value" : 19200.0
    	      }
    	    }
    	  ]
    	}
    }

(3) moving\_fn

对按照sold\_date分组的平均价格, 按时间窗口求最小平均价格

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "sold_date_aggs": {
          "date_histogram": {
            "field": "sold_date",
            "calendar_interval": "month",
            "format": "yyyy-MM-dd"
          },
          "aggs": {
            "avg_price_aggs": {
              "avg": {
                "field": "price"
              }
            },
            "moving_fn_avg_price_aggs": {
              "moving_fn": {
                "buckets_path": "avg_price_aggs",
                "window": 10,
                "script": "MovingFunctions.min(values)"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"sold_date_aggs" : {
    	  "buckets" : [
    	    {
    	      "key_as_string" : "2019-05-01",
    	      "key" : 1556668800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 3000.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-06-01",
    	      "key" : 1559347200000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-07-01",
    	      "key" : 1561939200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1500.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 3000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-08-01",
    	      "key" : 1564617600000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1200.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 1500.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-09-01",
    	      "key" : 1567296000000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-10-01",
    	      "key" : 1569888000000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 1000.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 1200.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-11-01",
    	      "key" : 1572566400000,
    	      "doc_count" : 2,
    	      "avg_price_aggs" : {
    	        "value" : 2000.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 1000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2019-12-01",
    	      "key" : 1575158400000,
    	      "doc_count" : 0,
    	      "avg_price_aggs" : {
    	        "value" : null
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-01-01",
    	      "key" : 1577836800000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 8000.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 1000.0
    	      }
    	    },
    	    {
    	      "key_as_string" : "2020-02-01",
    	      "key" : 1580515200000,
    	      "doc_count" : 1,
    	      "avg_price_aggs" : {
    	        "value" : 2500.0
    	      },
    	      "moving_fn_avg_price_aggs" : {
    	        "value" : 1000.0
    	      }
    	    }
    	  ]
    	}
    }

### Matrix Aggregation ###

支持对多个字段进行matrix\_stats统计分析(字段只能是数值类型), 并对每个字段得出以下统计信息:

*  count  
    计算中包含每个字段样本的数量.
 *  mean  
    每个字段的平均值.
 *  variance  
    方差.
 *  skewness  
    斜率.
 *  kurtosis  
    峰值.
 *  covariance  
    协方差.
 *  correlation  
    协方差矩阵缩放到-1到1的范围, 描述字段分布之间的关系.

实例:

GET /sales/_search
    {
      "size": 0,
      "aggs": {
        "statistics": {
          "matrix_stats": {
            "fields": ["price"]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"statistics" : {
    	  "doc_count" : 8,
    	  "fields" : [
    	    {
    	      "name" : "price",
    	      "count" : 8,
    	      "mean" : 2650.0,
    	      "variance" : 5108571.42857143,
    	      "skewness" : 1.8987996927997286,
    	      "kurtosis" : 5.212487613170579,
    	      "covariance" : {
    	        "price" : 5108571.42857143
    	      },
    	      "correlation" : {
    	        "price" : 1.0
    	      }
    	    }
    	  ]
    	}
    }

### 聚合的作用范围及排序 ###

Elasticsearch聚合分析的默认作用范围是query的查询结果集, 同时Elasticsearch还支持以下方式改变聚合的作用范围.

*  filter
 *  post\_filter
 *  global

**query**

先使用query进行过滤, 然后再进行聚合操作.

GET /sales/_search
    {
      "size": 0, 
      "query": {
        "range": {
          "price": {
            "gte": 2000
          }
        }
      },
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**filter**

聚合内的filter只对当前的子聚合语句生效.

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "price_filter": {
          "filter": {
            "range": {
              "price": {
                "gte": 2000
              }
            }
          },
          "aggs": {
            "group_by_brand": {
              "terms": {
                "field": "brand.keyword"
              }
            }
          }
        },
        "group_by_all_brand": {
          "terms": {
            "field": "brand.keyword"
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_all_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    }
    	  ]
    	},
    	"price_filter" : {
    	  "doc_count" : 5,
    	  "group_by_brand" : {
    	    "doc_count_error_upper_bound" : 0,
    	    "sum_other_doc_count" : 0,
    	    "buckets" : [
    	      {
    	        "key" : "小米",
    	        "doc_count" : 2
    	      },
    	      {
    	        "key" : "长虹",
    	        "doc_count" : 2
    	      },
    	      {
    	        "key" : "三星",
    	        "doc_count" : 1
    	      }
    	    ]
    	  }
    	}
    }

可以看到filter只对当前的子聚合语句生效, group\_by\_all\_brand这个聚合就没有受filter影响.

**post\_filter**

post\_filter是对聚合分析后的文档再次过滤, size不要设置为0.

GET /sales/_search
    {
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          }
        }
      },
      "post_filter": {
        "term": {
          "brand.keyword": "小米"
        }
      }
    }

结果:

"hits" : {
    	"total" : {
    	  "value" : 2,
    	  "relation" : "eq"
    	},
    	"max_score" : 1.0,
    	"hits" : [
    	  {
    	    "_index" : "sales",
    	    "_type" : "_doc",
    	    "_id" : "AJdFgG8B-Wwl0Pg0wUwx",
    	    "_score" : 1.0,
    	    "_source" : {
    	      "price" : 3000,
    	      "color" : "绿色",
    	      "brand" : "小米",
    	      "sold_date" : "2019-05-18"
    	    }
    	  },
    	  {
    	    "_index" : "sales",
    	    "_type" : "_doc",
    	    "_id" : "BZdFgG8B-Wwl0Pg0wUwx",
    	    "_score" : 1.0,
    	    "_source" : {
    	      "price" : 2500,
    	      "color" : "蓝色",
    	      "brand" : "小米",
    	      "sold_date" : "2020-02-12"
    	    }
    	  }
    	]
    },
    "aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**global**

global无视query过滤, 对全部文档进行统计.

GET /sales/_search
    {
      "size": 0, 
      "query": {
        "range": {
          "price": {
            "gte": 2000
          }
        }
      },
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          }
        },
        "all_price_aggs": {
          "global": {},
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"all_price_aggs" : {
    	  "doc_count" : 8,
    	  "avg_price" : {
    	    "value" : 2650.0
    	  }
    	},
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    }
    	  ]
    	}
    }

**排序**

通过order字段来指定排序, 默认情况下根据doc\_count降序排列.

(1) 根据doc\_count升序, key降序来排序

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword",
            "order": [
              {
                "_count": "asc"
              },
              {
                "_key": "desc"
              }
            ]
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2
    	    },
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3
    	    }
    	  ]
    	}
    }

(2) 基于子聚合的值排序

GET /sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword",
            "order": {
              "avg_price": "desc"
            }
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }

结果:

"aggregations" : {
    	"group_by_brand" : {
    	  "doc_count_error_upper_bound" : 0,
    	  "sum_other_doc_count" : 0,
    	  "buckets" : [
    	    {
    	      "key" : "三星",
    	      "doc_count" : 1,
    	      "avg_price" : {
    	        "value" : 8000.0
    	      }
    	    },
    	    {
    	      "key" : "小米",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 2750.0
    	      }
    	    },
    	    {
    	      "key" : "长虹",
    	      "doc_count" : 3,
    	      "avg_price" : {
    	        "value" : 1666.6666666666667
    	      }
    	    },
    	    {
    	      "key" : "TCL",
    	      "doc_count" : 2,
    	      "avg_price" : {
    	        "value" : 1350.0
    	      }
    	    }
    	  ]
    	}
    }

### Java Client ###

下面介绍Elasticsearch Java Client 的使用, 我们来将上文的实例转化为 Java Client.

[Elasticsearch聚合分析Java Client][Elasticsearch_Java Client]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2020010722325191.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdGlhbnhpYW5nX2thb2xh,size_16,color_FFFFFF,t_70
[Elasticsearch_Java Client]: https://blog.csdn.net/litianxiang_kaola/article/details/103919594