管道聚合處理來自其他聚合而不是文檔集的輸出,將信息添加到輸出樹中。
注:關於腳本聚合目前在本文中暫時不會涉及。
主要有如下兩種管道聚合方式:
- parent
- sibling
下面一一介紹ES定義的管道聚合。
1、Avg Bucket Aggregation
同級管道聚合,它計算同級聚合中指定度量的平均值。同級聚合必須是多桶聚合,針對的是度量聚合(metric Aggregation)。
示例如下:
{
"avg_bucket": {
"buckets_path": "the_sum" // @1
}
}
- buckets_path:指定聚合的名稱,支持多級嵌套聚合。
其他參數: - gap_policy
當管道聚合遇到不存在的值,有點類似於term等聚合的(missing)時所採取的策略,可選擇值爲:skip、insert_zeros。 - skip:此選項將丟失的數據視爲bucket不存在。它將跳過桶並使用下一個可用值繼續計算。
- insert_zeros:默認使用0代替。
- format
用於格式化聚合桶的輸出(key)。
示例如下:
POST /_search
{
"size": 0,
"aggs": {
"sales_per_month": { // @1
"date_histogram": {
"field": "date",
"interval": "month"
},
"aggs": { // @2
"sales": {
"sum": {
"field": "price"
}
}
}
},
"avg_monthly_sales": { // @3
"avg_bucket": {
"buckets_path": "sales_per_month>sales"
}
}
}
}
代碼@1:首先定義第一級聚合(按月)直方圖聚合。
代碼@2:定義第二級聚合,在按月聚合的基礎上,對每個月的文檔求sum。
代碼@3:對上面的聚合求平均值。
其返回結果如下:
{
... // 省略
"aggregations": {
"sales_per_month": {
"buckets": [
{
"key_as_string": "2015/01/01 00:00:00",
"key": 1420070400000,
"doc_count": 3,
"sales": {
"value": 550.0
}
},
{
"key_as_string": "2015/02/01 00:00:00",
"key": 1422748800000,
"doc_count": 2,
"sales": {
"value": 60.0
}
}
]
},
"avg_monthly_sales": { // 這是對二級聚合的結果再進行一次求平均值聚合。
"value": 328.33333333333333
}
}
}
對應的JAVA示例如下:
public static void test_pipeline_avg_buncket_aggregation() {
RestHighLevelClient client = EsClient.getClient();
try {
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("aggregations_index02");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
AggregationBuilder aggregationBuild = AggregationBuilders.terms("seller_agg")
.field("sellerId")
.subAggregation(AggregationBuilders.sum("seller_num_agg")
.field("num")
)
;
sourceBuilder.aggregation(aggregationBuild);
// 添加 avg bucket pipeline
sourceBuilder.aggregation(new AvgBucketPipelineAggregationBuilder("seller_num_agg_av", "seller_agg>seller_num_agg"));
sourceBuilder.size(0);
searchRequest.source(sourceBuilder);
SearchResponse result = client.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(result);
} catch (Throwable e) {
e.printStackTrace();
} finally {
EsClient.close(client);
}
}
2、Percentiles Bucket Aggregation
同級管道聚合,百分位管道聚合。其JAVA示例如下:
public static void test_Percentiles_buncket_aggregation() {
RestHighLevelClient client = EsClient.getClient();
try {
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("aggregations_index02");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
AggregationBuilder aggregationBuild = AggregationBuilders.terms("seller_agg")
.field("sellerId")
.subAggregation(AggregationBuilders.sum("seller_num_agg")
.field("num")
)
;
sourceBuilder.aggregation(aggregationBuild);
// 添加 avg bucket pipeline
sourceBuilder.aggregation(new PercentilesBucketPipelineAggregationBuilder("seller_num_agg_av", "seller_agg>seller_num_agg"));
sourceBuilder.size(0);
searchRequest.source(sourceBuilder);
SearchResponse result = client.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(result);
} catch (Throwable e) {
e.printStackTrace();
} finally {
EsClient.close(client);
}
}
其返回值如下:
{
... // 省略其他屬性
"aggregations":{
"lterms#seller_agg":{
"doc_count_error_upper_bound":0,
"sum_other_doc_count":12,
"buckets":[
{
"key":45,
"doc_count":567,
"sum#seller_num_agg":{
"value":911
}
},
{
"key":31,
"doc_count":324,
"sum#seller_num_agg":{
"value":353
}
} // 省略其他桶的顯示
]
},
"percentiles_bucket#seller_num_agg_av":{
"values":{
"1.0":5,
"5.0":5,
"25.0":10,
"50.0":20,
"75.0":290,
"95.0":911,
"99.0":911
}
}
}
}
3、Cumulative Sum Aggregation
累積管道聚合,就是就是依次將每個管道的sum聚合進行累加。
其語法(restfull)如下:
{
"cumulative_sum": {
"buckets_path": "the_sum"
}
}
支持的參數說明:
- buckets_path
桶聚合名稱,作爲管道聚合的輸入信息。 - format
格式化key。
使用示例如下:
POST /sales/_search
{
"size": 0,
"aggs" : {
"sales_per_month" : {
"date_histogram" : {
"field" : "date",
"interval" : "month"
},
"aggs": {
"sales": {
"sum": {
"field": "price"
}
},
"cumulative_sales": {
"cumulative_sum": {
"buckets_path": "sales"
}
}
}
}
}
}
其返回結果如下:
{
"took": 11,
"timed_out": false,
"_shards": ...,
"hits": ...,
"aggregations": {
"sales_per_month": {
"buckets": [
{
"key_as_string": "2015/01/01 00:00:00",
"key": 1420070400000,
"doc_count": 3,
"sales": {
"value": 550.0
},
"cumulative_sales": {
"value": 550.0
}
},
{
"key_as_string": "2015/02/01 00:00:00",
"key": 1422748800000,
"doc_count": 2,
"sales": {
"value": 60.0
},
"cumulative_sales": {
"value": 610.0
}
},
{
"key_as_string": "2015/03/01 00:00:00",
"key": 1425168000000,
"doc_count": 2,
"sales": {
"value": 375.0
},
"cumulative_sales": {
"value": 985.0
}
}
]
}
}
}
從結果可知,cumulative_sales的值等於上一個cumulative_sales + 當前桶的sum聚合。
對應的JAVA示例如下:
{
"aggregations":{
"date_histogram#createTime_histogram":{
"buckets":{
"2015-12-01 00:00:00":{
"key_as_string":"2015-12-01 00:00:00",
"key":1448928000000,
"doc_count":6,
"sum#seller_num_agg":{
"value":16
},
"simple_value#Cumulative_Seller_num_agg":{
"value":16
}
},
"2016-01-01 00:00:00":{
"key_as_string":"2016-03-01 00:00:00",
"key":1456790400000,
"doc_count":10,
"sum#seller_num_agg":{
"value":11
},
"simple_value#Cumulative_Seller_num_agg":{
"value":31
}
}
// ... 忽略
}
}
}
}
4、Bucket Sort Aggregation
一種父管道聚合,它對其父多桶聚合的桶進行排序。並可以指定多個排序字段。每個bucket可以根據它的_key、_count或子聚合進行排序。此外,可以設置from和size的參數,以便截斷結果桶。
使用語法如下:
{
"bucket_sort": {
"sort": [
{"sort_field_1": {"order": "asc"}},
{"sort_field_2": {"order": "desc"}},
"sort_field_3"
],
"from": 1,
"size": 3
}
}
支持的參數說明如下:
- sort
定義排序結構。 - from
用與對父聚合的桶進行截取,該值之前的所有桶將忽略,也就是不參與排序,默認爲0。 - size
返回的桶數。默認爲父聚合的所有桶。 - gap_policy
當管道聚合遇到不存在的值,有點類似於term等聚合的(missing)時所採取的策略,可選擇值爲:skip、insert_zeros。 - skip:此選項將丟失的數據視爲bucket不存在。它將跳過桶並使用下一個可用值繼續計算。
- insert_zeros:默認使用0代替。
官方示例如下:
POST /sales/_search
{
"size": 0,
"aggs" : {
"sales_per_month" : {
"date_histogram" : {
"field" : "date",
"interval" : "month"
},
"aggs": {
"total_sales": {
"sum": {
"field": "price"
}
},
"sales_bucket_sort": {
"bucket_sort": {
"sort": [
{"total_sales": {"order": "desc"}}
],
"size": 3
}
}
}
}
}
}
對應的JAVA示例如下:
public static void test_bucket_sort_Aggregation() {
RestHighLevelClient client = EsClient.getClient();
try {
//構建日期直方圖聚合 時間間隔,示例中按月統計
DateHistogramInterval interval = new DateHistogramInterval("1M");
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("aggregations_index02");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
AggregationBuilder aggregationBuild = AggregationBuilders.dateHistogram("createTime_histogram")
.field("createTime")
.dateHistogramInterval(interval)
.keyed(true)
.subAggregation(AggregationBuilders.sum("seller_num_agg")
.field("num")
)
.subAggregation(new BucketSortPipelineAggregationBuilder("seller_num_agg_sort", Arrays.asList(
new FieldSortBuilder("seller_num_agg").order(SortOrder.ASC)))
.from(0)
.size(3))
// BucketSortPipelineAggregationBuilder(String name, List<FieldSortBuilder> sorts)
.subAggregation(new CumulativeSumPipelineAggregationBuilder("Cumulative_Seller_num_agg", "seller_num_agg"))
// .format("yyyy-MM-dd") // 對key的格式化
;
sourceBuilder.aggregation(aggregationBuild);
sourceBuilder.size(0);
sourceBuilder.query(
QueryBuilders.termQuery("sellerId", 24)
);
searchRequest.source(sourceBuilder);
SearchResponse result = client.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(result);
} catch (Throwable e) {
e.printStackTrace();
} finally {
EsClient.close(client);
}
}
返回值:
{
"aggregations":{
"date_histogram#createTime_histogram":{
"buckets":{
"2016-04-01 00:00:00":{
"key_as_string":"2016-04-01 00:00:00",
"key":1459468800000,
"doc_count":2,
"sum#seller_num_agg":{
"value":2
},
"simple_value#Cumulative_Seller_num_agg":{
"value":2
}
},
"2017-05-01 00:00:00":{
"key_as_string":"2017-05-01 00:00:00",
"key":1493596800000,
"doc_count":3,
"sum#seller_num_agg":{
"value":3
},
"simple_value#Cumulative_Seller_num_agg":{
"value":5
}
},
"2017-02-01 00:00:00":{
"key_as_string":"2017-02-01 00:00:00",
"key":1485907200000,
"doc_count":4,
"sum#seller_num_agg":{
"value":4
},
"simple_value#Cumulative_Seller_num_agg":{
"value":9
}
}
}
}
}
5、Max Bucket Aggregation
與 avg類似。
6、Min Bucket Aggregation
與 avg類似。
7、Sum Bucket Aggregation
與 avg類似。
8、Stats Bucket Aggregation
與 avg類似。
本節詳細介紹了ES Pipeline Aggregation 管道聚合的使用方法,重點介紹了Avg Bucket Aggregation、Percentiles Bucket Aggregation、Cumulative Sum Aggregation、Bucket Sort Aggregation、Max Bucket Aggregation、Min Bucket Aggregation、Sum Bucket Aggregation、Stats Bucket Aggregation。
原文發佈時間爲:2019-03-18
本文作者:丁威,《RocketMQ技術內幕》作者。
本文來自中間件興趣圈,瞭解相關信息可以關注中間件興趣圈。