Hive分析窗口函數(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

這幾個分析函數通常用於OLAP中，不能累加，而且需要根據不同維度上鑽和下鑽的指標統計，比如，分小時、天、月的UV數。

Hive版本爲 apache-hive-0.13.1

數據準備：

1. 2015-03,2015-03-10,cookie1
2. 2015-03,2015-03-10,cookie5
3. 2015-03,2015-03-12,cookie7
4. 2015-04,2015-04-12,cookie3
5. 2015-04,2015-04-13,cookie2
6. 2015-04,2015-04-13,cookie4
7. 2015-04,2015-04-16,cookie4
8. 2015-03,2015-03-10,cookie2
9. 2015-03,2015-03-10,cookie3
10. 2015-04,2015-04-12,cookie5
11. 2015-04,2015-04-13,cookie6
12. 2015-04,2015-04-15,cookie3
13. 2015-04,2015-04-15,cookie2
14. 2015-04,2015-04-16,cookie1
15.  
16. CREATE EXTERNAL TABLE lxw1234 (
17. month STRING,
18. day STRING,
19. cookieid STRING
20. ) ROW FORMAT DELIMITED
21. FIELDS TERMINATED BY ','
22. stored as textfile location '/tmp/lxw11/';
23.  
24.  
25. hive> select * from lxw1234;
26. OK
27. 2015-03 2015-03-10 cookie1
28. 2015-03 2015-03-10 cookie5
29. 2015-03 2015-03-12 cookie7
30. 2015-04 2015-04-12 cookie3
31. 2015-04 2015-04-13 cookie2
32. 2015-04 2015-04-13 cookie4
33. 2015-04 2015-04-16 cookie4
34. 2015-03 2015-03-10 cookie2
35. 2015-03 2015-03-10 cookie3
36. 2015-04 2015-04-12 cookie5
37. 2015-04 2015-04-13 cookie6
38. 2015-04 2015-04-15 cookie3
39. 2015-04 2015-04-15 cookie2
40. 2015-04 2015-04-16 cookie1

GROUPING SETS

在一個GROUP BY查詢中，根據不同的維度組合進行聚合，等價於將不同維度的GROUP BY結果集進行UNION ALL

1. SELECT
2. month,
3. day,
4. COUNT(DISTINCT cookieid) AS uv,
5. GROUPING__ID
6. FROM lxw1234
7. GROUP BY month,day
8. GROUPING SETS (month,day)
9. ORDER BY GROUPING__ID;
10.  
11. month day uv GROUPING__ID
12. ------------------------------------------------
13. 2015-03 NULL 5 1
14. 2015-04 NULL 6 1
15. NULL 2015-03-10 4 2
16. NULL 2015-03-12 1 2
17. NULL 2015-04-12 2 2
18. NULL 2015-04-13 3 2
19. NULL 2015-04-15 2 2
20. NULL 2015-04-16 2 2
21.  
22.  
23. 等價於
24. SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month
25. UNION ALL
26. SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY day

再如：

1. SELECT
2. month,
3. day,
4. COUNT(DISTINCT cookieid) AS uv,
5. GROUPING__ID
6. FROM lxw1234
7. GROUP BY month,day
8. GROUPING SETS (month,day,(month,day))
9. ORDER BY GROUPING__ID;
10.  
11. month day uv GROUPING__ID
12. ------------------------------------------------
13. 2015-03 NULL 5 1
14. 2015-04 NULL 6 1
15. NULL 2015-03-10 4 2
16. NULL 2015-03-12 1 2
17. NULL 2015-04-12 2 2
18. NULL 2015-04-13 3 2
19. NULL 2015-04-15 2 2
20. NULL 2015-04-16 2 2
21. 2015-03 2015-03-10 4 3
22. 2015-03 2015-03-12 1 3
23. 2015-04 2015-04-12 2 3
24. 2015-04 2015-04-13 3 3
25. 2015-04 2015-04-15 2 3
26. 2015-04 2015-04-16 2 3
27.  
28.  
29. 等價於
30. SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month
31. UNION ALL
32. SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY day
33. UNION ALL
34. SELECT month,day,COUNT(DISTINCT cookieid) AS uv,3 AS GROUPING__ID FROM lxw1234 GROUP BY month,day

其中的 GROUPING__ID，表示結果屬於哪一個分組集合。

 

CUBE

根據GROUP BY的維度的所有組合進行聚合。

1. SELECT
2. month,
3. day,
4. COUNT(DISTINCT cookieid) AS uv,
5. GROUPING__ID
6. FROM lxw1234
7. GROUP BY month,day
8. WITH CUBE
9. ORDER BY GROUPING__ID;
10.  
11.  
12. month day uv GROUPING__ID
13. --------------------------------------------
14. NULL NULL 7 0
15. 2015-03 NULL 5 1
16. 2015-04 NULL 6 1
17. NULL 2015-04-12 2 2
18. NULL 2015-04-13 3 2
19. NULL 2015-04-15 2 2
20. NULL 2015-04-16 2 2
21. NULL 2015-03-10 4 2
22. NULL 2015-03-12 1 2
23. 2015-03 2015-03-10 4 3
24. 2015-03 2015-03-12 1 3
25. 2015-04 2015-04-16 2 3
26. 2015-04 2015-04-12 2 3
27. 2015-04 2015-04-13 3 3
28. 2015-04 2015-04-15 2 3
29.  
30.  
31.  
32. 等價於
33. SELECT NULL,NULL,COUNT(DISTINCT cookieid) AS uv,0 AS GROUPING__ID FROM lxw1234
34. UNION ALL
35. SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month
36. UNION ALL
37. SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY day
38. UNION ALL
39. SELECT month,day,COUNT(DISTINCT cookieid) AS uv,3 AS GROUPING__ID FROM lxw1234 GROUP BY month,day
41. 等價於
42. select month,day,count(distinct cookieid) as uv,grouping__id
43. from lxw1234
44. group by month,day
45. grouping sets((),month,day,(month,day))
46. order by grouping__id;

ROLLUP

是CUBE的子集，以最左側的維度爲主，從該維度進行層級聚合。

1. 比如，以month維度進行層級聚合：
2. SELECT
3. month,
4. day,
5. COUNT(DISTINCT cookieid) AS uv,
6. GROUPING__ID
7. FROM lxw1234
8. GROUP BY month,day
9. WITH ROLLUP
10. ORDER BY GROUPING__ID;
11.  
12. month day uv GROUPING__ID
13. ---------------------------------------------------
14. NULL NULL 7 0
15. 2015-03 NULL 5 1
16. 2015-04 NULL 6 1
17. 2015-03 2015-03-10 4 3
18. 2015-03 2015-03-12 1 3
19. 2015-04 2015-04-12 2 3
20. 2015-04 2015-04-13 3 3
21. 2015-04 2015-04-15 2 3
22. 2015-04 2015-04-16 2 3
23.  
24. 可以實現這樣的上鑽過程：
25. 月天的UV->月的UV->總UV
26. 等價於
27. SELECT NULL,NULL,COUNT(DISTINCT cookieid) AS uv,0 AS GROUPING__ID FROM lxw1234
28. UNION ALL
29. SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month
30. UNION ALL
31. SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY month, day
33. 等價於
34. SELECT month,day,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID  
35. FROM lxw1234
36. GROUP BY month,day
37. grouping sets((),month,(month,day))
38. ORDER BY GROUPING__ID;

1. --把month和day調換順序，則以day維度進行層級聚合：
2.  
3. SELECT
4. day,
5. month,
6. COUNT(DISTINCT cookieid) AS uv,
7. GROUPING__ID
8. FROM lxw1234
9. GROUP BY day,month
10. WITH ROLLUP
11. ORDER BY GROUPING__ID;
12.  
13.  
14. day month uv GROUPING__ID
15. -------------------------------------------------------
16. NULL NULL 7 0
17. 2015-04-13 NULL 3 1
18. 2015-03-12 NULL 1 1
19. 2015-04-15 NULL 2 1
20. 2015-03-10 NULL 4 1
21. 2015-04-16 NULL 2 1
22. 2015-04-12 NULL 2 1
23. 2015-04-12 2015-04 2 3
24. 2015-03-10 2015-03 4 3
25. 2015-03-12 2015-03 1 3
26. 2015-04-13 2015-04 3 3
27. 2015-04-15 2015-04 2 3
28. 2015-04-16 2015-04 2 3
29.  
30. 可以實現這樣的上鑽過程：
31. 天月的UV->天的UV->總UV
32. （這裏，根據天和月進行聚合，和根據天聚合結果一樣，因爲有父子關係，如果是其他維度組合的話，就會不一樣）

這種函數，需要結合實際場景和數據去使用和研究，只看說明的話，很難理解。

官網的介紹： https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup