MaxCompute - ODPS重裝上陣第十二彈 - PIVOT/UNPIVOT

前言

MaxCompute（原ODPS）是阿里雲自主研發的具有業界領先水平的分佈式大數據處理平臺, 尤其在集團內部得到廣泛應用，支撐了多個BU的核心業務。 MaxCompute除了持續優化性能外，也致力於提升SQL語言的用戶體驗和表達能力，提高廣大ODPS開發者的生產力。

MaxCompute基於ODPS2.0新一代的SQL引擎，顯著提升了SQL語言編譯過程的易用性與語言的表達能力。

本文將向您介紹MaxCompute支持的新語法 - PIVOT/UNPIVOT，即通過PIVOT關鍵字基於聚合將一個或者多個指定值的行轉換爲列；通過UNPIVOT關鍵字可將一個或者多個列轉換爲行。常見的場景入下：

場景1

某個業務表，需要把表中的值當做新的列，並且根據每個值聚合現有的結果，從而實現行轉列的效果。在沒有支持PIVOT前，要實現這個需求，需要結合GROUP BY語法+聚合函數+Filter語法過濾來實現。

場景2

某個業務表，需要構造一個新的列，把原有的幾個列名合併在這個列裏面，並且用另一個新列來放置原來幾個列的值，從而實現列轉行的效果。在沒有支持UNPIVOT前，要實現這個需求，需要結合CROSS JOIN語法+CASE WHEN表達式來構造實現。

PIVOT/UNPIVOT功能

PIVOT

PIVOT概述

PIVOT語法將指定的行旋轉爲多列，並且對其餘列值聚合得到結果並旋轉表。PIVOT語法是FROM子句的一部分。

SELECT ... 
FROM ... 
PIVOT ( 
    <aggregate function> [AS <alias>] [, <aggregate function> [AS <alias>]] ... 
    FOR (<column> [, <column>] ...) 
    IN ( 
        (<value> [, <value>] ...) AS <new column> 
        [, (<value> [, <value>] ...) AS <new column>] 
        ... 
       ) 
    ) 
[...]

<aggregate_function>

表示行轉列時需要計算的聚合函數，且聚合函數的外層不能嵌套任何函數，可以是Scalar函數和列組成的表達式。同時聚合函數的參數內部不能有其他聚合函數、Window函數，以及聚合函數的列只能是上游表中的列。

<alias>

表示行轉列時需要計算的聚合函數的對應列的別名。

<column>

表示行轉列的對應行的列名，不能是任何的表達式。

<value>

表示行轉列的對應行的值，也可以是表達式，但是不允許有任何的聚合函數和窗口函數，並且每一個元組內的元素數量要與<column>數量一致。

<new_column>

表示行轉列後新的列的別名，不指定別名時，會試圖推測別名，推測失敗會由系統自動生成一個別名。

更詳細的語法使用說明可參考文檔。

PIVOT語法可以等效爲group by + aggregate function + filter的結合。以下面這個例子爲例

SELECT ...
FROM ...
PIVOT (
 agg1 AS a, agg2 AS b, ...
 FOR (axis1, ..., axisN)
 IN (
     (v11, ..., v1N) AS label1,
     (v21, ..., v2N) AS label2, 
     ...)
 )

上面的語法等效於

SELECT 
 k1, ... kN, 
 agg1 AS label1_a FILTER (where axis1 = v11 and ... and axisN = v1N), 
 agg2 AS label1_b FILTER (where axis1 = v21 and ... and axisN = v2N), 
 ..., 
 agg1 AS label2_a FILTER (where axis1 = v11 and ... and axisN = v1N),
 agg2 AS label2_b FILTER (where axis1 = v21 and ... and axisN = v2N), 
 ..., 
 FROM xxxxxx
 GROUP BY k1, ... kN

其中FROM內的表是PIVOT上游的結果，k1, ... kN是所有未在agg1, agg2, ...和axis1, ..., axisN出現的列的集合。

PVIOT示例

數據準備。以下表代表幾家連鎖店對應物品在對應年份的銷售情況。

create table shops_table as select * from (select * from values
('pen', 10, 500, 'shop1', 2020),
('pen', 11, 500, 'shop2', 2020),
('pen', 9, 300, 'shop3', 2020),
('pen', 12, 400,'shop4', 2020),
('pen', 15, 200, 'shop1', 2021),
('pen', 16, 300, 'shop2', 2021),
('pen', 16, 400, 'shop3', 2021),
('pen', 15, 300, 'shop4', 2021),
('ruler', 20, 700, 'shop1', 2020),
('ruler', 19, 900, 'shop2', 2020),
('ruler', 22, 800, 'shop3', 2020),
('ruler', 19, 700, 'shop4', 2020),
('ruler', 25, 300, 'shop1', 2021),
('ruler', 20, 500, 'shop2', 2021),
('ruler', 23, 500, 'shop3', 2021),
('ruler', 26, 600, 'shop4', 2021)
shops(item_name, count, sales, shop_name, year));
select * from shops_table;
-- 結果如下：
+-----------+------------+------------+-----------+------------+
| item_name | count      | sales      | shop_name | year       |
+-----------+------------+------------+-----------+------------+
| pen       | 10         | 500        | shop1     | 2020       |
| pen       | 11         | 500        | shop2     | 2020       |
| pen       | 9          | 300        | shop3     | 2020       |
| pen       | 12         | 400        | shop4     | 2020       |
| pen       | 15         | 200        | shop1     | 2021       |
| pen       | 16         | 300        | shop2     | 2021       |
| pen       | 16         | 400        | shop3     | 2021       |
| pen       | 15         | 300        | shop4     | 2021       |
| ruler     | 20         | 700        | shop1     | 2020       |
| ruler     | 19         | 900        | shop2     | 2020       |
| ruler     | 22         | 800        | shop3     | 2020       |
| ruler     | 19         | 700        | shop4     | 2020       |
| ruler     | 25         | 300        | shop1     | 2021       |
| ruler     | 20         | 500        | shop2     | 2021       |
| ruler     | 23         | 500        | shop3     | 2021       |
| ruler     | 26         | 600        | shop4     | 2021       |
+-----------+------------+------------+-----------+------------+

統計各個年份各個店對物品的賣出數量情況。
- 沒有支持PVIOT語法前，實現如下：

SELECT  item_name
        ,year
        ,SUM(CASE shop_name WHEN 'shop1' THEN count END) AS shop1
        ,SUM(CASE shop_name WHEN 'shop2' THEN count END) AS shop2
        ,SUM(CASE shop_name WHEN 'shop3' THEN count END) AS shop3
        ,SUM(CASE shop_name WHEN 'shop4' THEN count END) AS shop4
FROM    shops_table
GROUP BY item_name
         ,year
;
--結果如下：
+-----------+------------+------------+------------+------------+------------+
| item_name | year       | 'shop1'    | 'shop2'    | 'shop3'    | 'shop4'    |
+-----------+------------+------------+------------+------------+------------+
| pen       | 2020       | 10         | 11         | 9          | 12         |
| pen       | 2021       | 15         | 16         | 16         | 15         |
| ruler     | 2020       | 20         | 19         | 22         | 19         |
| ruler     | 2021       | 25         | 20         | 23         | 26         |

- 通過PVIOT語法實現如下：

select * from (select item_name, year,count,shop_name from shops_table)
pivot (sum(count) for shop_name in ('shop1', 'shop2', 'shop3', 'shop4'));
--結果如下：
+------------+------------+------------+------------+------------+------------+
| item_name  | year       | 'shop1'    | 'shop2'    | 'shop3'    | 'shop4'    | 
+------------+------------+------------+------------+------------+------------+
| pen        | 2020       | 10         | 11         | 9          | 12         | 
| pen        | 2021       | 15         | 16         | 16         | 15         | 
| ruler      | 2020       | 20         | 19         | 22         | 19         | 
| ruler      | 2021       | 25         | 20         | 23         | 26         | 
+------------+------------+------------+------------+------------+------------+

可以在此時爲聚合函數和新的列起別名，列名根據下劃線合併：

select * from (select item_name, count, shop_name, year from shops_table)
pivot (sum(count) as sum_count for shop_name in ('shop1' as shop_name_1, 'shop2' as shop_name_2, 'shop3' as shop_name_3, 'shop4' as shop_name_4));
--結果如下：
+------------+------------+-----------------------+-----------------------+-----------------------+-----------------------+
| item_name  | year       | shop_name_1_sum_count | shop_name_2_sum_count | shop_name_3_sum_count | shop_name_4_sum_count | 
+------------+------------+-----------------------+-----------------------+-----------------------+-----------------------+
| pen        | 2020       | 10                    | 11                    | 9                     | 12                    | 
| pen        | 2021       | 15                    | 16                    | 16                    | 15                    | 
| ruler      | 2020       | 20                    | 19                    | 22                    | 19                    | 
| ruler      | 2021       | 25                    | 20                    | 23                    | 26                    | 
+------------+------------+-----------------------+-----------------------+-----------------------+-----------------------+

計算每個物品每家商店每年的總賣出數量和最高銷售額，通過PIVOT實現如下：

select * from shops_table
pivot (sum(count) as sum_count, max(sales) as max_sales for shop_name in ('shop1' as shop_name_1, 'shop2' as shop_name_2, 'shop3' as shop_name_3, 'shop4' as shop_name_4));
--結果如下：
+-----------+------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
| item_name | year       | shop_name_1_sum_count | shop_name_2_sum_count | shop_name_3_sum_count | shop_name_4_sum_count | shop_name_1_max_sales | shop_name_2_max_sales | shop_name_3_max_sales | shop_name_4_max_sales |
+-----------+------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
| pen       | 2020       | 10                    | 11                    | 9                     | 12                    | 500                   | 500                   | 300                   | 400                   |
| pen       | 2021       | 15                    | 16                    | 16                    | 15                    | 200                   | 300                   | 400                   | 300                   |
| ruler     | 2020       | 20                    | 19                    | 22                    | 19                    | 700                   | 900                   | 800                   | 700                   |
| ruler     | 2021       | 25                    | 20                    | 23                    | 26                    | 300                   | 500                   | 500                   | 600                   |
+-----------+------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+

只計算shop1在2020年和2021對於每件物品的總賣出數量和最高銷售額，通過PIVOT實現如下：

select * from shops_table
pivot (sum(count) as sum_count, max(sales) as max_sales for (shop_name, year) in (('shop1', 2021) as shop1_2021, ('shop1', 2020) as shop1_2020));
--結果如下：
+-----------+----------------------+----------------------+----------------------+----------------------+
| item_name | shop1_2021_sum_count | shop1_2020_sum_count | shop1_2021_max_sales | shop1_2020_max_sales |
+-----------+----------------------+----------------------+----------------------+----------------------+
| pen       | 15                   | 10                   | 200                  | 500                  |
| ruler     | 25                   | 20                   | 300                  | 700                  |
+-----------+----------------------+----------------------+----------------------+----------------------+

UNPIVOT

UNPIVOT概述

UNPIVOT語法通過將列轉換爲行來旋轉表格，UNPIVOT語法是FROM子句的一部分。

SELECT ...
FROM ...
UNPIVOT [EXCLUDE NULLS] (
    <new_column_of_name> [, <new_column_of_name>] ...
    FOR (<new_column_of_value> [, <new_column_of_value>] ...)
    IN (
        (<column> [, <column>] ...) AS (<column_value> [, <column_value>] ...)
        [, (<column> [, <column>] ...) AS (<column_value> [, <column_value>] ...)]
        ...
       )
    )
[...]

[EXCLUDE NULLS]

若指定該語法，則會過濾掉所有都是null的行。

<new_column_of_name>

列轉行以後用於存儲原有的列名的列，必須爲列名不能是表達式也不能重名。數量需要和每一個<column value>元祖內部元素的數量相同，其中<column value>不指定時，MaxCompute會自動生成一組string類型的元祖。

<new_column_of_value>

列轉行以後用於存儲原有的列對應值的列，必須爲列名不能是表達式也不能重名，數量需要和每一個<column>元祖內部元素的數量相同。

<column>

用於列轉行的原有的列。

<column_value>

用於列轉行的原有的列的別名，可以用於替換原有的列名，內部不允許有列名。

更詳細的語法使用說明可參考文檔。

UNPIVOT語法可以等效爲CROSS JOIN + CASE WHEN表達式的結合。以下面這個例子爲例：

SELECT ...
FROM ...
UNPIVOT [exclude nulls] (
 (measure1, ..., measureM)
 FOR (axis1, ..., axisN)
 IN ((c11, ..., c1M) AS (value11, ..., value1N),
     (c21, ..., c2M) AS (value21, ..., value2N), ...))
[...]

上面的語法等效於

SELECT  * FROM
(
 SELECT
 k1, ... kN,
 CASE 
 WHEN axis1 = value11 AND ... AND axisN = value1N THEN c11
 WHEN axis1 = value21 AND ... AND axisN = value2N THEN c21
 ...
 ELSE null AS measure1,
 ..., 
 CASE 
 WHEN axis1 = value11 AND ... AND axisN = value1N THEN c1M
 WHEN axis1 = value21 AND ... AND axisN = value2N THEN c2M
 ELSE null AS measureM, 
 axis1, ..., axisN
 FROM xxxx 
 JOIN (VALUES (value11, ..., value1N),(value21, ..., value2N), ... AS generated_table_name(axis1, ..., axisN))
)
[WHERE measure1 is not null OR ... OR measureM is not null]

UNPIVOT示例

數據準備。以下表代表幾家連鎖店對應物品在對應年份的銷售情況：

create table shops as select * from （select * from values
('pen', 2020, 100, 200, 300, 400),
('pen', 2021, 100, 200, 200, 100),
('ruler', 2020, 300, 400, 300, 200),
('ruler', 2021, 400, 300, 100, 100)
shops(item_name, year, shop1, shop2, shop3, shop4)）;
SELECT * from shops;
--執行結果：
+-----------+------------+------------+------------+------------+------------+
| item_name | year       | shop1      | shop2      | shop3      | shop4      |
+-----------+------------+------------+------------+------------+------------+
| pen       | 2020       | 100        | 200        | 300        | 400        |
| pen       | 2021       | 100        | 200        | 200        | 100        |
| ruler     | 2020       | 300        | 400        | 300        | 200        |
| ruler     | 2021       | 400        | 300        | 100        | 100        |
+-----------+------------+------------+------------+------------+------------+

旋轉表，得到各個商店的銷售數量，並且用新的列名count來替代。
- 沒有UNPIVOT前的實現方式：

select * from(
select item_name,year, 'shop1' as shop_name, shop1 as count from shops
union ALL 
select item_name,year, 'shop2' as shop_name, shop2 as count from shops
UNION ALL 
select item_name,year, 'shop3' as shop_name, shop3 as count from shops
UNION ALL  
select item_name,year, 'shop4' as shop_name, shop4 as count from shops
);
--執行結果
+------------+------------+------------+------------+
| item_name  | year       | shop_name  | count      | 
+------------+------------+------------+------------+
| pen        | 2020       | shop1      | 100        | 
| pen        | 2021       | shop1      | 100        | 
| ruler      | 2020       | shop1      | 300        | 
| ruler      | 2021       | shop1      | 400        | 
| pen        | 2020       | shop2      | 200        | 
| pen        | 2021       | shop2      | 200        | 
| ruler      | 2020       | shop2      | 400        | 
| ruler      | 2021       | shop2      | 300        | 
| pen        | 2020       | shop3      | 300        | 
| pen        | 2021       | shop3      | 200        | 
| ruler      | 2020       | shop3      | 300        | 
| ruler      | 2021       | shop3      | 100        | 
| pen        | 2020       | shop4      | 400        | 
| pen        | 2021       | shop4      | 100        | 
| ruler      | 2020       | shop4      | 200        | 
| ruler      | 2021       | shop4      | 100        | 
+------------+------------+------------+------------+

- 通過UNPIVOT實現：

select * from shops
unpivot (count for shop_name in (shop1, shop2, shop3, shop4));
--執行結果
+------------+------------+------------+------------+
| item_name  | year       | count      | shop_name  | 
+------------+------------+------------+------------+
| pen        | 2020       | 100        | shop1      | 
| pen        | 2020       | 200        | shop2      | 
| pen        | 2020       | 300        | shop3      | 
| pen        | 2020       | 400        | shop4      | 
| pen        | 2021       | 100        | shop1      | 
| pen        | 2021       | 200        | shop2      | 
| pen        | 2021       | 200        | shop3      | 
| pen        | 2021       | 100        | shop4      | 
| ruler      | 2020       | 300        | shop1      | 
| ruler      | 2020       | 400        | shop2      | 
| ruler      | 2020       | 300        | shop3      | 
| ruler      | 2020       | 200        | shop4      | 
| ruler      | 2021       | 400        | shop1      | 
| ruler      | 2021       | 300        | shop2      | 
| ruler      | 2021       | 100        | shop3      | 
| ruler      | 2021       | 100        | shop4      | 
+------------+------------+------------+------------+

如果shop1和shop2是東區商店，shop3和shop4是西區商店，接下來需要一個新的列表示東區商店和西區商店。其中count1和count2兩列分別存儲了兩店的銷售數量。

select * from shops
unpivot ((count1, count2) for shop_name in ((shop1, shop2) as 'east_shop', (shop3, shop4) as 'west_shop'));
--執行結果
+------------+------------+------------+------------+------------+
| item_name  | year       | count1     | count2     | shop_name  | 
+------------+------------+------------+------------+------------+
| pen        | 2020       | 100        | 200        | east_shop  | 
| pen        | 2020       | 300        | 400        | west_shop  | 
| pen        | 2021       | 100        | 200        | east_shop  | 
| pen        | 2021       | 200        | 100        | west_shop  | 
| ruler      | 2020       | 300        | 400        | east_shop  | 
| ruler      | 2020       | 300        | 200        | west_shop  | 
| ruler      | 2021       | 400        | 300        | east_shop  | 
| ruler      | 2021       | 100        | 100        | west_shop  | 
+------------+------------+------------+------------+------------+

別名可以是多列，但是對應的需要生成的新的列名要相應增加：

select * from shops
unpivot ((count1, count2) for (shop_name, location) in ((shop1, shop2) as ('east_shop', 'east'), (shop3, shop4) as ('west_shop', 'west')));
--執行結果
+------------+------------+------------+------------+------------+------------+
| item_name  | year       | count1     | count2     | shop_name  | location   | 
+------------+------------+------------+------------+------------+------------+
| pen        | 2020       | 100        | 200        | east_shop  | east       | 
| pen        | 2020       | 300        | 400        | west_shop  | west       | 
| pen        | 2021       | 100        | 200        | east_shop  | east       | 
| pen        | 2021       | 200        | 100        | west_shop  | west       | 
| ruler      | 2020       | 300        | 400        | east_shop  | east       | 
| ruler      | 2020       | 300        | 200        | west_shop  | west       | 
| ruler      | 2021       | 400        | 300        | east_shop  | east       | 
| ruler      | 2021       | 100        | 100        | west_shop  | west       | 
+------------+------------+------------+------------+------------+------------+

小結

PIVOT/UNPIVOT語法，以更簡潔易用的方式滿足行轉列和列轉行的需求，簡化了查詢語句，提高了廣大大數據開發者的生產力。

點擊立即免費試用雲產品開啓雲上實踐之旅！

原文鏈接

本文爲阿里雲原創內容，未經允許不得轉載

MaxCompute - ODPS重裝上陣第十二彈 - PIVOT/UNPIVOT

前言

PIVOT/UNPIVOT功能

PIVOT

UNPIVOT

小結

.Net 8.0 下的新RPC，IceRPC之試試的新玩法"打洞"

完美替代postman的軟件

Vue mockjs mock.js

關於遊戲付費的一點想法

我通過CKA和CKS啦！

安裝chromadb注意事項

《最新出爐》系列入門篇-Python+Playwright自動化測試-42-強大的可視化追蹤利器Trace Viewer

大數據怎麼學？對大數據開發領域及崗位的詳細解讀，完整理解大數據開發領域技術體系

Java集合中的Set

安裝chromadb注意事項

前端面試題 - null是原始類型，但爲什麼typeof null的結果是object？

前端面試題 - 如何實現promise？

Java中的List

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

MaxCompute - ODPS重裝上陣 第十二彈 - PIVOT/UNPIVOT

前言

PIVOT/UNPIVOT功能

PIVOT

UNPIVOT

小結

MaxCompute - ODPS重裝上陣第十二彈 - PIVOT/UNPIVOT