浅谈hive常用窗口函数
目录
简介
窗口函数又名开窗函数，属于分析函数的一种，用于解决复杂报表统计需求的功能强大的函数。窗口函数用来计算基于组的某种聚合值，它和聚合函数的不同之处是：对于每个组返回多行，而聚合函数对于每个组只返回一行。
开窗函数指定了分析函数工作的数据窗口大小，这个数据窗口大小可能会随着行的变化而变化。
常用窗口函数
over
- over() 通常与聚合函数共同使用，比如 count()、sum()、min()、max()、avg() 等。
- over() 具有一定的窗口语义 ，如：OVER(ROWS ((CURRENT ROW) | (UNBOUNDED) PRECEDING) AND (UNBOUNDED |(CURRENT ROW) ) FOLLOWING )，不过这些窗口定义经常与聚合函数（sum min max）相结合使用，像一些序列函数（row number、rank等）是不可以使用的
- over() 直接使用时，通常是指定全量数据，当我们想要按某列的不同值进行窗口划分时，可以在 over() 中加入 partition by 语句。
在单独进行明细和count聚合的时候都会报错，但是加上窗口就可以正常执行
select *,count(*) from t_dw_orders_his
-------------------------------------------------------------------------------------------
Error while compiling statement: FAILED: SemanticException [Error 10025]: Expression not in GROUP BY key orderid
select *,count(*) over() from t_dw_orders_his where p_event_date="2015-08-22"
//可以正常得到结果
10 2015-08-22 2015-08-22 支付 2015-08-22 9999-12-31 2015-08-22 17
9 2015-08-22 2015-08-22 创建 2015-08-22 9999-12-31 2015-08-22 17
8 2015-08-21 2015-08-22 支付 2015-08-22 9999-12-31 2015-08-22 17
8 2015-08-21 2015-08-21 创建 2015-08-21 2015-08-21 2015-08-22 17
7 2015-08-20 2015-08-21 支付 2015-08-21 9999-12-31 2015-08-22 17
7 2015-08-20 2015-08-21 支付 2015-08-20 2015-08-20 2015-08-22 17
6 2015-08-20 2015-08-22 支付 2015-08-22 9999-12-31 2015-08-22 17
6 2015-08-20 2015-08-20 创建 2015-08-20 2015-08-21 2015-08-22 17
5 2015-08-19 2015-08-20 支付 2015-08-19 9999-12-31 2015-08-22 17
4 2015-08-19 2015-08-21 完成 2015-08-21 9999-12-31 2015-08-22 17
4 2015-08-19 2015-08-21 完成 2015-08-19 2015-08-20 2015-08-22 17
3 2015-08-19 2015-08-21 支付 2015-08-21 9999-12-31 2015-08-22 17
3 2015-08-19 2015-08-21 支付 2015-08-19 2015-08-20 2015-08-22 17
2 2015-08-18 2015-08-22 完成 2015-08-22 9999-12-31 2015-08-22 17
2 2015-08-18 2015-08-18 创建 2015-08-18 2015-08-21 2015-08-22 17
1 2015-08-18 2015-08-22 支付 2015-08-22 9999-12-31 2015-08-22 17
SUM,AVG,MIN,MAX
此类聚合函数用户类似，在此我们以SUM为例结合OVER的窗口语句进行总结
准备数据
CREATE TABLE orders1(
`orderid` int,
`createtime` string,
`money` int)
-----------------------
SELECT * FROM orders1
-----------------------
1 2015-08-18 72
1 2015-08-19 19
1 2015-08-20 67
1 2015-08-21 78
1 2015-08-22 62
1 2015-08-23 62
各种over参数情况下效果如下
SELECT orderid,
createtime,
money,
SUM(money) OVER() AS money1,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC) AS money2,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS money3,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS money4,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS money5,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS money6
FROM orders1;
结果如图
总结
PRECEDING：往前数几行
FOLLOWING：往后
CURRENT ROW：当前行
UNBOUNDED：起点，UNBOUNDED PRECEDING 表示从前面的起点， UNBOUNDED FOLLOWING：表示到后面的终点
特别注意当上面的那个demo如果createtime有重复值,则会意想不到的效果，结果如下面请参考
SELECT * FROM orders1
1 2015-08-18 72
1 2015-08-18 72
1 2015-08-19 78
1 2015-08-19 19
1 2015-08-19 72
1 2015-08-20 62
1 2015-08-20 62
1 2015-08-21 67
1 2015-08-22 78
1 2015-08-22 24
1 2015-08-23 67
1 2015-08-23 19
1 2015-08-23 19
同样的执行下面这个sql
SELECT orderid,
createtime,
money,
SUM(money) OVER() AS money1,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC) AS money2,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS money3,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS money4,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS money5,
SUM(money) OVER(PARTITION BY orderid ORDER BY createtime ASC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS money6
FROM orders1;
结果如下
NTILE
NTILE(n)，切片函数，用于将分组数据按照顺序切分成n片，返回当前切片值，如果切片不均匀，默认增加第一个切片的分布
准备数据如下
CREATE TABLE orders1(
`orderid` int,
`createtime` string,
`money` int)
-----------------------
SELECT * FROM orders1
-----------------------
1 2015-08-18 72
1 2015-08-19 19
1 2015-08-20 67
1 2015-08-21 78
1 2015-08-22 62
1 2015-08-23 62
执行sql效果如下
SELECT
orderid,
createtime,
money,
NTILE(2) OVER(PARTITION BY orderid ORDER BY createtime) AS rn1,
NTILE(3) OVER(PARTITION BY orderid ORDER BY createtime) AS rn2,
NTILE(4) OVER(ORDER BY createtime) AS rn3
FROM orders1
结果如下，可以注意下分成4个切片的情况，数据共有6组，分成4组切片的时候每组不足两个，结果第三组和第四组各有1个
ROW_NUMBER
ROW_NUMBER() –从1开始，按照顺序，生成分组内记录的序列，这个函数是非常常用的一个窗口函数，应用场景非常广泛，如在各种求日活月活的场景（配合where rn=1的用法比较多）
SELECT
orderid,
createtime,
money,
row_number() OVER(PARTITION BY orderid ORDER BY createtime) AS rn
FROM orders1
RANK & DENSE_RANK
—RANK() 生成数据项在分组中的排名，排名相等会在名次中留下空位,数字是不连续的
—DENSE_RANK() 生成数据项在分组中的排名，排名相等会在名次中不会留下空位，数字是连续的
SELECT
orderid,
createtime,
money,
rank() OVER(PARTITION BY orderid ORDER BY money) AS rn1,
dense_rank() OVER(PARTITION BY orderid ORDER BY money) AS rn2
FROM orders1
CUME_DIST&PERCENT_RANK
–CUME_DIST 小于等于当前值的行数/分组内总行数
–比如，统计小于等于当前薪水的人数，所占总人数的比例
–PERCENT_RANK 分组内当前行的RANK()函数值-1/分组内总行数-1
SELECT
orderid,
createtime,
money,
cume_dist() OVER(PARTITION BY orderid ORDER BY money) AS rn1,
percent_rank() OVER(PARTITION BY orderid ORDER BY money) AS rn2,
rank() OVER(PARTITION BY orderid ORDER BY money) AS rn3
FROM orders1
LAG
LAG(col,n,DEFAULT) 用于统计窗口内往上第n行值
第一个参数为列名，第二个参数为往上第n行（可选，默认为1），第三个参数为默认值（当往上第n行为NULL时候，取默认值，如不指定，则为NULL）
SELECT
orderid,
createtime,
money,
lag(money,1,null) OVER(PARTITION BY orderid ORDER BY money) AS rn1,
lag(money,2,22) OVER(PARTITION BY orderid ORDER BY money) AS rn2,
lag(money,3,33) OVER(PARTITION BY orderid ORDER BY money) AS rn3
FROM orders1
LEAD
与LAG相反
LEAD(col,n,DEFAULT) 用于统计窗口内往下第n行值
第一个参数为列名，第二个参数为往下第n行（可选，默认为1），第三个参数为默认值（当往下第n行为NULL时候，取默认值，如不指定，则为NULL）
SELECT
orderid,
createtime,
money,
lead(money,1,null) OVER(PARTITION BY orderid ORDER BY money) AS rn1,
lead(money,2,22) OVER(PARTITION BY orderid ORDER BY money) AS rn2,
lead(money,3,33) OVER(PARTITION BY orderid ORDER BY money) AS rn3
FROM orders1
FIRST_VALUE&LAST_VALUE
--FIRST_VALUE取分组内排序后，截止到当前行，第一个值
--LAST_VALUE取分组内排序后，截止到当前行，最后一个值
SELECT
orderid,
createtime,
money,
first_value(money) OVER(PARTITION BY orderid ORDER BY money) AS rn1,
last_value(money) OVER(PARTITION BY orderid ORDER BY money) AS rn2,
first_value(money) OVER(PARTITION BY orderid ORDER BY money desc) AS rn11,
last_value(money) OVER(PARTITION BY orderid ORDER BY money desc) AS rn22
FROM orders1
