本案例來自hive官方文檔
需求:一份有關觀看電影的日誌(u.data),有10W條數據,分析出一週內那天觀影人數多
1、創建原表
CREATE TABLE u_data (
userid INT,
movieid INT,
rating INT,
unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH '/usr/datas/u.data' into table u_data;
2、創建新表
CREATE TABLE u_data_new (
userid INT,
movieid INT,
rating INT,
weekday INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t';
3、編寫python腳本用來處理unixtime轉化爲星期數
import sys
import datetime
for line in sys.stdin:
line = line.strip()
userid, movieid, rating, unixtime = line.split('\t')
weekday = datetime.datetime.fromtimestamp(float(unixtime)).isoweekday()
print '\t'.join([userid, movieid, rating, str(weekday)])
4、使用腳本並插入數據到新表
add FILE /usr/datas/weekday_mapper.py;
INSERT OVERWRITE TABLE u_data_new
SELECT
TRANSFORM (userid, movieid, rating, unixtime)
USING 'python weekday_mapper.py'
AS (userid, movieid, rating, weekday)
FROM u_data;
5、查詢
SELECT weekday, COUNT(1) count
FROM u_data_new
GROUP BY weekday order by count desc;