現有hdfs路徑hadoop fs -du -h /user/portal/ODM/RECOMMEND/abc;
路徑下有每天分區
139.1 M /user/portal/ODM/RECOMMEND/readAgentInfo/abc
744.4 K /user/portal/ODM/RECOMMEND/readAgentInfo/abc.2018-12-04
1.1 M /user/portal/ODM/RECOMMEND/readAgentInfo/abc.2018-12-05
現有兩種建表方法,
1.直接插入時間分區, 然後存儲json字符串, 而後解析;
2.不插入時間分區, 底層表拆分字段, 然後解析底層表中的時間字段, 用底層表的時間字段當partition
我選的第二種
建表的時候注意:
CREATE EXTERNAL TABLE IF NOT EXISTS portal.odm_recommend_readagentinfo_recid_info_day
(
abc string comment "
)
COMMENT ‘asd’
row format delimited
fields terminated by ‘\t’ lines terminated by ‘\n’ stored as textfile
LOCATION ‘/user/portal/ODM/RECOMMEND/readAgentInfo’;
然後上層建表
注意時間的選擇
concat(substr(insertTime,1,4),substr(insertTime,6,2),substr(insertTime,9,2)) = ‘${ts}’