一、背景分析
有近20年的廣告數據，需要對外提供查詢分析服務（對數據進行Ad-Hoc式查詢，Ad-Hoc：即席查詢，允許終端用戶靈活的自定義、創建查詢條件，後端引擎根據發送過來的查詢請求生成用戶要求的報表、統計分析結果。即席查詢是非定製化的，不可預知的。），當前採用了Apache Impala作爲查詢分析引擎。這些廣告數據涵蓋了各種媒介，每種媒介又進行定義了不同的媒體。
因此一條數據可以表示爲，誰（廣告主）在什麼時候（時間）、在哪裏（媒體）播放了一條什麼樣的（產品）廣告，這樣的一條日誌數據。在數據生產的過程中，存在一定的識別誤差，在後續的複查環節需要對T-1或者T-7的數據進行修正，但T-1或者T-7的數據已經進入數倉且對外提供數據分析服務（這裏已經產生數據震盪），因此需要對數倉中的數據進行修正。
生產環節對數據進行修正的粒度不是行級的，而是媒體/天的粒度（一個媒體這一天的數據都要進行修正），層層傳遞到數倉也是媒體/天的粒度進行數據的修正更新。這纔有了這個按照媒體/天的動態分區。
二、Impala分區
當數據的體量大時，使用分區技術能很好的縮小查詢數據的範圍，從而提高查詢的效率以及節省磁盤的IO、CPU、內存等資源。分區分爲兩種：靜態分區、動態分區。
靜態分區：Specifying all the partition columns in a SQL statement is called static partitioning, because the statement affects a single predictable partition. For example, you use static partitioning with an ALTER TABLE statement that affects only one partition, or with an INSERT statement that inserts all values into the same partition:
insert into t1 partition(x=10, y='a') select c1 from some_other_table;
動態分區：When you specify some partition key columns in an INSERT statement, but leave out the values, Impala determines which partition to insert. This technique is called dynamic partitioning:
insert into t1 partition(x, y='b') select c1, c2 from some_other_table;
-- Create new partition if necessary based on variable year, month, and day; insert a single value.
insert into weather partition (year, month, day) select 'cloudy',2014,4,21;
-- Create new partition if necessary for specified year and month but variable day; insert a single value.
insert into weather partition (year=2014, month=04, day) select 'sunny',22;
The more key columns you specify in the PARTITION clause, the fewer columns you need in the SELECT list. The trailing columns in the SELECT list are substituted in order for the partition key columns with no specified value.
三、操作步驟
INSERT INTO medium_data.tv_data partition (media_id,ds)
SELECT
tv.id
,tv.m_type
,tv.ad_date
,tv.sub_id
,tv.ad_cost
,tv.ad_time
,tv.ad_length
,tv.cate_id
,tv.lang_id
,tv.qual_id
,tv.spec_id
,tv.ad_posit
,tv.t_break
,tv.t_b_net
,tv.pre_s_id
,tv.post_s_id
,tv.pre_pg_id
,tv.post_pg_id
,CAST(weekofyear(ad_date) AS INT) AS week_of_year
,concat_ws(' ', substr(AD_Date,1, 10), ad_time) AS ad_timestamp
,CAST(CASE WHEN dayofweek(ad_date) = 1 THEN 7 ELSE dayofweek(ad_date) - 1 END AS INT) AS day_of_week
,year(tv.ad_date) AS `year`
,case when month(ad_date) in (1,2,3) THEN 1 when month(ad_date) in (4,5,6) THEN 2 when month(ad_date) in (7,8,9) THEN 3 when month(ad_date) in (10,11,12) THEN 4 END AS quarter
,v.prod_id AS product_id
,v.factory_id
,v.brand_id
,tv.media_id
,from_unixtime(unix_timestamp(ad_date), 'yyyymmdd') AS ds
FROM
kudu.tv_data tv
LEFT JOIN
(
SELECT
vtv.ver_id,
p.prod_id,
p.factory_id,
p.brand_id AS brand_id
FROM
kudu_dim.v_tv vtv
LEFT JOIN
kudu_dim.product p
ON
vtv.prod_id = p.prod_id
WHERE p.valid_flag != 'X'
) v
ON
tv.sub_id = v.ver_id
WHERE tv.s_type = 1
檢查SQL的執行計劃
+-----------------------------------------------------------------------------------------------------------------------------------------+
| Explain String |
+-----------------------------------------------------------------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=80.00MB Threads=7 |
| Per-Host Resource Estimates: Memory=5.11GB |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| kudu.tv_data, kudu_dim.product, kudu_dim.v_tv |
| |
| WRITE TO HDFS [medium_data.tv_data, OVERWRITE=false, PARTITION-KEYS=(media_id,from_unixtime(unix_timestamp(ad_date), 'yyyyMMdd'))] |
| | partitions=unavailable |
| | |
| 08:SORT |
| | order by: media_id ASC NULLS LAST, from_unixtime(unix_timestamp(ad_date), 'yyyyMMdd') ASC NULLS LAST |
| | |
| 07:EXCHANGE [HASH(tv.media_id,from_unixtime(unix_timestamp(ad_date), 'yyyyMMdd'))] |
| | |
| 04:HASH JOIN [LEFT OUTER JOIN, BROADCAST] |
| | hash predicates: tv.sub_id = vtv.ver_id |
| | |
| |--06:EXCHANGE [BROADCAST] |
| | | |
| | 03:HASH JOIN [LEFT OUTER JOIN, BROADCAST] |
| | | hash predicates: vtv.prod_id = p.prod_id |
| | | other predicates: p.valid_flag != 'X' |
| | | |
| | |--05:EXCHANGE [BROADCAST] |
| | | | |
| | | 02:SCAN KUDU [kudu_dim.product p] |
| | | predicates: p.valid_flag != 'X' |
| | | |
| | 01:SCAN KUDU [kudu_dim.v_tv vtv] |
| | |
| 00:SCAN KUDU [kudu.tv_data tv] |
| kudu predicates: tv.subj_type = 1 |
+-----------------------------------------------------------------------------------------------------------------------------------------+
分區信息
| media_id | ds | #Row | #Files | Size | Byte Cached | Cache Replication | Format | Incremental stats | Location
..................
| 1032 | 20190119 | -1 | 2 | 48.00KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190119 |
| 1032 | 20190120 | -1 | 2 | 43.99KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190120 |
| 1032 | 20190121 | -1 | 2 | 44.94KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190121 |
| 1032 | 20190122 | -1 | 2 | 42.78KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190122 |
| 1032 | 20190123 | -1 | 2 | 44.27KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190123 |
| 1032 | 20190124 | -1 | 2 | 44.39KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190124 |
| 1032 | 20190125 | -1 | 2 | 45.65KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190125 |
| 1032 | 20190126 | -1 | 2 | 45.19KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190126 |
| 1032 | 20190127 | -1 | 2 | 43.16KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190127 |
| 1032 | 20190128 | -1 | 2 | 43.96KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190128 |
| 1032 | 20190129 | -1 | 2 | 42.52KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190129 |
| 1032 | 20190130 | -1 | 2 | 43.99KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190130 |
| 1032 | 20190131 | -1 | 2 | 44.68KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://nameservice1/user/hive/warehouse/medium_data.db/tv_data/media_id=1032/ds=20190131 |
| Total | | -1 | 38656 | 816.16MB | 0B | | | | |
+----------+----------+-------+--------+----------+--------------+-------------------+---------+-------------------+-----------------------------------------------------------------------------------------------+
Fetched 19376 row(s) in 15.49s
四、結果分析
根據Impala檢測平臺，整個任務執行完成約1小時20分鐘，而執行最耗時的是在Sort階段，裝載掃描數據僅僅耗時幾秒；分區的中文件也特別小，大約在50KB左右，總共19376個分區。
完成對一個分區COUNT的統計，需要4.77秒
完成對所有分區的COUNT統計，需要2分50秒
按媒體/天粒度，在分區目錄下的文件比較小，這是業務要求；但對於分區粒度不合適，會產生大量的磁盤IO（小文件隨機存放，在讀取的時候會特別消耗IO），不一定能從整體上提高查詢效率。如果在查詢中，對分區進行了裁剪，查詢還是很快的，隨着分區數量的增加，響應時間會慢慢的降低。在Impala中，單表通常分區數量不要超過1萬，太多的分區也會造成元數據管理的性能下降，在表元數據同步失敗時，就會造成查不出數據或者看不到分區。
PS：Impala Catalog
1、表信息
2、表分區信息
3、表以及分區下的文件、Block塊信息
4、表以及分區的統計信息
五、主機配置
TODO：測試集羣的角色規劃有些亂，後續會進行角色的規劃以及動態資源池、靜態資源池的設置。
Impala Daemon（mem_limit）的內存設置爲8G（由守護程序本身強制執行的 Impala Daemon 的內存限制（以字節爲單位）。如果達到該限制，Impalad Daemon 上運行的查詢可能會被停止。將其留空可以讓 Impala 選擇自己的限制。使用 -1 B 值將指定無任何限制）。
Impala Daemon 的 Java 堆棧大小（Impala Daemon JVM Heap）爲4G（Java 進程堆棧內存的最大大小（以字節爲單位），已傳遞到 Java -Xmx。）。
Idle Query Timeout（idle_query_timeout）設置爲0（The time that a query may be idle (i.e. no processing work is done and no updates are received from the client) before it is cancelled. If 0, idle queries are never expired.）。這裏需要進行修改 空閒查詢：如果結果已經取出但沒有關閉查詢，或者結果部分取出但客戶端程序不再請求更多的數據查詢。這種情況一般只發生在JDBC、ODBC接口的程序中。