Apache Kylin 在有讚的高性能運維實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在本月舉辦的 Apache Kylin Meetup 中,我們邀請到了來自有贊科技的鄭生俊來分享 Apache Kylin 在有讚的高性能運維實踐:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面對客戶增長分析等靈活業務場景,有贊是如何使用 Kylin 提供高質量服務的?"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於 Kylin 查詢、構建和故障恢復等性能瓶頸,有贊又有哪些優化?"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 K8s 彈性資源使用 Spark 構建 Cube 體驗如何?"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過有贊測試的 Kylin 4 到底表現如何?性能相較 Kylin on HBase 有何突破?"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下爲本次會議實錄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次分享主要包含以下四個部分:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有贊 OLAP 的介紹"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 在有贊多模塊應用場景"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"增強 Kylin 穩定性的經驗分享"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 4.0 在有讚的計劃"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"有贊 OLAP 介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先介紹一下有贊 OLAP 的發展歷程:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/55\/f1\/557d2c9555aaf2b23aa33a3577cc10f1.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"早期 OLAP 平臺:預計算 + MySQL"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此方式存在靈活性差、開發效率低下等問題。但有贊當時業務處於起步階段,這種方式技術棧簡單,學習成本也很低。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018 年左右,有贊引進 Druid,但很快就無法滿足有讚的業務需求"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當時引進 Druid 是因爲其支持實時報表的開發,並且查詢靈活度比較高。但隨着有贊業務越來越複雜,Druid 的缺點也日益明顯,首先是不支持精確去重,其次是聚合度不高,還有實時數據的修復必須在 T+1,在 SaaS 場景下 RT(Response Time)和快速恢復無法滿足,同時也不支持明細的 ROLAP。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2019 年至今,有贊基本完成了把 Druid 逐步往 Kylin 和 ClickHouse 上遷移的工作"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"選擇 Kylin 的主要原因是:Kylin 的 RT 在所有預計算 OLAP 引擎裏是最低的,同時聚合度也比較高,精確去重和非精確去重的性能最強。引入 ClickHouse 則主要是支持明細 OLAP 的查詢。有贊目前和將來的 OLAP 就會包含 Kylin 和 ClickHouse 兩個技術棧。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kylin 在有讚的多模塊應用場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前有贊 OLAP 的業務場景,幾乎覆蓋了有贊所有的業務板塊。簡單介紹一下有贊目前的業務情況:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/82\/a7\/825ecb19740de5e41907c793de5005a7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務量:約 500 萬+ 的存量商家;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2020 年 GMV 1037 億;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Kylin 平臺構建 300+ Cube;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每日涉入構建數據約 100 億+(包含埋點、用戶行爲等數據,也包含交易、財務、庫存管理等各種數據。)"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"有贊 Kylin 業務場景:商家後臺報表"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/01\/c0\/01d2a847e56c8288ac1f9b8b26407dc0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先介紹 Kylin 在有讚的多模塊應用場景。上圖爲有贊商家後臺功能,右側是有贊商家後臺,包括財務報表、流量報表、庫存報表、履約、供應鏈、優惠等營銷相關的報表,涉及多個對外場景,是 SaaS 服務的一部分,每天有大量商家關注這些報表數據,因此對 RT 和穩定性的要求非常高的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"有贊 Kylin 業務場景:客戶增長分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以往大家會覺得 Kylin 只能做一些固定場景的,表結構固定、靈活度要求不高的一些業務場景,但其實並非如此。接下來介紹一個比較靈活的應用場景——客戶增長分析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖左側是有贊之前產品化的 PRD(Product Requirement Document)圖,最上面的產品能支持查詢某個部門下的客戶的流失、新增情況,同時可以支持跨部門的上卷下鑽。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/73\/6f\/73498d2abb25032f65061230661f086f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先簡單介紹背後的邏輯結構,存在兩個有挑戰的地方:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"組織架構需要靈活可變動"},{"type":"text","text":" ;在企業實際場景中,部門的層級和個數是靈活變動的,在這種情況下,如果使用通常的星型數據模型是無法解決這種靈活多變的需求。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"對於千萬級別粉絲的大店鋪,有贊能夠 "},{"type":"text","marks":[{"type":"strong"}],"text":"支持秒級以內 RT 的精確去重、上卷和下鑽"},{"type":"text","text":" 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對第一個問題,有贊選擇把樹狀的部門結構打平,例如部門 A 的新增人數就等於部門 B + 部門 C + 部門 D 新增人數的總和,對此有贊只存一個部門字段,並不會把所有樹狀結構存到 Kylin 裏,這樣就能避免表結構要隨着部門增加一直變化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了實現支持千萬級別粉絲會員商家的精確去重、上卷和下鑽,有贊採用了 Kylin Bitmap 的精確去重,具體的原理介紹這裏不再展開。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個數據開發的流程如下:首先是好友和粉絲關係的變動,當新增粉絲、微信好友添加,就判斷這種事件是客戶新增;當微信好友失聯、粉絲取消關注,就判斷爲客戶流失。我們將這些明細表存到數倉的拉鍊表裏,通過拉鍊表來避免同一個用戶在同一天時間內關係頻繁變動造成的重複計算;然後將這個拉鍊表通過樹狀結構打平變成數倉寬表,通過 Kylin 進行預計算;最後在頁面上就可以呈現趨勢和比例的分析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上是有贊中一個比較簡單但同時又有一定代表性的場景,我們通過一些 ETL 數據開發就能在 Kylin 裏實現這樣比較靈活的需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kylin 在有讚的技術實踐"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d2\/12\/d26099df55a76858687a124331688812.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先來簡單介紹一下 Kylin,如上圖所示,左側是一張 Hive 的明細表,通過各種的維度組合,計算出每個維度組合不同度量的值,存在 HBase 裏就變成一個經過預計算的,有一定膨脹度的長表了。此時如果有 Select from,Group by 等簡單查詢就會命中一個 Cuboid。在 Kylin 的 QueryServer 會進行 Calcite 語法規則的優化,包括消除聚合、消除 Join 等,最終轉化成一個簡單 HBase Get 的操作,通過預計算和優化可以將一個較爲複雜的聚合查詢轉換成一個 HBase Get 查詢,可以說 Kylin 有很好的 Response Time,在這裏 Rowkey 的維度順序和查詢的重要性是相關的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來介紹一下 Kylin 在有讚的技術實踐。有贊內部有一個業務和健康監測機制。有讚的業務使用有以下幾個原則:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大數據量查詢需要使用分區條件;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"固定和區分度大的過濾條件必須在 Rowkey 內順序靠前;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢儘量滿足 HBase Rowkey 前綴匹配的規則,有贊是在電商行業,做的是 SaaS 場景,店鋪 ID 是作爲 Rowkey 的第一位,從而保證一個查詢落在少量的 Region 內;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢非必要避免使用精確去重,儘量使用近似去重;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通常要求業務方做一個 Cube 剪枝;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"儘量減少 Lookup Snapshot Table 的使用,因爲會佔用大量的內存。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/05\/9d\/05b499a2c9ab6e89203399e71bb1519d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有讚的健康檢測機制會在 Kylin QueryServer 端加一些 Query metrics 實時發送到 Kafka,之後會有一個任務去監控各種 metrics, DB 會有一些 Cube 信息,比如膨脹率等,再錄到 DB 做一些審計和看板。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來介紹有贊遇到的一些性能的瓶頸點是如何優化的,主要從查詢、構建和故障恢復等三個方面進行介紹。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"查詢:In 查詢優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": 有讚的微商城和零售是兩套不同的業務體系,在技術棧統一的過程中,是需要將線上和線下的數據放到一個 Cube,導致單個 Cube 的數據量從原來的幾十幾百 G 上升到幾百 T。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這種場景下,需要查詢上百甚至上千家子店鋪,往往是用 in 作爲查詢條件,in 後會有幾百個店鋪 ID,再加上時間維度組合,就會在 Kylin 裏轉化成很多小區間的 HBase 的 Rowkey scan,這些小查詢叫作 Fuzzykey scan。當這些小查詢超過了一定量值,Kylin 會進行優化,把不連續的小區間查詢轉換化成一個大範圍的 HBase scan,但在數據量膨脹很多的情況下,可能會導致原本需要掃描的數據量乘以幾千倍,甚至幾萬倍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/e9\/93\/e96cffd3fae5c18df79d7f418090ce93.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了避免這種查詢場景下性能的損耗,有贊通過以下方式來解決:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其一,調大 Max 跟 Fuzzykey scan 之間的閾值,因爲這個閾值默認是 200,當查詢指定了幾十家門店,再加上指定幾天的時間,閾值就很容易就超過 200;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其二,減小 HBase Region 大小。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過以上兩種優化方式,有贊做到了在 Cube 數據量增加幾百倍的情況下,in 查詢的 RT 依舊能得到保證。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"查詢:使用 LRU Cache 緩存維度字典"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有贊遇到的第二個查詢 RT 不穩定的原因是在維度字典。Kylin 維度字典包括 Kylin 2 和 Kylin 3 的維度字典,是使用 Guava Loading Cache 去實現,Guava 的 Loading Cache 默認是一個 LRU 的 Cache,但存在一個問題,就是目前的 Cache 是基於 Soft Reference,實際上很容易被 GC(Garbage Collection)去清理的,一旦被 GC 清理,字典就需要從 HBase 去加載,當字典超過 10M,會從 HDFS 去加載,這樣就導致這個查詢的 RT 是不穩定的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/41\/86\/4136c173b32d30b030a28595aeeaa286.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對以上問題,有贊進行了優化,從而支持字典 Cache 緩存的 stress,例如 soft、wake 和 stress 等,目的是讓字典不會被 GC 清理,技術團隊可以通過控制緩存數量來實現內存控制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來第二部分,會介紹有贊在 Kylin 構建方面的實踐與優化。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"構建:EncodeBaseCuboid 優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": Kylin 在 EncodeBaseCuboid 的過程中,同時也是 Spark 構建 Cuboid 數據的第一步,會逐行逐列進行字典編碼,將字典編碼的維度、精確去重字段轉換爲整型。當 Cube 中有多個高基數、需要字典編碼的列存在時,效率會變得低下,因爲構建一行的過程中會涉及到不同列字典換進換出,降低構建的效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7f\/6f\/7f1c138c98e4133016f77755abf73e6f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示,有贊在構建過程做了一些優化,將逐行逐列的編碼改爲逐列逐行的編碼,會先構建 Column A1,對 A 列進行編碼,把 A1 編碼成 1,A2 編碼成 2,從而避免在構建一行的時候,內存中的字典需要在不停的列之間進行切換,從而提升構建效率。在有贊一個多個基數較大的維度字典、全局字典的場景中,經過優化,構建時間從原本的一個多小時提速到僅需 20 分鐘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於細節的實現可以關注:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/issues.apache.org\/jira\/browse\/KYLIN-4941"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"構建:重刷數據時限流策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": 有讚的主要的離線計算集羣規模在數萬 CPU 左右,計算量大、計算資源充足;線上查詢基於獨立集羣,規模較小,這就導致構建與查詢集羣資源不對等,刷數需要限流避免對線上集羣產生影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/97\/51\/97c9897e2a569207edb79fd67cc68c51.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有贊針對刷數任務,擴展了一個基於寫 HFile 的限流策略。這個限流策略可以配置一個 Job 在生成 HFlile 的過程中,能夠使用的流量。大致的實現思路是:在 Kylin 構建中,寫 HFile 的 Reduce Tasks 數目在上幾個階段根據 Segment Statistics 就已經確定了;我們通過這個流量除以 Reduce 的併發數,獲取到每一個 Reduce 允許的流量限制,再通過限流器在 Reduce 中的寫入過程進行限速。通過以上優化,我們能夠做到在刷數據的時候,保持前期較快的構建速度,同時在最終寫 HFile 的時候,又有一定程度的限速,保證在線查詢的穩定性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來介紹一下有贊故障恢復相關的優化。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"故障恢復:基於 K8s 彈性資源構建 Cube"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": 之前故障恢復需要重刷數據,往往耗費一、兩個小時。在商家較多的情況下,很容易就接收到大量商家的投訴,給有讚的服務質量帶來許多挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/cd\/a6\/cd7bd6c045dd2c208d36abfa5d5d9fa6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 構建是支持 Spark 的,至於是 Spark on YARN 還是 Spark on K8s 對 Kylin 是透明的,唯一有差別是 Job Tracking URL 的獲取需要定製化。出於資源的彈性伸縮和混部訴求,有贊基於 Spark on K8s 來構建 Cube,關於雲主機的彈性伸縮和混部,有興趣的讀者可以參考有贊這篇技術文章: "},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAxOTY5MDMxNA==&mid=2455762177&idx=1&sn=e72732ed9026a9700a9cf79adda69452&scene=21#wechat_redirect","title":"","type":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Spark on K8s 在有讚的實踐"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了基於 K8s 彈性資源構建 Cube,有贊還遇到了一些比較有挑戰的故障恢復的場景。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"故障恢復:元數據修復避免重新構建"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": 在 Cube Build 過程中或者 Merge 過程中,Job 分爲多個階段。會存在某些異常場景導致一個構建任務被多個 Job 實例獲取,最終導致元數據錯誤,無法查詢到數據。對於合併跨度大的 Segment,重刷數據耗時長。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何快速修復此故障?經過有贊團隊的研究和發現,查詢相關的元數據基本包含 4 個元數據信息:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/67\/30\/6779d65590e5de6babd0911b44854f30.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是 Segment Status,比較容易修復,通過 API 把 Segment 的 Status 置爲 Ready 就完成了;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二部分是 Cuboid Shard Map,這也是比較重要的一部分,當維度裏有空的 Shardby,這個 Shardby Number 就會作爲 Rowkey 的一部分,如果 Cuboid Shardby Number 不正確,就會影響查詢的準確性。比較幸運的是 Kylin 在構建的過程中會把一些 Cube statistics 信息存到構建的臨時文件,讀取這個臨時文件去更新 Cuboid Shard Map,就可以保證元數據是準確的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三部分是 Input Records Count,這個實際上是 KylinQureyServer 用來判斷 Segment 是否爲空的依據,只要這個 Input Records Count 不爲 0,就能被查詢到,其修復來源也是存儲在 HDFS Job 的臨時構建目錄裏。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後一部分是 HBase Segment 對應的 HBase Table Name,此信息在 Segment 創建時就已確定,創建過程中任何一個步驟都不會再去修改 HBase 對應的 Storage Identifier,所以這部分元數據是不需要去修復的。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過以上這幾個修復步驟,即使在元數據錯亂更新的情況下,也無需重算元數據,就能快速恢復。其實目前 Kylin 社區也有一些用戶遇到同樣類型的元數據更新混亂的場景,但是總體概率不高,許多都和自身的硬件以及機器情況有關。如果公司的基礎運維能力和監控體系足夠完善,其實是完全可以避免的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來簡單分享一些有贊踩過的坑,其中比較有代表性的就是全局字典的構建錯誤。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"踩坑:全局字典構建錯誤"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"背景"},{"type":"text","text":": 由於全局字典亂序,Merge Segment 時,合併字典出錯。原因在於爲了避免字典樹某個節點過大,會對節點進行 Split。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/45\/be\/45f616d8ae63eac63736587e96b229be.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根本原因如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Split 的兩個節點產生兩個字典值"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Split 超過一個字節的 UTF-8 的字符串會導致 Split 後的字串無序,歸併排序失敗"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"詳情請訪問:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/issues.apache.org\/jira\/browse\/KYLIN-4810"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kylin 4.0 在有讚的計劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後分享 Kylin 4.0 在有讚的計劃,目前主要分爲五個階段:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/84\/2c\/8499f6fc94679d5884a2e5a053f8292c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一階段就是 "},{"type":"text","marks":[{"type":"strong"}],"text":"原理調研和可用性測試"},{"type":"text","text":" ,這個階段歷時較久,首先有贊調研了 Kylin 4 的構建和查詢原理,當然 Kylin 4 是基於 Spark,有一定的學習成本;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二階段是 "},{"type":"text","marks":[{"type":"strong"}],"text":"語法兼容性測試"},{"type":"text","text":" 。有贊基於標準的壓測,基於 YCSB 去進行修改,來支持在 Kylin 上去跑,這個過程中,也發現了一些語法兼容性問題和其他 BUG,例如 PreparedStatement 在新版本不支持,還有分頁查詢場景常用到的 Limit\/Offset 也不支持等,這些問題我們陸續修復了;第二階段完成之後,我們團隊認爲 Kylin 4 基本上是一個完備的,可用的,具備灰度上生產資格的版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三階段開始着手去 "},{"type":"text","marks":[{"type":"strong"}],"text":"遷移元數據"},{"type":"text","text":" ,準備把目前機器上的元數據遷移到 Kylin 4,由於是新版本,我們做了一些比較保守的方案,把線上所有的 Kylin 2 、Kylin 3 的流量錄製下來,再回放到 Kylin 4。在這個過程中,我們也發現小查詢的性能實際上是不夠優化的,和 Kylin 2、Kylin 3 還是有一定差距的,有贊團隊也花費了很多時間和精力去優化小查詢, "},{"type":"text","marks":[{"type":"strong"}],"text":"目前基本做到了和 Kylin on HBase 相當的性能"},{"type":"text","text":" ,不過還存在可優化的空間。如果優化完成,有贊團隊會再去分享如何讓 Kylin 4 的性能接近 Kylin on HBase,此處說的小查詢是指命中 HBase Cuboid 這種類型的小\/點查詢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四階段有贊會把線上所有的 300+ Cube 構建到新的集羣,去把線上的流量做一個 "},{"type":"text","marks":[{"type":"strong"}],"text":"雙跑"},{"type":"text","text":" 的方式去回放到一個新的集羣,"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第五個階段就是 "},{"type":"text","marks":[{"type":"strong"}],"text":"逐步的去遷流量"},{"type":"text","text":" ,逐步的去下線舊版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在這裏分享一下,爲什麼有贊目前會選擇 Kylin 4 呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"其實有一個很重要的原因,有讚的大商家越來越多,場景也越來越複雜,對大查詢的性能 Kylin on HBase 已經無法滿足,並且 Kylin 4 在構建這部分也換了一個算法,性能相較舊版本也有很大的提升,選擇使用 Kylin 4 其實就是有贊現階段業務發展的需要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"},{"type":"text","text":":"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鄭生俊,有贊數據中臺-數據基礎平臺負責人,主要負責有贊離線計算、實時計算、OLAP 和在線存儲。Kylin 社區活躍貢獻者,推動了 Kylin 4.0 在有讚的實踐與優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文轉載自公衆號ApacheKylin(ID:apachekylin)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接"},{"type":"text","text":":"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653082315&idx=1&sn=e5433fc351433b5ae6e3601ae0a666f4&chksm=80a4ac3ab7d3252c7519e5459f3152902113332c3ce1b8c9c1396e5944f590abd79886961e17&token=1020922772&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"Apache Kylin 在有讚的高性能運維實踐"}]}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章