Apache Kylin 在貝殼找房指標體系的應用

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"引言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"“貝殼找房在做的就是推動居住服務這個傳統行業進行深度的互聯網化，在這個過程中數據是非常重要的支撐。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"right","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"——張如松，貝殼找房數據平臺高級工程師"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近日，Apache Kylin 5 週年在線慶典順利結束，來自貝殼找房的張如松老師爲大家介紹了 Apache Kylin 在在貝殼找房指標體系的應用，包含 Kylin 升級歷程以及對 Kylin 4.0 版本的展望。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下是張如松老師的現場分享實錄。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 在貝殼找房的發展歷程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"貝殼找房是以技術驅動的品質居住服務平臺，聚合和賦能全行業的優質服務者，打造開放的品質居住服務生態，致力於爲3億家庭提供包括二手房、新房、租賃、裝修和社區服務等全方位的居住服務，涉及到了居住服務的方方面面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"貝殼找房有 4+ 年的 Kylin 使用經驗。從 2016 年下半年開始，當時有一個 Hive + MySQL 的平臺，內部代號是地動儀，用來解決業務線的多維分析需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/8f\/6a\/8fc5e22d3374d64a1d05827fd35c216a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Kylin 在貝殼找房中的架構體系"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖展現了平臺數據的流轉過程，負責數倉的同學會將數據進行初步的預聚合，再通過關係數據庫來提供查詢。 "},{"type":"text","marks":[{"type":"strong"}],"text":"但是隨着數據量的快速增長，查詢響應的時間變得越來越長，底層數據庫的運維壓力也越來越大。"},{"type":"text","text":" 爲了解決這些問題，同時爲了支撐公司的指標體系建設，需要一個既能夠支持大規模數據計算，也可以對查詢作出快速響應的引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"經過調研，從可支持海量數據計算、亞秒級查詢響應、支持標準SQL、以及可維護性，涉及的技術棧以及社區活躍度上，Kylin 都符合作爲數據引擎來支撐企業指標體系建設的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2017 年 3 月，Kylin 1.6 版本上線"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着指標平臺的上線，Kylin 開始對外提供服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2017 年底，貝殼已經累計創建了 300 + Cube，每天有 20 多萬的查詢量"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/6f\/b5\/6fe5f099eea655ffaacf753d8a5251b5.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018 年初，隨着指標在各業務線的推廣，有越來越多的數據產品開始接入 Kylin"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如像 Merlin、Turing 等數據產品，這些產品從PC端到手機端覆蓋的範圍非常廣泛，涉及到公司組織架構的各個層級，都有相應的數據需求。同時爲了保障重點數據的產出和查詢，我們又部署了一套集羣來給重點業務使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018 年底，貝殼一共有 2 套集羣，累計創建了 600+ Cube，每天的查詢量達到了 200 萬。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b2\/ee\/b29974db3yydefd466833154fdc280ee.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2019 年初，我們 Kylin Team 定下了兩個 KPI，在機制方面要保障重點數據在每天上午 9 點之前產出，在查詢上要達成 3 秒鐘內響應占比 99.7%，將 Kylin 升級到 3.1 版本，主要來做實時多維分析的應用。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/c4\/8b\/c46cc1735c4a775c480a85994dac2a8b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了達成這兩個目標，在計算方面我們把集羣從 1.6.0升級到 2.5.2，引入了 Spark 組件，將重點 Cube 構建的方式從 MR 改爲了 Spark。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"上圖是調優前後的對比，重點 Cube 的平均構建時間從 70 分鐘降到了 43 分鐘，近 40% 左右的提升；在查詢方面也通過一系列的優化，在 12 月就達成了 3 秒內佔比 99.7% 的目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是當時每天的統計數據，到 19 年底貝殼還是兩套集羣，版本是 2.5.2，累計 700+ Cubes，每天的查詢量超過了 1000 萬。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4c\/7d\/4cfbfayy70a38ac0982445ca3de0177d.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年初，Kylin 升級到 3.1.0，引入了 Flink 組件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是公司的一級指標使用 Flink 組件前後花費時間的對比，可以看到提升比較明顯，截止到 2020 年底，貝殼有兩個3.1的集羣，累計 800+ Cubes，每天的查詢量最高超過了 2300 萬。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/11\/ff\/112def38315ab1b10789f7cd1d9a48ff.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"使用 Flink 前後的構建時長對比"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總的來說，這幾年貝殼找房圍繞 Kylin 主要在做平臺化的建設。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖右邊最下面是 Kylin 集羣，我們把集羣的節點分成了 3 種角色，一臺 Master 節點，負責接受提交的構建任務和提供元數據查詢服務。Master 節點既不參與查詢也不參與構建。有多臺構建器（Job節點）和查詢機器（Query節點）提供服務，這是在集羣節點上的劃分。集羣本身是不對外開放的，通過上層平臺提供服務。在平臺側主要是圍繞 API、任務、查詢、元數據做了一些工作。我們封裝了 Kylin 的 API，對創建的 Cube 的流程進行了簡化，同時對接了公司的權限體系來對模型進行權限控制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/c5\/a6\/c5yycffde68a4475abfb97bd894943a6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在任務管理上，由平臺控制任務的提交，包括任務的優先級，任務的運行數，還有對任務狀態的監控和異常數據的報警等。在查詢上，包括對 Cube 所在集羣的路由，以及對查詢的實時監控和分析。在元數據管理上，我們對 Cube 進行了生命週期管理，當符合規則的時候，會啓動 Cube 下線的流程。元數據管理還包括對 Cube 在不同集羣之間的遷移和集羣的版本控制、配置管理等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖展示的是平臺裏非常有意思的一個功能，叫做 Cube 查詢分析，每個小時都會分析一次Kylin的查詢日誌，統計出這些 Cube 被查詢了多少次，有哪些產品使用 Cube 的數據，上面這個圖就是一個 Cube 被不同產品查詢次數的佔比，可以看到這個 Cube 有 7 個產品在用，下面這個圖是 Cube 的響應時間在不同範圍內的查詢次數和佔比，可以看到這個 Cube 被查詢了69萬次，3秒內佔比達到了99.99%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/40\/c6\/40a9ca1b4f5910e654686156901c96c6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們會解析 Cube 查詢解析的每一條 SQL，拿到 SQL 用到的維度組合以及對應的響應時間，下圖包括三個方面內容："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cube 被使用最多的維度組合排行"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cube 查詢慢的組合排行"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近 30 天都沒有用到過的維度"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a9\/03\/a9371e0e35d16db104fe2615yy636203.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這些數據可以讓 Kylin 的用戶更好的瞭解數據的使用情況，也可以根據這些信息做一些針對性的優化，比如在構建和查詢方面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 在貝殼能高效的運用離不開內部同學的貢獻。下圖是這幾年貝殼找房的同學貢獻到社區的一些記錄，先後有4位同學向 Kylin 貢獻代碼，涉及任務調度、Web頁面、構建和查詢的優化等等多個方面，覆蓋了從1.6到3.1的各個版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/34\/80\/34bec88fcccca067fa12764029e5ea80.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"貝殼找房開發人員對 Kylin 的貢獻"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kylin 在貝殼找房指標體系建設過程中的作用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"貝殼找房是在 2016 年下半年開始規劃指標體系的建設，爲了明確指標的定義，統計數據的口徑，提高數據的共享性和安全性，同時規劃了指標平臺來承載指標體系的建設，並且使用 Kylin 來作爲指標體系的數據引擎，來提供數據服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是以 Kylin 爲基礎的指標平臺的架構，通過對數倉數據的建模計算，提供給指標平臺使用，指標平臺以 API 的方式對外提供服務，API 的基礎是指標，業務方定義的 API 可以包含一個或多個指標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/65\/3d\/65d6c9cfd1bc90f5f7d5b0f61560d13d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是基於 Kylin 的指標計算和使用的流程，首先數倉的同學會根據業務過程進行建模，從源數據有一個 ETL 的過程，最後會在 OLAP 層產生一張事實表，接着會在 Kylin 上關聯維表創建模型和 Cube，創建完 Cube 後會自動在調度系統生成一個依賴事實表和維表的任務，接着會在指標平臺定義指標，配置一下計算方式，支持的維度等信息，創建完指標之後就可以在 API 配置使用，調度系統會根據任務依賴來觸發 Cube 的構建，數據構建完之後各種數據產品就可以通過 API 來使用這些數據，這是基於 Kylin 的指標創建和使用的流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/97\/23\/97124a0b8dd651fe0d1db15e1cd07923.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來爲大家介紹兩個指標的例子，兩種不同的計算方式，一個是SUM類型，一個是COUNT DISTINCT類型的精確去重。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在貝殼找房指標體系裏面，精確去重是非常強的需求，尤其是一些涉及到業績類的指標，比如經紀人的帶看量，精確去重也是 Kylin 的優勢之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f2\/42\/f25602405ed3d88cf07cf3262f2e1a42.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖左右兩邊是手機端的產品，中間是PC端的報表產品，這幾款產品都是通過固定的維度組合來獲取相應的指標數據，只需要篩選不同的過濾條件就可以快速獲取報表。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一種場景就是可以隨意進行維度組合的自助分析場景，做一些探索性的嘗試，下圖是我們公司自研的Odin可視化平臺，圖左側兩個紅框分別是維度和指標，用戶可以隨意選擇他想要的維度和需要的指標，配置篩選條件，右側的圖是根據用戶的選擇，實時查詢Kylin生成的圖表，當他確定要使用這些維度和指標之後，就可以把當前的配置保存成固定的報表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/80\/87\/80fbdd1a90b998786a3be0e240f8ef87.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不管是固定報表還是自助分析，底層的查詢流程是一樣的，下圖左側框是業務方發起指標調用的形式，裏面的字段是他需要的維度，同時要指定時間範圍和過濾條件，發起一次API的調用，中間的框是指標平臺，接受API的請求，將API的參數轉化爲標準SQL，然後提交給Kylin集羣執行查詢，查詢完了之後會將查詢結果返回給指標平臺，指標平臺將數據封裝成固定的格式返回給業務方。這就是貝殼找房各種數據產品使用Kylin的底層查詢流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b1\/cb\/b16f32a1dd777223d084ffdde28688cb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖展示了目前 Kylin 在貝殼找房的一個使用情況，因爲對接了公司的指標體系，所以 Kylin 的使用覆蓋了所有的業務線， "},{"type":"text","marks":[{"type":"strong"}],"text":"爲超過 30 多個數據產品提供查詢服務，支撐了10000+ 指標的計算需求，每天的查詢量最高超過 2300萬，我們承諾的查詢響應時間是 3 秒內佔比是99.7%，目前來說 Kylin 都能很好的完成這些目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/53\/c5\/536d90f00755f1cac445bd5ba20870c5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"對 Kylin 未來發展的展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對 Kylin 的展望主要是針對 Kylin 4.0，貝殼在 9 月份的時候也做了一次簡單的測試，總體來說非常期待 4.0 GA 版本的發佈。希望在建模的流程上能夠更簡化更靈活，比如支持 Schema 的動態更新。在當前的情況下，只要涉及到 Cube 的改動就比較繁瑣，希望 Kylin 4.0 能改變這種情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於 Local segment cache，4.0 裏面生成的文件是存儲在集羣上，每次查詢都需要實時去讀取集羣上的文件，同時對集羣的資源和性能依賴比較大，可以考慮一下引入爲 HDFS 提速的組件，比如 Alluxio，可以將 Segment 的文件緩存到本地來提升查詢性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於多租戶，希望在查詢層面做到對多租戶的支持，來避免不同業務之間互相的影響，因爲貝殼現在的業務方還是比較多，互相影響的情況也會發生。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於 Kubernetes，現在的機器數和實例數也比較多，運維的成本比較高，後續會嘗試把 Kylin 部署到 Kubernetes 來降低維護成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"張如松，貝殼找房數據平臺高級工程師，負責 OLAP 引擎的開發和維護以及運維指標體系的建設。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文轉載自公衆號apachekylin（ID：ApacheKylin）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653082075&idx=1&sn=490afaf5b50be9bc170c4ad0cf34f224&chksm=80a4af2ab7d3263c0e83b29a2310b3371dc23313fc1fbf4611101d578e5130ab3b0030106284&token=1845978438&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"Apache Kylin 在貝殼找房指標體系的應用"}]}]}]}