{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 5 Year Milestone"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2015年底，Kylin 正式從 Apache 軟件基金會畢業，成爲第一個來自中國的Apache 頂級開源項目。在過去的5 年裏，Kylin 社區不斷髮展壯大，有了非常活躍的技術社區，協同開發，推動着 Kylin 成長爲一款高性能的大數據分析型數據倉庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ce\/72\/ce4ca6ec56dc80c7875230029d427c72.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Kylin 過往 5年裏主要的發展里程碑"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Kylin 技術發展概覽"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2016年初，Kylin 1.5 版本支持 Plug-in 架構"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲未來開發出更多的數據源、構建引擎以及存儲引擎打下了堅實基礎。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2016年底，Kylin 1.6 版本支持 Kafka 實時數據源"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過微批次的方式加載數據，大大降低了數據加載的延遲，從過去的天級或若干小時級變成了分鐘級。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2017年4月，Kylin 2.0 發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正式支持 Spark 作爲構建引擎，以及增加了對雪花模型的支持。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018年3月，eBay 團隊貢獻了 Cube Planner 和Dashboard"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這讓 Kylin Cube 優化比過去方便了很多，用戶可以對數據進行採樣和分析，讓算法決定哪些Cuboid 需要計算，再結合 Dashboard 中收集到的查詢歷史，可以讓 Cube 進一步的瘦身優化，在此基礎上性能和存儲都得到大大的改善。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018年9月，Kylin 2.5 版本支持了 Hadoop 3.0 以及 HBase 2.0"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以進一步從分鐘級的延遲降低到秒級。Kylin 也從此能同時支持歷史查詢、準實時查詢以及實時查詢。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年中，Kylin 3.1 版本發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持了 Flink 作爲計算引擎，至此 MapReduce、 Spark 以及 Flink 都與 Kylin 實現對接，用戶可以根據他們的喜好選擇合適的引擎做 Cube 計算。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年9月，Kylin 4.0 Alpha 版本正式發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全新的構建引擎和查詢引擎極大地提升構建性能和查詢性能，解決了查詢單點問題等痛點；去除了 HBase 依賴，很大程度地解決了 Kylin 的難運維問題，也使得 Kylin 的計算和存儲分離變爲可能。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）Kylin 社區成長"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在過去的 5 年中，Kylin 俘獲了非常多的用戶，在數據分析及報表展示中廣泛得應用，用戶羣不僅包括 eBay、Yahoo 日本、美團、網易等這些互聯網廠商，也包括了一些製造業廠商，如小米、華爲、VIVO 等；此外也有一批海外用戶，如 VISA、CISCO、迪卡儂、沃達豐等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/71\/3c\/712532d07fd8c92bd4959ba49380103c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 的社區也在這 5 年中不斷的發展和壯大，2015 年只有 16 個初始的開發人員，當時 Kylin 發佈了 5 個版本，從那之後 Kylin 社區在穩步的發展，截至到 2020 年 12 月，共有 44 個 Committer，包含 24 個 PMC Members，除此之外還有超過180+ 位 Code Contributors。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從 2015 年到 2020 年，Kylin 共發佈了 36 個 Apache Release，平均算下來每年會發布 6-7 個版本。另外，社區的活躍度從 Jira Issue 以及 GitHub Star 上也可以得到一個概覽。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ed\/8b\/ed11ed103691cf991dc043a229ff028b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"可以從這個曲線上看到 JIRA issue 一直在增長，說明每年有許多用戶在 Kylin 社區活躍，GitHub Star 數也同樣是呈持續增長態勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"最新版本：Apache Kylin 4.0"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Kylin 4.0 開發節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先從時間上來看，Kylin 4.0 Beta 版本最快將在本月底或 2021 年初的時候發佈，GA 版本將在 2021 年中旬正式發佈。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/fc\/e3\/fcc2f56f70c481520806d7eafac7ffe3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 4.0 改革的一大目標就是將 HBase 存儲替換成 Parquet 存儲。這一目標考慮的是 HBase 的功能比較複雜，對 Kylin 運維帶來非常大的挑戰，隨之而來的是企業維護的成本及擴容成本高。未來我們希望 Kylin 能發展成存儲和計算分離的架構，通過使用輕量級的列式存儲來幫助用戶更容易運維。Cube 的計算引擎及查詢引擎都會替換成 Spark，整個這個體系架構也是基於 Kylin 的可插拔架構來實現的，但是因爲存儲是一個非常基礎的模塊，所以它對上層和對下層都有不少的改動。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，4.0 Alpha 版本已經在 Kylin 官網上，感興趣的小夥伴可以登錄官網下載預覽，基礎功能包括 Cube 的計算和查詢相對來說比較完整。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"預計在本月底或 2021 年初，4.0 Beta版本將正式發佈，目前正在開發一些高級功能以及它們跟新的存儲及查詢引擎的對接，包括 Cube Planner、Dashboard，以及對存儲和查詢引擎的性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"待明年（2021年）中旬，4.0 GA 的時候還會增加對 Kafka 的數據源的對接，以及實現 Spark 3.0 的升級，其它一些正在規劃中的高級功能也會在GA 版本中跟大家見面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）Kylin 4.0 性能概覽"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來會主要展示 Kylin 4.0 版本與 3.0 版本在離線的 Cube 計算性能以及在線 SQL 查詢性能對比情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"離線的 Cube 計算性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d4\/c2\/d48e5807726eccdf09300677683f9fc2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先來看下離線 Cube 計算性能 ，"},{"type":"text","marks":[{"type":"strong"}],"text":"左圖顯示了構建時間的對比，使用的是 3.0 的 MapReduce 引擎和 4.0 的 Spark引擎做對比，可以看到 4.0 的構建時間相比 3.0 能減少 30%—50%，也就是說理想情況下 4.0 版本中 Cube 構建速度可以比 3.0 快一倍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從存儲方面來看，右邊圖顯示 "},{"type":"text","marks":[{"type":"strong"}],"text":"4.0 版本的 Cube 大小比 3.0 能減少差不多 40%"},{"type":"text","text":"，因爲 3.0 版本中 Cube 數據會存儲兩份：一份是在 HDFS 上的中間結果用於未來的 segment 合併，還有一份是在 HBase 中用於查詢；但 4.0版本中只需用一份 Parquet 數據就可以來做合併以及查詢。所以4.0的存儲大小大概是 3.0 的 1\/3。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"在線 SQL 查詢性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/02\/c5\/02c1e1acc69a2141d51c8bd9712bc8c5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中用了典型的兩個 Benchmark 數據來做對比，左邊是 SSB 數據集，它相對來說是一種比較簡單的分析場景。從左圖看到大多數簡單場景的查詢下，4.0 的性能和 3.0 是比較接近，但會略微慢一點，3.0 版本的查詢時間大概在 0.5 秒左右，4.0 比 3.0 略微慢了 0.2、0.3秒左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右圖是在複雜的 TPCH 模型下，查詢會比較複雜，可以看到 3.0 上很多查詢響應差不多是在 10 秒或者 20-30 秒之間，但Kylin 4.0 藉助 Spark 的分佈式計算和分佈式聚合，使得 Kylin 的 query 節點的負載進一步被分散（也不再有字典的加載和解碼），使得查詢性能取得進一步的提升，圖上明顯可以看到在一些慢的查詢上，Kylin 4.0 相比於 3.0 有差不多 10 倍的性能提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總而言之，Kylin 4.0 的性能比 3.0 在小而簡單的查詢下基本持平，但是在複雜且慢查詢下會有非常大幅度的提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 未來展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Cloud-Native"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲計算的普及，過去可能很多原本只能在 Hadoop 上做的事情有了新的選擇。但云隨之而來給企業的應用，特別是給 IT 架構帶來很大的變化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b1\/20\/b1825fb5c615e3003ca2b61519a48320.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在越來越多的企業正在擁抱雲原生（Cloud-Native）架構，目的是讓企業能更適應於在雲上部署。對於 Kylin 而言，Kylin 也會做計算和存儲分離，這樣就能讓計算資源和存儲資源分別獨立的彈性伸縮，從而實現資源的更高效利用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，爲了能實現應用之間更好的隔離，促進應用往更高性能、更高穩定性發展，我們也希望對 Kylin 整體架構的調整能更好得在雲上部署，來使用雲原生的存儲計算的框架，以及監控運維的框架。在這方面可以看到最火的就是以 Kubernetes 爲代表的容器編排技術，以及雲上以 S3 所代表的新的分佈式對象存儲技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）實時分析（Real-time）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"過去大數據處理大部分是使用批處理來對數據做加工、清洗和聚合；因爲批處理相對來說比較簡單，以及高吞吐的經濟效率比較高。但是批處理最大問題在於數據的延遲比較久，通常在 T+1，就是1天或者是若干個小時之後才能看到這個數據分析的結果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天越來越多的企業對數據時效性提出了更高要求，業務希望數據的處理時間能夠降低在分鐘級甚至是秒級，這些年可以看到實時計算的技術越來越火，包括Spark Structed Streaming、Flink 以及一些其它的流計算框架。對 OLAP 來說，未來我們希望 Kylin 一方面能夠繼續支持批量的數據加載，另一方面也能支持流數據的處理，實現流批一體化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/cc\/5a\/cca2a11c20bbefe8bbc8461fe0afac5a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家可以從 Kylin3.0 版本就能看到，我們統一了流計算和批計算，未來用戶只用運維一套 OLAP 平臺就可以服務不同的場景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3）人工智能與機器學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人工智能就是讓機器學會人的思考方式，通過模型的訓練讓機器可以自動的做出判斷決策，從而減少人工的投入，提高人類的工作效率。如何才能讓 AI 更加的智能，模型更加的準確，成熟度更高？在這背後就需要更多的數據進行訓練，而這就是大數據與機器學習所結合的價值所在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a9\/ca\/a94ddf53a0d020d5d21e7bf4b8d0d0ca.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 過去主要服務的場景是在 BI 領域，也就是說把數據採集出來，通過對接 BI 賦能分析人員來看到數據中發生了哪些本質的變化。在未來我們希望 Kylin 可以通過這些數據來賦能於 AI，能直接從數據中挖掘出來價值告訴人類，而不是讓人通過 BI 來獲取這些信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們總結未來對 Kylin 的期待。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/86\/94\/865195bf15e8b3f7b56d3cc2265d1394.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們希望 Kylin 能成長爲一個雲原生的，可以支持批處理以及流處理的分析型數據倉庫。一方面能服務於 BI，另一方面可以實現對 AI 的支持，相比於其它技術和引擎來說，它在高性能和高併發上具有明顯的優勢。未來它的底層架構可以直接運行在輕量級的分佈式計算框架上，也可以直接部署在容器上，可以對接多種數據源，包括文件集合，實時流，傳統的 RDBMS，或是現在很熱門的數據湖。此外未來也希望 Kylin 的部署、運維、監控、擴容、縮容都會變得更加容易，最終也可以讓用戶的總體成本比以前有一個大幅降低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/80\/75\/8055f6e6cbdcfd0399d3510822645a75.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到 Kylin 5.0 的時候，我們希望它能夠基於 Cloud-Native 架構再次統一流批 OLAP 分析，並實現對 Machine Learning 及 AI 的支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"史少鋒，Kyligence 合夥人 & 首席軟件架構師，Apache Kylin 核心開發者和項目管理委員會主席 (PMC Chair)，專注於大數據分析和雲計算技術。曾任eBay全球分析基礎架構部大數據高級工程師，IBM雲計算軟件架構師。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文轉載自公衆號apachekylin（ID：ApacheKylin）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653081989&idx=1&sn=7bf3941ef5f63eebf3f800cf92299862&chksm=80a4af74b7d32662914b25b8197633a8d708637eebb5d2a39fb88301583d80a5c676d6fa5434&token=1340822333&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"Kylin 5 年的成長與未來規劃"}]}]}]}
Kylin 5 年的成長與未來規劃
{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 5 Year Milestone"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2015年底，Kylin 正式從 Apache 軟件基金會畢業，成爲第一個來自中國的Apache 頂級開源項目。在過去的5 年裏，Kylin 社區不斷髮展壯大，有了非常活躍的技術社區，協同開發，推動着 Kylin 成長爲一款高性能的大數據分析型數據倉庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ce\/72\/ce4ca6ec56dc80c7875230029d427c72.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Kylin 過往 5年裏主要的發展里程碑"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Kylin 技術發展概覽"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2016年初，Kylin 1.5 版本支持 Plug-in 架構"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲未來開發出更多的數據源、構建引擎以及存儲引擎打下了堅實基礎。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2016年底，Kylin 1.6 版本支持 Kafka 實時數據源"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過微批次的方式加載數據，大大降低了數據加載的延遲，從過去的天級或若干小時級變成了分鐘級。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2017年4月，Kylin 2.0 發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正式支持 Spark 作爲構建引擎，以及增加了對雪花模型的支持。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018年3月，eBay 團隊貢獻了 Cube Planner 和Dashboard"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這讓 Kylin Cube 優化比過去方便了很多，用戶可以對數據進行採樣和分析，讓算法決定哪些Cuboid 需要計算，再結合 Dashboard 中收集到的查詢歷史，可以讓 Cube 進一步的瘦身優化，在此基礎上性能和存儲都得到大大的改善。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018年9月，Kylin 2.5 版本支持了 Hadoop 3.0 以及 HBase 2.0"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以進一步從分鐘級的延遲降低到秒級。Kylin 也從此能同時支持歷史查詢、準實時查詢以及實時查詢。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年中，Kylin 3.1 版本發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持了 Flink 作爲計算引擎，至此 MapReduce、 Spark 以及 Flink 都與 Kylin 實現對接，用戶可以根據他們的喜好選擇合適的引擎做 Cube 計算。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年9月，Kylin 4.0 Alpha 版本正式發佈"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全新的構建引擎和查詢引擎極大地提升構建性能和查詢性能，解決了查詢單點問題等痛點；去除了 HBase 依賴，很大程度地解決了 Kylin 的難運維問題，也使得 Kylin 的計算和存儲分離變爲可能。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）Kylin 社區成長"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在過去的 5 年中，Kylin 俘獲了非常多的用戶，在數據分析及報表展示中廣泛得應用，用戶羣不僅包括 eBay、Yahoo 日本、美團、網易等這些互聯網廠商，也包括了一些製造業廠商，如小米、華爲、VIVO 等；此外也有一批海外用戶，如 VISA、CISCO、迪卡儂、沃達豐等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/71\/3c\/712532d07fd8c92bd4959ba49380103c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 的社區也在這 5 年中不斷的發展和壯大，2015 年只有 16 個初始的開發人員，當時 Kylin 發佈了 5 個版本，從那之後 Kylin 社區在穩步的發展，截至到 2020 年 12 月，共有 44 個 Committer，包含 24 個 PMC Members，除此之外還有超過180+ 位 Code Contributors。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從 2015 年到 2020 年，Kylin 共發佈了 36 個 Apache Release，平均算下來每年會發布 6-7 個版本。另外，社區的活躍度從 Jira Issue 以及 GitHub Star 上也可以得到一個概覽。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ed\/8b\/ed11ed103691cf991dc043a229ff028b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"可以從這個曲線上看到 JIRA issue 一直在增長，說明每年有許多用戶在 Kylin 社區活躍，GitHub Star 數也同樣是呈持續增長態勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"最新版本：Apache Kylin 4.0"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Kylin 4.0 開發節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先從時間上來看，Kylin 4.0 Beta 版本最快將在本月底或 2021 年初的時候發佈，GA 版本將在 2021 年中旬正式發佈。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/fc\/e3\/fcc2f56f70c481520806d7eafac7ffe3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 4.0 改革的一大目標就是將 HBase 存儲替換成 Parquet 存儲。這一目標考慮的是 HBase 的功能比較複雜，對 Kylin 運維帶來非常大的挑戰，隨之而來的是企業維護的成本及擴容成本高。未來我們希望 Kylin 能發展成存儲和計算分離的架構，通過使用輕量級的列式存儲來幫助用戶更容易運維。Cube 的計算引擎及查詢引擎都會替換成 Spark，整個這個體系架構也是基於 Kylin 的可插拔架構來實現的，但是因爲存儲是一個非常基礎的模塊，所以它對上層和對下層都有不少的改動。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，4.0 Alpha 版本已經在 Kylin 官網上，感興趣的小夥伴可以登錄官網下載預覽，基礎功能包括 Cube 的計算和查詢相對來說比較完整。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"預計在本月底或 2021 年初，4.0 Beta版本將正式發佈，目前正在開發一些高級功能以及它們跟新的存儲及查詢引擎的對接，包括 Cube Planner、Dashboard，以及對存儲和查詢引擎的性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"待明年（2021年）中旬，4.0 GA 的時候還會增加對 Kafka 的數據源的對接，以及實現 Spark 3.0 的升級，其它一些正在規劃中的高級功能也會在GA 版本中跟大家見面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）Kylin 4.0 性能概覽"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來會主要展示 Kylin 4.0 版本與 3.0 版本在離線的 Cube 計算性能以及在線 SQL 查詢性能對比情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"離線的 Cube 計算性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d4\/c2\/d48e5807726eccdf09300677683f9fc2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先來看下離線 Cube 計算性能 ，"},{"type":"text","marks":[{"type":"strong"}],"text":"左圖顯示了構建時間的對比，使用的是 3.0 的 MapReduce 引擎和 4.0 的 Spark引擎做對比，可以看到 4.0 的構建時間相比 3.0 能減少 30%—50%，也就是說理想情況下 4.0 版本中 Cube 構建速度可以比 3.0 快一倍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從存儲方面來看，右邊圖顯示 "},{"type":"text","marks":[{"type":"strong"}],"text":"4.0 版本的 Cube 大小比 3.0 能減少差不多 40%"},{"type":"text","text":"，因爲 3.0 版本中 Cube 數據會存儲兩份：一份是在 HDFS 上的中間結果用於未來的 segment 合併，還有一份是在 HBase 中用於查詢；但 4.0版本中只需用一份 Parquet 數據就可以來做合併以及查詢。所以4.0的存儲大小大概是 3.0 的 1\/3。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"在線 SQL 查詢性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/02\/c5\/02c1e1acc69a2141d51c8bd9712bc8c5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中用了典型的兩個 Benchmark 數據來做對比，左邊是 SSB 數據集，它相對來說是一種比較簡單的分析場景。從左圖看到大多數簡單場景的查詢下，4.0 的性能和 3.0 是比較接近，但會略微慢一點，3.0 版本的查詢時間大概在 0.5 秒左右，4.0 比 3.0 略微慢了 0.2、0.3秒左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右圖是在複雜的 TPCH 模型下，查詢會比較複雜，可以看到 3.0 上很多查詢響應差不多是在 10 秒或者 20-30 秒之間，但Kylin 4.0 藉助 Spark 的分佈式計算和分佈式聚合，使得 Kylin 的 query 節點的負載進一步被分散（也不再有字典的加載和解碼），使得查詢性能取得進一步的提升，圖上明顯可以看到在一些慢的查詢上，Kylin 4.0 相比於 3.0 有差不多 10 倍的性能提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總而言之，Kylin 4.0 的性能比 3.0 在小而簡單的查詢下基本持平，但是在複雜且慢查詢下會有非常大幅度的提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 未來展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1）Cloud-Native"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲計算的普及，過去可能很多原本只能在 Hadoop 上做的事情有了新的選擇。但云隨之而來給企業的應用，特別是給 IT 架構帶來很大的變化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b1\/20\/b1825fb5c615e3003ca2b61519a48320.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在越來越多的企業正在擁抱雲原生（Cloud-Native）架構，目的是讓企業能更適應於在雲上部署。對於 Kylin 而言，Kylin 也會做計算和存儲分離，這樣就能讓計算資源和存儲資源分別獨立的彈性伸縮，從而實現資源的更高效利用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，爲了能實現應用之間更好的隔離，促進應用往更高性能、更高穩定性發展，我們也希望對 Kylin 整體架構的調整能更好得在雲上部署，來使用雲原生的存儲計算的框架，以及監控運維的框架。在這方面可以看到最火的就是以 Kubernetes 爲代表的容器編排技術，以及雲上以 S3 所代表的新的分佈式對象存儲技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2）實時分析（Real-time）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"過去大數據處理大部分是使用批處理來對數據做加工、清洗和聚合；因爲批處理相對來說比較簡單，以及高吞吐的經濟效率比較高。但是批處理最大問題在於數據的延遲比較久，通常在 T+1，就是1天或者是若干個小時之後才能看到這個數據分析的結果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天越來越多的企業對數據時效性提出了更高要求，業務希望數據的處理時間能夠降低在分鐘級甚至是秒級，這些年可以看到實時計算的技術越來越火，包括Spark Structed Streaming、Flink 以及一些其它的流計算框架。對 OLAP 來說，未來我們希望 Kylin 一方面能夠繼續支持批量的數據加載，另一方面也能支持流數據的處理，實現流批一體化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/cc\/5a\/cca2a11c20bbefe8bbc8461fe0afac5a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家可以從 Kylin3.0 版本就能看到，我們統一了流計算和批計算，未來用戶只用運維一套 OLAP 平臺就可以服務不同的場景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3）人工智能與機器學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人工智能就是讓機器學會人的思考方式，通過模型的訓練讓機器可以自動的做出判斷決策，從而減少人工的投入，提高人類的工作效率。如何才能讓 AI 更加的智能，模型更加的準確，成熟度更高？在這背後就需要更多的數據進行訓練，而這就是大數據與機器學習所結合的價值所在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a9\/ca\/a94ddf53a0d020d5d21e7bf4b8d0d0ca.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 過去主要服務的場景是在 BI 領域，也就是說把數據採集出來，通過對接 BI 賦能分析人員來看到數據中發生了哪些本質的變化。在未來我們希望 Kylin 可以通過這些數據來賦能於 AI，能直接從數據中挖掘出來價值告訴人類，而不是讓人通過 BI 來獲取這些信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們總結未來對 Kylin 的期待。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/86\/94\/865195bf15e8b3f7b56d3cc2265d1394.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們希望 Kylin 能成長爲一個雲原生的，可以支持批處理以及流處理的分析型數據倉庫。一方面能服務於 BI，另一方面可以實現對 AI 的支持，相比於其它技術和引擎來說，它在高性能和高併發上具有明顯的優勢。未來它的底層架構可以直接運行在輕量級的分佈式計算框架上，也可以直接部署在容器上，可以對接多種數據源，包括文件集合，實時流，傳統的 RDBMS，或是現在很熱門的數據湖。此外未來也希望 Kylin 的部署、運維、監控、擴容、縮容都會變得更加容易，最終也可以讓用戶的總體成本比以前有一個大幅降低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/80\/75\/8055f6e6cbdcfd0399d3510822645a75.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到 Kylin 5.0 的時候，我們希望它能夠基於 Cloud-Native 架構再次統一流批 OLAP 分析，並實現對 Machine Learning 及 AI 的支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"史少鋒，Kyligence 合夥人 & 首席軟件架構師，Apache Kylin 核心開發者和項目管理委員會主席 (PMC Chair)，專注於大數據分析和雲計算技術。曾任eBay全球分析基礎架構部大數據高級工程師，IBM雲計算軟件架構師。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文轉載自公衆號apachekylin（ID：ApacheKylin）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653081989&idx=1&sn=7bf3941ef5f63eebf3f800cf92299862&chksm=80a4af74b7d32662914b25b8197633a8d708637eebb5d2a39fb88301583d80a5c676d6fa5434&token=1340822333&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"Kylin 5 年的成長與未來規劃"}]}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章