{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"10 月 30 日下午，2021 WeDataSphere 社區大會在深圳灣科技生態園創新廣場順利舉行。現場，開源界大咖、WeDataSphere 社區貢獻者和維護者，以及近兩百位開源愛好者，大家共聚一堂，交流技術，共論開源之道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/df81062844d48590f75c1a0a5cb05bd4.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，本次大會不僅邀請到開放原子開源基金會 TOC 主席堵俊平，而且包括 WeDataSphere 社區發起人邸帥、天翼雲大數據平臺技術專家王小剛、薩摩耶雲數據業務部總監易小華和四位微衆銀行大數據平臺工程師均到場。作爲開源大數據行業的觀察者和實踐者，他們紛紛發表了精彩的演講內容。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"堵俊平：Hadoop 沒死，開源大數據基於技術和潮流在不斷演進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"堵俊平是開源大數據行業的“老兵”，他開場先帶領大家回顧了開源大數據最近十幾年的發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 20 世紀 90 年代，隨着互聯網的快速發展，數據快速增長，但以前的技術手段無法解決新問題。後來，谷歌基於搜索引擎的多年實踐提出“三駕馬車”，即 GFS、MapReduce 和 BigTable。這“三駕馬車”給 Hadoop 創始人 Doug Cutting 很大的啓發，於是，他在 2006 年寫出第一個 Hadoop 引擎。這算是業界開源大數據的起點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fa\/faec81c4b26819ce0ec51c45c5c6bc97.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"開放原子開源基金會 TOC 主席堵俊平"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop 橫空出世後，被很多互聯網公司和科技公司紛紛採用，因爲大家的需求類似——怎麼以更低成本提升硬件利用效率，從而最大化地發揮數據價值。之後，Hadoop 在 2008 年從 Apache 畢業，這成爲標誌性事件。再過一年，第一款大數據雲服務在 AWS 上線，這意味着開源大數據走向雲時代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此後，Hadoop 生態加速發展，除了 Hadoop，又有了 Spark、Flink 等。 到 2.0 時代，Yarn 從 Hadoop 中分離， 它把資源管理跟上層的應用調度做了兩層分離，這樣，在 Hadoop 底層，Yarn 更像一個通用平臺，上面是更多的引擎，整個體系可以不斷演進。3.0 時代，Hadoop 開始與更多的技術相結合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"縱觀整個開源大數據的生態系統，既有新的大數據項目在不斷孵化，又有一些式微的開源項目。總體上，它根據技術和潮流的方向不斷演進，而非一成不變的靜態系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大體而言，這個領域的技術有以下趨勢：一是統一的 SQL，即 Unified SQL。二是批流的統一。生態趨勢上，開源大數據不斷衍生和迭代，Hadoop 也沒有死，但如果把 Hadoop 視爲開源大數據生態體系，那麼它仍在蓬勃發展，只是引擎沒人在用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9dda93631c2ec583b58f735c38af42e2.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了回顧開源大數據的發展歷程，堵俊平還分享了自己對開源的看法。開源項目有多個角色，包括開發者、用戶、廠商和基金會等，每個角色只要持續參與進來，從中找到價值，生態就會持續演進。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"邸帥：企業數據平臺的建設應該量體裁衣，因地制宜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲 WeDataSphere 社區發起人、微衆銀行大數據平臺團隊負責人，邸帥分享了 “WeDataSphere 大數據平臺套件的建設思路和開源歷程”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼要建數據平臺？邸帥認爲這是由於兩方面的需要：一是在機構或公司，它是商業化挑戰的需要；二是技術挑戰的需要。從容量、性能、效率和成本方面，企業要考慮以什麼樣的技術形式解決構建數據平臺的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ce\/cebeda64db1dadfb4a46fa36ee0df4f1.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"WeDataSphere 社區發起人 邸帥"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他表示，數據平臺對於企業，猶如血液循環系統之於人體。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼，企業如何建設數據平臺？借用《平臺革命》一書，平臺最核心的價值是促成很多最核心的交互，即“參與者 + 價值單元 + 過濾器 =>核心交互”。而最能體現平臺價值的能力是吸引、促進和匹配。在數據領域，問題在於怎樣提供更好的數據工具和服務，促使數據平臺完成吸引、促進和匹配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bb\/bb53c2c827c6345b6f272b41e04168f5.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論是鋼鐵俠的 AI、雲上數倉（比如 Snowflake）或 Gartner 十大數據和分析趨勢，在邸帥看來，每個公司或團隊真正需要的數據平臺應該要量體裁衣，因地制宜，根據自己的情況找到比較好的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以量體爲例，企業需要在數據管理能力層面、數據平臺能力層面和數據應用情況方面摸清現狀，從而確立自己的目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"邸帥表示，“我最想表達的是，如何建設好數據平臺，要從數據管理、數據平臺、數據應用三個層面摸清現狀，然後基於 WeDataSphere + 其他優秀開源項目提供的“布料”來“量體裁衣”。我們 WeDataSphere 的建設思路最核心的三點是：首先方向選擇是主力投入上層功能工具系統的建設和開發，底層引擎層更多是做補充，比如 bug 修復。然後兩個最核心的設計要點是“一站式”和“全連通”。“"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 WeDataSphere 建設思路上，他指出：數據平臺範圍龐大，團隊規模資源有限，要想做出特色和優勢，須從產品架構設計，和多團隊共建模式上進一步優化。借用微衆銀行副行長兼首席信息官馬智濤先生的一句話就是“小刀鋸大樹，必須靠方法。”具體而言，一是通過合理的構建功能工具集成開發框架和中間件層，獲得更極致的連通、解耦、易擴展、高度複用能力，簡化整體架構和調用關係，大幅降低新功能工具開發和平臺運維成本；二是基於“開源”吸引一切可以團結的力量，多方聯合共建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在社區建設層面，微衆銀行、天翼雲、仙翁科技、薩摩耶雲、Boss 直聘的團隊聯合開發了 Streamis。同時，今年，微衆銀行與合作伙伴還構建了兩個大的項目：Exchangis 1.0和 DataModelCenter。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 WeDataSphere 的未來，邸帥表示，首先做得更深，其次做得更廣。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"尹強：Apache Linkis 是站在四個巨人肩膀上的開源工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 WeDataSphere 中，計算中間件 Linkis 解決前臺各種工具、應用，和後臺各種計算存儲引擎間連接簡化和複用問題，是非常重要的一環。尹強是 Apache Linkis PPMC，他分享了 Linkis 的建設及開源思路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先說說大數據平臺面臨的計算治理問題。大數據平臺不斷髮展，底層有很多的計算存儲引擎，上層有很多的應用工具，包括批量計算、交互式計算、流式計算、數據分析工具、數據治理工具等。平臺組件越來越多，開發維護愈加困難。這樣的背景下，Linkis 誕生，主要解決上層應用工具和底層存儲引擎之間關於“連通、擴展、管控、編排、複用等‘計算治理‘問題”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/42f8302015fb722000f2381b9c277522.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據技術專家 尹強"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對此問題，目前主要有四個開源社區解決方案：一是 Apache Livy，通過提供 REST 服務，它能簡化用戶與 Spark 集羣的交互。同時，它聚焦於“連接”能力。通過 Job 或代碼片段，簡化 Spark 任務提交。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二是 Apache Zeppelin，它是基於 Web 的交互式數據分析 Notebook，聚焦於“擴展”能力。獨有的 Interpretor 架構，可快速對接新的大數據引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三是 Netflix Geine，它是分佈式作業編排引擎，聚焦於“管控”能力。強大的標籤體系，支持按需路由作業到不同的 Hadoop 集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"四是 openLooKeng，它是一款高性能數據虛擬化引擎，提供統一 SQL 接口，具備跨數據源 \/ 數據中心分析能力。聚焦於“編排”能力。通過編排 SQL，支持跨數據中心、跨雲的異構數據源查詢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在尹強看來，這四個解決方案各自特點非常突出，比如 Apache Livy 在連接上做得很好。但是，它們又有不足，因此企業級大數據平臺呼喚“既具備擴展又具備管控，還具有編排、連通、複用能力的中間件”。Linkis 是一款站在巨人肩膀上的開源工具，可以極大簡化大數據平臺的架構，降低開發和運維的複雜度。爲什麼這麼說？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linkis 構建解耦計算中間件層，具備連通、擴展、管控,、編排和複用能力。連通上，告別應用孤島，它可以打通用戶資源 & 運行時環境。擴展上，通過實現 Linkis EngineConnPlugin，完成新引擎適配，統一解決高併發、高可用、多租戶等問題。管控上，它具備基於標籤的多級精細化資源控制和回收能力，可以實現多級精細化參數化配置控制。編排上，它基於 Orchestrator 服務的雙活策略設計和混算策略設計。複用上，它大大降低上層應用 \/ 工具開發後臺代碼量，並基於 Linkis 快速高效打造數據平臺工具套件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，尹強談到了 Apache Linkis 的開源情況和未來規劃。從 2019 年 7 月，Linkis 第一個開源版本發佈，去年 12 月，社區主導的第一個版本發佈。此後，它不斷迭代和發展，於 2021 年 7 月進入 Apache 基金會孵化器，並在 9 月榮獲 2021 屆開源產業大會“OSCAR 尖峯開源項目和開源社區”獎項。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於未來，一方面是它在 Apache 基金會好好孵化；另一方面，優化社區運營，與生態相關頂級項目建立更緊密合作，同時，產品迭代加速。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"王小剛：WeDataSphere 開源組件在天翼雲大數據平臺產品中的應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲天翼雲大數據平臺團隊技術專家，王小剛分享了 WeDataSphere 開源組件在天翼雲大數據平臺產品中的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先談到爲什麼會與 WeDataSphere 結緣？他解釋：內因是電信集團進行“雲改數轉”戰略，天翼雲過去的集羣存在建設分散，整體資源複用率低，以及數據加工腳本管理混亂，故障頻發等問題。外因在於，CDH、HDP 合併，商業版軟件訂閱費用昂貴，技術綁定，同時，市場缺乏優質的同類開源產品。我們拿自己開刀花了一年多時間從 CDH 商業版升級到 Hadoop 最新版，但光去 CDH 還不夠，核心是用戶需要統一的大數據平臺。我們同時圍繞 Hadoop 最新版，擁抱開源、打造自主可控、技術領先、安全穩定的大數據平臺。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6a\/6a659f55c57aabdb2de6ea136182bc9c.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"天翼雲大數據平臺技術專家 王小剛"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之所以瞭解 WeDataSphere，主要在於 Linkis。“當時，看到它的時候感覺很新鮮，並且和我們的思路一致，它已經開源，平臺完善度高，社區也很活躍。它可封裝，可水平復制，一站式，非常開放。對我們來說，改造成本適中，運維和維護成本較低。所以，我們選了 Linkis“。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在王小剛看來，Linkis 有強大的入口能力，這正是他們所需的，用它來隔離底層複雜性，保證持續智能優化的可能性。其次，Linkis 奉行開放開源的策略，親切且先進的架構設計，有相對完善的文檔，這些對於開發者來說都是非常友好的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"並且，他們展示了基於 Linkis 持續在做的一些工作，比如更完善的容器化、更大規模的驗證、數據源管理、數據集成能力、實時計算能力等，以及後續希望實現跨集羣作業提交、SQL 路由、安全能力等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在具體實踐上，用他的話說：先“玩”起來，深度試用，與內部底座進行整合；然後，逐步完善，修復 bug；接着，針對遇到的一些問題進行較爲深度的二次開發；最後就是擁抱開源與社區共建，現在我們已經完全深度參與社區，和社區小夥伴們分工合作，共同研發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而說到用戶體驗側，王小剛提到，隨着大數據平臺的不斷普及，各層級的用戶也日益增多，市場主流產品存在一定同質化現象，而在這其中，往往使產品能夠脫穎而出的，就是用戶易用性的提升。天翼雲在這塊也做了很多嘗試，包括新手引導、產品即文檔（產品內置的智能用戶手冊）、換膚（Dark 模式）、界面優化設計等，都是以用戶體驗爲核心進行優化和打磨。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，新手引導主要解決用戶首次進入大數據平臺的操作問題，幫助用戶以輕量化的形式，快速上手常用的功能按鍵，熟悉產品的界面操作使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"產品即文檔是內置的彈出式幫助中心，主要協助用戶解決當前頁面 \/ 功能模塊的上手痛點，包括對於該模塊的功能介紹，典型步驟介紹，常見問題等。該功能試圖排除用戶在使用中碰到的 80% 以上的問題，切實給客戶提供使用幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Dark 模式是針對開發人員和大數據平臺使用者專門打造的護眼模式界面皮膚，基於原配色深度優化，符合開發者的使用和交互習慣，增加開發效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，隨着平臺已經擁有越來越多的試用和正式用戶，小剛提到，持續圍繞“客戶成功”這一核心理念來打造產品，是我們始終貫徹的。團隊堅持“用戶至上、開放共贏、坦誠清晰”，將整個團隊也打造成一個產品，給用戶帶來最好的體驗。作爲一隻充滿活力，開源開放的大數據平臺團隊，希望未來能繼續和社區的小夥伴一起把 DataSphere Studio 打造的更友好易用。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"王和平：用戶需要一站式交互體驗的開源數據分析工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲 DataSphere Studio 核心開發 ，王和平現場分享了《WeDataSphere 數據分析工場的建設和開源思路》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他首先介紹了開源數據分析工具的選型思路。在構建數據分析時，用戶面臨的痛點有哪些？王和平表示，一是數據從哪來、數據安全怎麼保證，二是數據怎麼分析，怎樣進行模型訓練，三是數據質量怎麼把控，四是分析出的結果如何彙報，五是數據調度怎麼做。最後，這麼多工具，如何給用戶帶來一站式的交互體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a2\/a2e41b01314d8531f1fefe9821030512.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 王和平"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於用戶面臨的挑戰，業界都有相應的工具去解決問題，比如數據分析工具執行交互式分析和模型訓練以及任務調度，還有數據交換工具、數據治理工具等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，業界有很多開源的數據交換工具，例如 DATAX、DataX Web、Dbus 和 Exchangis。在數據分析工具上，業界也有 Apache Zeppelin、Scriptis 和 Hue。同樣，業界的開源數據治理工具有 pydqc、Apache Atlas、Deequ、Data Hub、GriFFin 和 Qualitis。開源調度工具，業界有 Dolphin Scheduler、XXL、Apache Airflow 和 Schedulis。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然工具不少，但是對一個數據分析師來說，它可能並沒有真正解決問題。並且，問題還有缺乏標準規範，工程管理體系不統一，用戶權限不統一，UI 不統一；應用孤島，資源物料不互通，運行時上下文不互通。此外，重複造輪子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲解決這些問題，DataSphere Studio 誕生，它具備一站式、全連通、全流程、可插拔和強管控特點，可以覆蓋需求、設計、開發、測試到生產所有階段，一條工作流串通所有應用工具，上層新工具只需實現 AppConn 插件，便可快速打通其他工具。並且，基於工作空間的多租戶隔離，實現以租戶爲單位進行統一的權限管控、成員管理和協同開發等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"王和平說：“DataSphere Studio 有一個 AppConn，它去做連接，集成其他優秀的開源產品或工具。如果其他開源工具希望集成進來，有三層規範：一層是 SSO 規範，二層是工程組織結構規範（角色規範、工程規範），三層是應用開發流程規範。“"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"周可：數據治理的六大痛點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周可是微衆銀行大數據平臺工程師，他分享了《WeDataShpere 數據治理工場實踐之路》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，企業在數據治理上遇到的痛點有："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一、主數據管理缺失。數據零散、碎片化，數據重複利用和共享存在障礙，存在冗餘數據，增加數據使用成本；缺乏全生命週期的主數據管理流程和工具。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二、數據信息存在孤島。數據分散，形成信息孤島，共享困難；無法清楚的知道企業裏面有哪些數據？無法知道數據的具體結構？存儲在什麼地方？數據的屬主是誰？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三、數據關係難追蹤。數據是從哪裏來的？數據之間有什麼關係？數據是如何在企業內的各系統間傳遞的？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"四、數據質量不高。業務理解與實際開發脫節，數據質量問題突出；缺乏有效的方法和工具提升數據質量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"五、數據安全管控不全面。數據安全等級 \/ 敏感數據無法自動識別；數據安全風險無法及時發現；數據安全脫敏缺少有效的審計，無法確認數據是否合規使用。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"六、數據標準不統一，不規範。數據多樣化，缺少統一標準，集成困難；數據裏面的同一個術語存在不同的解釋，理解困難。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/574f33a2c0f1d556658b1a29e3378b83.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 周可"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這些問題，開源社區有三種解決方案："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種，Pull-based Architecture，特點是較少的組件依賴、一個團隊可以搞定，代表性的開源產品有 Amundsen、Metacat、WhereHows。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種，Push-based Architecture，特點是統一接入方式、可編程，Marquez 是代表性的開源產品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三種，Event-sourced Architecture，特點是元數據可實時更新、元數據模型是開放的，方便拓展。其典型開源產品有 Apache Atlas、DataHub、OpenMetadata。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在數據質量方面，開源社區也有一些解決方案，主要有 Apache Griffin、Great Expectations 和 AWS Deequ。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從個人角度，周可分享了開源產品選型思路。一方面是技術特性，它是否滿足場景需求、是否有完善的技術體系、技術產品的穩定性怎麼樣；另一方面，還需要考慮時間成本與人力成本、是否配置維護團隊、新 feature 的開發模式。因此，某種意義上，需要做取捨。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回到 WeDataSphere，周可闡述了 WeDataSphere 在這方面的實踐。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，隨着互聯網技術的發展，各大金融機構積累了海量豐富的數據，對數據的管理和應用能力已經成爲其核心競爭力之一。同時，大數據的發展帶來了企業對個人隱私信息的過度採集和使用的問題。並且，監管部門也對金融數據治理提出了更高的要求。現在，加強數據治理不僅是銀行自身發展的需要，也是行業政策規範的重點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，WeDataSphere 在數據治理上經過三個階段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一階段，平臺化。基於 WeDataSphere 大數據平臺，提升數據處理能力和應用效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二階段，工具化。Data Governance Studio 實現數據管理線上化，並推進安全、質量、成本管控，管好數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三階段，資產化。數據中臺、自動化取數服務。同時，促進數據應用，積累沉澱有價值的數據，實現數據資產化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周可指出，治理基線在於不僅完善數據治理制度，明確數據授權管理及數據認責、數據分級體系，保護用戶隱私，而且規範數據架構，明確數據生命週期管理要求，有力支撐數據應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體說來，數據治理工場 (Data Governance Studio) 要解決數據標準、數據字典、數據訪問控制、數據脫敏、數據血緣、數據質量六大問題，爲多數據源提供端到端數據治理能力。那麼，其基礎是建立數據目錄，指引數據的獲取、訪問和使用，輔助數據的有效利用，工具化落地數據管理規範。數據目錄可以解決數據在哪裏、數據去何處、數據誰在管和數據誰要用的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了構建好統一的數據目錄，內部要先建設元數據中間件，從全局視角管理數據狀況。有了元數據目錄，就可以形成自動化審批流程，從而在運維裏實現自動化處理。有了數據目錄，可以實現數據授權統一管理，確保數據合規使用，有跡可循。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"易小華：WeDataShpere 在薩摩耶雲的應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易小華是薩摩耶雲數據業務部總監，他分享了《WeDataShpere 在薩摩耶雲的應用》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"薩摩耶雲作爲中國領先的獨立雲服務科技解決方案供應商，以人工智能、大數據、移動互聯網和雲計算等核心技術能力爲支持，通過深度應用 AI 決策，自主研發一系列 SaaS 產品組合及數智薩摩耶雲平臺，爲各行各業的客戶提供端到端雲原生科技解決方案，包括：金融雲解決方案、產業雲解決方案、信用雲解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b4\/b4bb3f6aab3980b2538991e9394a830a.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"薩摩耶雲數據業務部總監 易小華"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對薩摩耶雲來說，使用 WeDataShpere 之前，流程是業務部門向大數據團隊提需求，大數據團隊評估需求，然後是排期開發，大數據團隊交付成果，最後業務部門驗收。當時，數據團隊負責公司十幾個部門數據分析類需求，但團隊規模有限，除了數據類需求，他們還要進行大數據平臺的高可用維護、數據倉庫迭代開發維護等。往往存在一定的業務開發類需求需要排隊等待排期，與業務要求的快速上線相違背。這是數據團隊人力資源上的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，薩摩耶雲總部位於深圳，分別在上海、長沙等城市設有職場，這讓面對面溝通存在一定困難，這也是團隊在溝通方面遇到的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，大數據組件衆多，這導致形成了大量數據應用孤島，計算和存儲資源使用也不合理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過調研後，薩摩耶雲最終選擇 WeDataSphere。易小華說：“我們也看過其他的同類產品，不是付費，就是功能不完善，社區不活躍，以及國內用戶少，開源版本問題較多。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼選擇 WeDataShpere？原因主要有五點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一，WeDataSphere 提供源代碼，社區非常活躍，響應速度很快；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二，它可以提供本地化部署，能滿足數據安全合規需求；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，WeDataSphere 能進行定製化開發和功能擴展；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，它能解決業務數據的開發痛點；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第五，WeDataSphere 能解決業務自主上線調度的痛點。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除此之外，WeDataSphere 在國內用戶較多，認可度較高；提供中文文檔，還有微信羣在線服務。同時，它用於微衆銀行內部，在不斷迭代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2020 年 5 月，薩摩耶雲正式上線 WeDataSphere，包括可視化、質量監控系統、計算系統、工作流系統和開發系統以及調度。同時，讓 WeDataSphere 0.9 版本分別適配內部使用的 CDH 5.14.4 和 CDH 6.3.2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易小華表示，在使用開源版本過程中，他們發現很多 bug，各組件累計修復了 100+bug。並且，他們還修改了界面風格、圖標和文字描述，方便業務人員使用，從而更適合薩摩耶雲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據易小華介紹，業務主要用到數據開發、數據工作流、數據調度和報表可視化等功能，提高數據開發效率 400%+。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"薩摩耶雲還與微衆銀行、天翼雲等聯合研發基於 Linkis 1.0 版本的實時開發組件 Streamis，其中，微衆銀行負責項目整體協調與把控，微衆銀行與 Boss 直聘負責 StreamPlugin 和 Stream WorkFlow 模塊，天翼雲則負責 Stream JobManager 前端，Stream JobManager 後端由仙翁科技負責，薩摩耶雲則負責 Stream Datasource 模塊。五方利用騰訊會議、企業微信和微信羣每週開一次週會，同步進度，共同解決問題。比如，在開發 streamis-datasource-transfer 模塊對接 linkis 數據源時，缺少 linkis-datasource 模塊，無法推進開發測試，後經微衆側協調，與天翼雲團隊多方共同推進，解決 streamis 依賴 linkis-datasource 問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從 2020 年 5 月上線到現在，WeDataSphere 在薩摩耶雲十幾個部門得到應用，覆蓋用戶超 300 人，涵蓋風控部門、營銷獲客部門和研發部門以及職能部門。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在易小華看來，雖然薩摩耶雲使用 WeDataSphere 的時間不長，但是團隊技能得到很大提升。以前，他們團隊人員主要做業務腳本的開發，通過引入 WeDataSphere，提升了團隊人員的 Java 應用開發能力。同時，團隊具備一戰式數據應用開發、集成和治理平臺的建設能力，提升了實時計算平臺建設能力，比如 Prophecis 二次開發能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他說：“向優秀團隊學習架構設計、溝通、編碼、問題定位等，這進一步提升了我們團隊成員的技能。同時，積極參與開源社區，提高了團隊影響力和公司口碑。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業務層面，通過使用 WeDataSphere，藉助開源社區，幫助業務提升數據開發和模型開發效率，降低了業務成本。實時數據使用能力提升，“以前，最快的是 10 分鐘才能使用數據，現在基本達到秒級”，幫助業務更快響應。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“我們做業務數據中臺和 AI 中臺，WeDataSphere 是我們中臺重要的技術組件。“他補充道。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"吳梓煜：在 WeDataSphere 中構建機器學習工場 Prophecis Studio"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲當天活動的最後一位分享嘉賓，微衆銀行大數據平臺工程師吳梓煜分享了《在 WeDataSphere 中構建機器學習工場 Prophecis Studio 之旅》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"吳梓煜指出，近年來，隨着企業數字化轉型，金融、零售等行業積累了大量的業務數據。如何從數據中發現價值，輔助業務進行決策，這是各行各業都在探索的。在技術層面，高性能計算、智能化算法等技術的快速發展，也爲企業從海量數據中低成本地發現數據價值提供了技術可行性。但是，在機器學習應用實際落地過程中，企業依然面臨一系列挑戰，包括數據接入難、應用碎片化、工程化複雜、模型部署複雜。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/be\/befbdb7ee838c0dc071f07754926584a.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 吳梓煜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 MLOps 是一種機器學習工程文化和手段，旨在統一機器學習系統開發 (Dev) 和機器學習系統運營 (Ops)。對企業來說，MLOps 實施意味着將在機器學習系統構建流程的所有步驟（包括集成、測試、發佈、部署和基礎架構管理）中實現自動化和監控。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，業界已經有開源的 MLOps 方案。吳梓煜介紹了三種主流的開源 MLOps 方案。第一種是 Kubeflow。它是一個基於 Kubernetes 構建的端到端生產級別機器學習平臺，基於雲原生的方案，覆蓋了機器學習從模型訓練、模型開發、模型部署等整個機器生命週期。其主要包括 Notebooks、Pipeline、AutoML、Serving、Arena 等組件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種方案是 MLFlow，它是一款開源端到端機器學習生命週期管理工具，基於 Python 開發，輕量級方案，專注於解決 ML 開發中的 Tracking、工程管理、模型管理與部署等痛點，主要包含 Tracking、Projects、Models 等模塊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三種方案是天樞人工智能開源平臺，國產開源機器學習平臺，提供了包括數據處理、模型開發、模型訓練和模型管理等功能，方便用戶一站式構建 AI 算法，底層基於 kubernetes，工作鏈較完善。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對此，WeDataSphere 提出 Prophecis Studio 機器學習工場，它是一站式機器學習平臺，主要包含 MLLabis、Machine Learning Flow、Model Factory、Application Factory 等模塊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，Prophecis Studio 集數據導入、模型開發、分佈式模型訓練、模型部署等功能於一體，基於 Kubernetes 提供計算集羣的多租戶管理能力，爲用戶提供機器學習應用開發的一站式體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它致力於打通 MLOps 與 WeDataSphere 數據組件的結合，幫助用戶更快、更便捷、更智能的挖掘和提升數據價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體實踐中，第一個模塊是機器學習開發環境，依託 Notebook Controller 管理 Notebook，同時，微衆開發了 Notebook Server，各個租戶的資源通過命名空間隔離，Controller 將 Notebook 創建到對應租戶的命名空間中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二個模塊是機器學習分佈式建模服務，主要是打通機器學習，把數據拉取、機器學習模型訓練、模型存儲、模型版本管理和鏡像構建以及模型部署全部打通，集合在一個模塊裏。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三個模塊是機器學習模型工廠。據悉，模型工廠基於開源模型部署工具 Seldon Core 進行構建，提供機器學習模型管理、模型部署測試、模型鏡像打包、模型報告等功能，“我們對其做了定製和拓展”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後是機器學習應用工廠。吳梓煜介紹，Prophecis AF 是基於青雲開源的 KubeSphere 進行構建的機器學習應用工廠，Kubersphere 是一款集分佈式、多租戶、多集羣、企業級能力的開源容器平臺。首先是監控和告警，吳梓煜表示，把它與公司內部 CMDB 系統和 IMS 系統打通，支持容器實例自動關聯內部應用信息。其次，是資源管理，支持限制 Namespace GPU 資源配額。第三是持久化存儲，對接內部的共享存儲和 MySQL。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"寫在最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲開源，大家相識；因爲 WeDataSphere，大家相聚。無論是開源大咖，還是 WeDataSphere 貢獻者和維護者，亦或是廣大社區參與者和開源愛好者，大家因熱愛技術，踐行開源之道，匯聚於此。"}]}]}
Hadoop 沒死，開源大數據基於技術和潮流不斷演進
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"10 月 30 日下午，2021 WeDataSphere 社區大會在深圳灣科技生態園創新廣場順利舉行。現場，開源界大咖、WeDataSphere 社區貢獻者和維護者，以及近兩百位開源愛好者，大家共聚一堂，交流技術，共論開源之道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/df81062844d48590f75c1a0a5cb05bd4.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，本次大會不僅邀請到開放原子開源基金會 TOC 主席堵俊平，而且包括 WeDataSphere 社區發起人邸帥、天翼雲大數據平臺技術專家王小剛、薩摩耶雲數據業務部總監易小華和四位微衆銀行大數據平臺工程師均到場。作爲開源大數據行業的觀察者和實踐者，他們紛紛發表了精彩的演講內容。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"堵俊平：Hadoop 沒死，開源大數據基於技術和潮流在不斷演進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"堵俊平是開源大數據行業的“老兵”，他開場先帶領大家回顧了開源大數據最近十幾年的發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 20 世紀 90 年代，隨着互聯網的快速發展，數據快速增長，但以前的技術手段無法解決新問題。後來，谷歌基於搜索引擎的多年實踐提出“三駕馬車”，即 GFS、MapReduce 和 BigTable。這“三駕馬車”給 Hadoop 創始人 Doug Cutting 很大的啓發，於是，他在 2006 年寫出第一個 Hadoop 引擎。這算是業界開源大數據的起點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fa\/faec81c4b26819ce0ec51c45c5c6bc97.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"開放原子開源基金會 TOC 主席堵俊平"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop 橫空出世後，被很多互聯網公司和科技公司紛紛採用，因爲大家的需求類似——怎麼以更低成本提升硬件利用效率，從而最大化地發揮數據價值。之後，Hadoop 在 2008 年從 Apache 畢業，這成爲標誌性事件。再過一年，第一款大數據雲服務在 AWS 上線，這意味着開源大數據走向雲時代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此後，Hadoop 生態加速發展，除了 Hadoop，又有了 Spark、Flink 等。 到 2.0 時代，Yarn 從 Hadoop 中分離， 它把資源管理跟上層的應用調度做了兩層分離，這樣，在 Hadoop 底層，Yarn 更像一個通用平臺，上面是更多的引擎，整個體系可以不斷演進。3.0 時代，Hadoop 開始與更多的技術相結合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"縱觀整個開源大數據的生態系統，既有新的大數據項目在不斷孵化，又有一些式微的開源項目。總體上，它根據技術和潮流的方向不斷演進，而非一成不變的靜態系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大體而言，這個領域的技術有以下趨勢：一是統一的 SQL，即 Unified SQL。二是批流的統一。生態趨勢上，開源大數據不斷衍生和迭代，Hadoop 也沒有死，但如果把 Hadoop 視爲開源大數據生態體系，那麼它仍在蓬勃發展，只是引擎沒人在用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9dda93631c2ec583b58f735c38af42e2.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了回顧開源大數據的發展歷程，堵俊平還分享了自己對開源的看法。開源項目有多個角色，包括開發者、用戶、廠商和基金會等，每個角色只要持續參與進來，從中找到價值，生態就會持續演進。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"邸帥：企業數據平臺的建設應該量體裁衣，因地制宜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲 WeDataSphere 社區發起人、微衆銀行大數據平臺團隊負責人，邸帥分享了 “WeDataSphere 大數據平臺套件的建設思路和開源歷程”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼要建數據平臺？邸帥認爲這是由於兩方面的需要：一是在機構或公司，它是商業化挑戰的需要；二是技術挑戰的需要。從容量、性能、效率和成本方面，企業要考慮以什麼樣的技術形式解決構建數據平臺的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ce\/cebeda64db1dadfb4a46fa36ee0df4f1.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"WeDataSphere 社區發起人 邸帥"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他表示，數據平臺對於企業，猶如血液循環系統之於人體。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼，企業如何建設數據平臺？借用《平臺革命》一書，平臺最核心的價值是促成很多最核心的交互，即“參與者 + 價值單元 + 過濾器 =>核心交互”。而最能體現平臺價值的能力是吸引、促進和匹配。在數據領域，問題在於怎樣提供更好的數據工具和服務，促使數據平臺完成吸引、促進和匹配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bb\/bb53c2c827c6345b6f272b41e04168f5.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論是鋼鐵俠的 AI、雲上數倉（比如 Snowflake）或 Gartner 十大數據和分析趨勢，在邸帥看來，每個公司或團隊真正需要的數據平臺應該要量體裁衣，因地制宜，根據自己的情況找到比較好的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以量體爲例，企業需要在數據管理能力層面、數據平臺能力層面和數據應用情況方面摸清現狀，從而確立自己的目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"邸帥表示，“我最想表達的是，如何建設好數據平臺，要從數據管理、數據平臺、數據應用三個層面摸清現狀，然後基於 WeDataSphere + 其他優秀開源項目提供的“布料”來“量體裁衣”。我們 WeDataSphere 的建設思路最核心的三點是：首先方向選擇是主力投入上層功能工具系統的建設和開發，底層引擎層更多是做補充，比如 bug 修復。然後兩個最核心的設計要點是“一站式”和“全連通”。“"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 WeDataSphere 建設思路上，他指出：數據平臺範圍龐大，團隊規模資源有限，要想做出特色和優勢，須從產品架構設計，和多團隊共建模式上進一步優化。借用微衆銀行副行長兼首席信息官馬智濤先生的一句話就是“小刀鋸大樹，必須靠方法。”具體而言，一是通過合理的構建功能工具集成開發框架和中間件層，獲得更極致的連通、解耦、易擴展、高度複用能力，簡化整體架構和調用關係，大幅降低新功能工具開發和平臺運維成本；二是基於“開源”吸引一切可以團結的力量，多方聯合共建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在社區建設層面，微衆銀行、天翼雲、仙翁科技、薩摩耶雲、Boss 直聘的團隊聯合開發了 Streamis。同時，今年，微衆銀行與合作伙伴還構建了兩個大的項目：Exchangis 1.0和 DataModelCenter。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 WeDataSphere 的未來，邸帥表示，首先做得更深，其次做得更廣。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"尹強：Apache Linkis 是站在四個巨人肩膀上的開源工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 WeDataSphere 中，計算中間件 Linkis 解決前臺各種工具、應用，和後臺各種計算存儲引擎間連接簡化和複用問題，是非常重要的一環。尹強是 Apache Linkis PPMC，他分享了 Linkis 的建設及開源思路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先說說大數據平臺面臨的計算治理問題。大數據平臺不斷髮展，底層有很多的計算存儲引擎，上層有很多的應用工具，包括批量計算、交互式計算、流式計算、數據分析工具、數據治理工具等。平臺組件越來越多，開發維護愈加困難。這樣的背景下，Linkis 誕生，主要解決上層應用工具和底層存儲引擎之間關於“連通、擴展、管控、編排、複用等‘計算治理‘問題”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/42f8302015fb722000f2381b9c277522.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據技術專家 尹強"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對此問題，目前主要有四個開源社區解決方案：一是 Apache Livy，通過提供 REST 服務，它能簡化用戶與 Spark 集羣的交互。同時，它聚焦於“連接”能力。通過 Job 或代碼片段，簡化 Spark 任務提交。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二是 Apache Zeppelin，它是基於 Web 的交互式數據分析 Notebook，聚焦於“擴展”能力。獨有的 Interpretor 架構，可快速對接新的大數據引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三是 Netflix Geine，它是分佈式作業編排引擎，聚焦於“管控”能力。強大的標籤體系，支持按需路由作業到不同的 Hadoop 集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"四是 openLooKeng，它是一款高性能數據虛擬化引擎，提供統一 SQL 接口，具備跨數據源 \/ 數據中心分析能力。聚焦於“編排”能力。通過編排 SQL，支持跨數據中心、跨雲的異構數據源查詢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在尹強看來，這四個解決方案各自特點非常突出，比如 Apache Livy 在連接上做得很好。但是，它們又有不足，因此企業級大數據平臺呼喚“既具備擴展又具備管控，還具有編排、連通、複用能力的中間件”。Linkis 是一款站在巨人肩膀上的開源工具，可以極大簡化大數據平臺的架構，降低開發和運維的複雜度。爲什麼這麼說？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linkis 構建解耦計算中間件層，具備連通、擴展、管控,、編排和複用能力。連通上，告別應用孤島，它可以打通用戶資源 & 運行時環境。擴展上，通過實現 Linkis EngineConnPlugin，完成新引擎適配，統一解決高併發、高可用、多租戶等問題。管控上，它具備基於標籤的多級精細化資源控制和回收能力，可以實現多級精細化參數化配置控制。編排上，它基於 Orchestrator 服務的雙活策略設計和混算策略設計。複用上，它大大降低上層應用 \/ 工具開發後臺代碼量，並基於 Linkis 快速高效打造數據平臺工具套件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，尹強談到了 Apache Linkis 的開源情況和未來規劃。從 2019 年 7 月，Linkis 第一個開源版本發佈，去年 12 月，社區主導的第一個版本發佈。此後，它不斷迭代和發展，於 2021 年 7 月進入 Apache 基金會孵化器，並在 9 月榮獲 2021 屆開源產業大會“OSCAR 尖峯開源項目和開源社區”獎項。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於未來，一方面是它在 Apache 基金會好好孵化；另一方面，優化社區運營，與生態相關頂級項目建立更緊密合作，同時，產品迭代加速。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"王小剛：WeDataSphere 開源組件在天翼雲大數據平臺產品中的應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲天翼雲大數據平臺團隊技術專家，王小剛分享了 WeDataSphere 開源組件在天翼雲大數據平臺產品中的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先談到爲什麼會與 WeDataSphere 結緣？他解釋：內因是電信集團進行“雲改數轉”戰略，天翼雲過去的集羣存在建設分散，整體資源複用率低，以及數據加工腳本管理混亂，故障頻發等問題。外因在於，CDH、HDP 合併，商業版軟件訂閱費用昂貴，技術綁定，同時，市場缺乏優質的同類開源產品。我們拿自己開刀花了一年多時間從 CDH 商業版升級到 Hadoop 最新版，但光去 CDH 還不夠，核心是用戶需要統一的大數據平臺。我們同時圍繞 Hadoop 最新版，擁抱開源、打造自主可控、技術領先、安全穩定的大數據平臺。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6a\/6a659f55c57aabdb2de6ea136182bc9c.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"天翼雲大數據平臺技術專家 王小剛"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之所以瞭解 WeDataSphere，主要在於 Linkis。“當時，看到它的時候感覺很新鮮，並且和我們的思路一致，它已經開源，平臺完善度高，社區也很活躍。它可封裝，可水平復制，一站式，非常開放。對我們來說，改造成本適中，運維和維護成本較低。所以，我們選了 Linkis“。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在王小剛看來，Linkis 有強大的入口能力，這正是他們所需的，用它來隔離底層複雜性，保證持續智能優化的可能性。其次，Linkis 奉行開放開源的策略，親切且先進的架構設計，有相對完善的文檔，這些對於開發者來說都是非常友好的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"並且，他們展示了基於 Linkis 持續在做的一些工作，比如更完善的容器化、更大規模的驗證、數據源管理、數據集成能力、實時計算能力等，以及後續希望實現跨集羣作業提交、SQL 路由、安全能力等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在具體實踐上，用他的話說：先“玩”起來，深度試用，與內部底座進行整合；然後，逐步完善，修復 bug；接着，針對遇到的一些問題進行較爲深度的二次開發；最後就是擁抱開源與社區共建，現在我們已經完全深度參與社區，和社區小夥伴們分工合作，共同研發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而說到用戶體驗側，王小剛提到，隨着大數據平臺的不斷普及，各層級的用戶也日益增多，市場主流產品存在一定同質化現象，而在這其中，往往使產品能夠脫穎而出的，就是用戶易用性的提升。天翼雲在這塊也做了很多嘗試，包括新手引導、產品即文檔（產品內置的智能用戶手冊）、換膚（Dark 模式）、界面優化設計等，都是以用戶體驗爲核心進行優化和打磨。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，新手引導主要解決用戶首次進入大數據平臺的操作問題，幫助用戶以輕量化的形式，快速上手常用的功能按鍵，熟悉產品的界面操作使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"產品即文檔是內置的彈出式幫助中心，主要協助用戶解決當前頁面 \/ 功能模塊的上手痛點，包括對於該模塊的功能介紹，典型步驟介紹，常見問題等。該功能試圖排除用戶在使用中碰到的 80% 以上的問題，切實給客戶提供使用幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Dark 模式是針對開發人員和大數據平臺使用者專門打造的護眼模式界面皮膚，基於原配色深度優化，符合開發者的使用和交互習慣，增加開發效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，隨着平臺已經擁有越來越多的試用和正式用戶，小剛提到，持續圍繞“客戶成功”這一核心理念來打造產品，是我們始終貫徹的。團隊堅持“用戶至上、開放共贏、坦誠清晰”，將整個團隊也打造成一個產品，給用戶帶來最好的體驗。作爲一隻充滿活力，開源開放的大數據平臺團隊，希望未來能繼續和社區的小夥伴一起把 DataSphere Studio 打造的更友好易用。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"王和平：用戶需要一站式交互體驗的開源數據分析工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲 DataSphere Studio 核心開發 ，王和平現場分享了《WeDataSphere 數據分析工場的建設和開源思路》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他首先介紹了開源數據分析工具的選型思路。在構建數據分析時，用戶面臨的痛點有哪些？王和平表示，一是數據從哪來、數據安全怎麼保證，二是數據怎麼分析，怎樣進行模型訓練，三是數據質量怎麼把控，四是分析出的結果如何彙報，五是數據調度怎麼做。最後，這麼多工具，如何給用戶帶來一站式的交互體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a2\/a2e41b01314d8531f1fefe9821030512.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 王和平"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於用戶面臨的挑戰，業界都有相應的工具去解決問題，比如數據分析工具執行交互式分析和模型訓練以及任務調度，還有數據交換工具、數據治理工具等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，業界有很多開源的數據交換工具，例如 DATAX、DataX Web、Dbus 和 Exchangis。在數據分析工具上，業界也有 Apache Zeppelin、Scriptis 和 Hue。同樣，業界的開源數據治理工具有 pydqc、Apache Atlas、Deequ、Data Hub、GriFFin 和 Qualitis。開源調度工具，業界有 Dolphin Scheduler、XXL、Apache Airflow 和 Schedulis。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然工具不少，但是對一個數據分析師來說，它可能並沒有真正解決問題。並且，問題還有缺乏標準規範，工程管理體系不統一，用戶權限不統一，UI 不統一；應用孤島，資源物料不互通，運行時上下文不互通。此外，重複造輪子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲解決這些問題，DataSphere Studio 誕生，它具備一站式、全連通、全流程、可插拔和強管控特點，可以覆蓋需求、設計、開發、測試到生產所有階段，一條工作流串通所有應用工具，上層新工具只需實現 AppConn 插件，便可快速打通其他工具。並且，基於工作空間的多租戶隔離，實現以租戶爲單位進行統一的權限管控、成員管理和協同開發等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"王和平說：“DataSphere Studio 有一個 AppConn，它去做連接，集成其他優秀的開源產品或工具。如果其他開源工具希望集成進來，有三層規範：一層是 SSO 規範，二層是工程組織結構規範（角色規範、工程規範），三層是應用開發流程規範。“"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"周可：數據治理的六大痛點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周可是微衆銀行大數據平臺工程師，他分享了《WeDataShpere 數據治理工場實踐之路》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，企業在數據治理上遇到的痛點有："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一、主數據管理缺失。數據零散、碎片化，數據重複利用和共享存在障礙，存在冗餘數據，增加數據使用成本；缺乏全生命週期的主數據管理流程和工具。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二、數據信息存在孤島。數據分散，形成信息孤島，共享困難；無法清楚的知道企業裏面有哪些數據？無法知道數據的具體結構？存儲在什麼地方？數據的屬主是誰？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三、數據關係難追蹤。數據是從哪裏來的？數據之間有什麼關係？數據是如何在企業內的各系統間傳遞的？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"四、數據質量不高。業務理解與實際開發脫節，數據質量問題突出；缺乏有效的方法和工具提升數據質量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"五、數據安全管控不全面。數據安全等級 \/ 敏感數據無法自動識別；數據安全風險無法及時發現；數據安全脫敏缺少有效的審計，無法確認數據是否合規使用。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"六、數據標準不統一，不規範。數據多樣化，缺少統一標準，集成困難；數據裏面的同一個術語存在不同的解釋，理解困難。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/574f33a2c0f1d556658b1a29e3378b83.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 周可"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這些問題，開源社區有三種解決方案："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種，Pull-based Architecture，特點是較少的組件依賴、一個團隊可以搞定，代表性的開源產品有 Amundsen、Metacat、WhereHows。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種，Push-based Architecture，特點是統一接入方式、可編程，Marquez 是代表性的開源產品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三種，Event-sourced Architecture，特點是元數據可實時更新、元數據模型是開放的，方便拓展。其典型開源產品有 Apache Atlas、DataHub、OpenMetadata。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在數據質量方面，開源社區也有一些解決方案，主要有 Apache Griffin、Great Expectations 和 AWS Deequ。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從個人角度，周可分享了開源產品選型思路。一方面是技術特性，它是否滿足場景需求、是否有完善的技術體系、技術產品的穩定性怎麼樣；另一方面，還需要考慮時間成本與人力成本、是否配置維護團隊、新 feature 的開發模式。因此，某種意義上，需要做取捨。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回到 WeDataSphere，周可闡述了 WeDataSphere 在這方面的實踐。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，隨着互聯網技術的發展，各大金融機構積累了海量豐富的數據，對數據的管理和應用能力已經成爲其核心競爭力之一。同時，大數據的發展帶來了企業對個人隱私信息的過度採集和使用的問題。並且，監管部門也對金融數據治理提出了更高的要求。現在，加強數據治理不僅是銀行自身發展的需要，也是行業政策規範的重點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，WeDataSphere 在數據治理上經過三個階段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一階段，平臺化。基於 WeDataSphere 大數據平臺，提升數據處理能力和應用效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二階段，工具化。Data Governance Studio 實現數據管理線上化，並推進安全、質量、成本管控，管好數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三階段，資產化。數據中臺、自動化取數服務。同時，促進數據應用，積累沉澱有價值的數據，實現數據資產化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周可指出，治理基線在於不僅完善數據治理制度，明確數據授權管理及數據認責、數據分級體系，保護用戶隱私，而且規範數據架構，明確數據生命週期管理要求，有力支撐數據應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體說來，數據治理工場 (Data Governance Studio) 要解決數據標準、數據字典、數據訪問控制、數據脫敏、數據血緣、數據質量六大問題，爲多數據源提供端到端數據治理能力。那麼，其基礎是建立數據目錄，指引數據的獲取、訪問和使用，輔助數據的有效利用，工具化落地數據管理規範。數據目錄可以解決數據在哪裏、數據去何處、數據誰在管和數據誰要用的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了構建好統一的數據目錄，內部要先建設元數據中間件，從全局視角管理數據狀況。有了元數據目錄，就可以形成自動化審批流程，從而在運維裏實現自動化處理。有了數據目錄，可以實現數據授權統一管理，確保數據合規使用，有跡可循。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"易小華：WeDataShpere 在薩摩耶雲的應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易小華是薩摩耶雲數據業務部總監，他分享了《WeDataShpere 在薩摩耶雲的應用》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"薩摩耶雲作爲中國領先的獨立雲服務科技解決方案供應商，以人工智能、大數據、移動互聯網和雲計算等核心技術能力爲支持，通過深度應用 AI 決策，自主研發一系列 SaaS 產品組合及數智薩摩耶雲平臺，爲各行各業的客戶提供端到端雲原生科技解決方案，包括：金融雲解決方案、產業雲解決方案、信用雲解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b4\/b4bb3f6aab3980b2538991e9394a830a.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"薩摩耶雲數據業務部總監 易小華"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對薩摩耶雲來說，使用 WeDataShpere 之前，流程是業務部門向大數據團隊提需求，大數據團隊評估需求，然後是排期開發，大數據團隊交付成果，最後業務部門驗收。當時，數據團隊負責公司十幾個部門數據分析類需求，但團隊規模有限，除了數據類需求，他們還要進行大數據平臺的高可用維護、數據倉庫迭代開發維護等。往往存在一定的業務開發類需求需要排隊等待排期，與業務要求的快速上線相違背。這是數據團隊人力資源上的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，薩摩耶雲總部位於深圳，分別在上海、長沙等城市設有職場，這讓面對面溝通存在一定困難，這也是團隊在溝通方面遇到的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，大數據組件衆多，這導致形成了大量數據應用孤島，計算和存儲資源使用也不合理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過調研後，薩摩耶雲最終選擇 WeDataSphere。易小華說：“我們也看過其他的同類產品，不是付費，就是功能不完善，社區不活躍，以及國內用戶少，開源版本問題較多。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼選擇 WeDataShpere？原因主要有五點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一，WeDataSphere 提供源代碼，社區非常活躍，響應速度很快；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二，它可以提供本地化部署，能滿足數據安全合規需求；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，WeDataSphere 能進行定製化開發和功能擴展；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，它能解決業務數據的開發痛點；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第五，WeDataSphere 能解決業務自主上線調度的痛點。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除此之外，WeDataSphere 在國內用戶較多，認可度較高；提供中文文檔，還有微信羣在線服務。同時，它用於微衆銀行內部，在不斷迭代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2020 年 5 月，薩摩耶雲正式上線 WeDataSphere，包括可視化、質量監控系統、計算系統、工作流系統和開發系統以及調度。同時，讓 WeDataSphere 0.9 版本分別適配內部使用的 CDH 5.14.4 和 CDH 6.3.2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易小華表示，在使用開源版本過程中，他們發現很多 bug，各組件累計修復了 100+bug。並且，他們還修改了界面風格、圖標和文字描述，方便業務人員使用，從而更適合薩摩耶雲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據易小華介紹，業務主要用到數據開發、數據工作流、數據調度和報表可視化等功能，提高數據開發效率 400%+。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"薩摩耶雲還與微衆銀行、天翼雲等聯合研發基於 Linkis 1.0 版本的實時開發組件 Streamis，其中，微衆銀行負責項目整體協調與把控，微衆銀行與 Boss 直聘負責 StreamPlugin 和 Stream WorkFlow 模塊，天翼雲則負責 Stream JobManager 前端，Stream JobManager 後端由仙翁科技負責，薩摩耶雲則負責 Stream Datasource 模塊。五方利用騰訊會議、企業微信和微信羣每週開一次週會，同步進度，共同解決問題。比如，在開發 streamis-datasource-transfer 模塊對接 linkis 數據源時，缺少 linkis-datasource 模塊，無法推進開發測試，後經微衆側協調，與天翼雲團隊多方共同推進，解決 streamis 依賴 linkis-datasource 問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從 2020 年 5 月上線到現在，WeDataSphere 在薩摩耶雲十幾個部門得到應用，覆蓋用戶超 300 人，涵蓋風控部門、營銷獲客部門和研發部門以及職能部門。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在易小華看來，雖然薩摩耶雲使用 WeDataSphere 的時間不長，但是團隊技能得到很大提升。以前，他們團隊人員主要做業務腳本的開發，通過引入 WeDataSphere，提升了團隊人員的 Java 應用開發能力。同時，團隊具備一戰式數據應用開發、集成和治理平臺的建設能力，提升了實時計算平臺建設能力，比如 Prophecis 二次開發能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"他說：“向優秀團隊學習架構設計、溝通、編碼、問題定位等，這進一步提升了我們團隊成員的技能。同時，積極參與開源社區，提高了團隊影響力和公司口碑。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業務層面，通過使用 WeDataSphere，藉助開源社區，幫助業務提升數據開發和模型開發效率，降低了業務成本。實時數據使用能力提升，“以前，最快的是 10 分鐘才能使用數據，現在基本達到秒級”，幫助業務更快響應。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“我們做業務數據中臺和 AI 中臺，WeDataSphere 是我們中臺重要的技術組件。“他補充道。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"吳梓煜：在 WeDataSphere 中構建機器學習工場 Prophecis Studio"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲當天活動的最後一位分享嘉賓，微衆銀行大數據平臺工程師吳梓煜分享了《在 WeDataSphere 中構建機器學習工場 Prophecis Studio 之旅》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"吳梓煜指出，近年來，隨着企業數字化轉型，金融、零售等行業積累了大量的業務數據。如何從數據中發現價值，輔助業務進行決策，這是各行各業都在探索的。在技術層面，高性能計算、智能化算法等技術的快速發展，也爲企業從海量數據中低成本地發現數據價值提供了技術可行性。但是，在機器學習應用實際落地過程中，企業依然面臨一系列挑戰，包括數據接入難、應用碎片化、工程化複雜、模型部署複雜。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/be\/befbdb7ee838c0dc071f07754926584a.webp","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"微衆銀行大數據平臺工程師 吳梓煜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 MLOps 是一種機器學習工程文化和手段，旨在統一機器學習系統開發 (Dev) 和機器學習系統運營 (Ops)。對企業來說，MLOps 實施意味着將在機器學習系統構建流程的所有步驟（包括集成、測試、發佈、部署和基礎架構管理）中實現自動化和監控。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，業界已經有開源的 MLOps 方案。吳梓煜介紹了三種主流的開源 MLOps 方案。第一種是 Kubeflow。它是一個基於 Kubernetes 構建的端到端生產級別機器學習平臺，基於雲原生的方案，覆蓋了機器學習從模型訓練、模型開發、模型部署等整個機器生命週期。其主要包括 Notebooks、Pipeline、AutoML、Serving、Arena 等組件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種方案是 MLFlow，它是一款開源端到端機器學習生命週期管理工具，基於 Python 開發，輕量級方案，專注於解決 ML 開發中的 Tracking、工程管理、模型管理與部署等痛點，主要包含 Tracking、Projects、Models 等模塊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三種方案是天樞人工智能開源平臺，國產開源機器學習平臺，提供了包括數據處理、模型開發、模型訓練和模型管理等功能，方便用戶一站式構建 AI 算法，底層基於 kubernetes，工作鏈較完善。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對此，WeDataSphere 提出 Prophecis Studio 機器學習工場，它是一站式機器學習平臺，主要包含 MLLabis、Machine Learning Flow、Model Factory、Application Factory 等模塊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"據悉，Prophecis Studio 集數據導入、模型開發、分佈式模型訓練、模型部署等功能於一體，基於 Kubernetes 提供計算集羣的多租戶管理能力，爲用戶提供機器學習應用開發的一站式體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它致力於打通 MLOps 與 WeDataSphere 數據組件的結合，幫助用戶更快、更便捷、更智能的挖掘和提升數據價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體實踐中，第一個模塊是機器學習開發環境，依託 Notebook Controller 管理 Notebook，同時，微衆開發了 Notebook Server，各個租戶的資源通過命名空間隔離，Controller 將 Notebook 創建到對應租戶的命名空間中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二個模塊是機器學習分佈式建模服務，主要是打通機器學習，把數據拉取、機器學習模型訓練、模型存儲、模型版本管理和鏡像構建以及模型部署全部打通，集合在一個模塊裏。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三個模塊是機器學習模型工廠。據悉，模型工廠基於開源模型部署工具 Seldon Core 進行構建，提供機器學習模型管理、模型部署測試、模型鏡像打包、模型報告等功能，“我們對其做了定製和拓展”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後是機器學習應用工廠。吳梓煜介紹，Prophecis AF 是基於青雲開源的 KubeSphere 進行構建的機器學習應用工廠，Kubersphere 是一款集分佈式、多租戶、多集羣、企業級能力的開源容器平臺。首先是監控和告警，吳梓煜表示，把它與公司內部 CMDB 系統和 IMS 系統打通，支持容器實例自動關聯內部應用信息。其次，是資源管理，支持限制 Namespace GPU 資源配額。第三是持久化存儲，對接內部的共享存儲和 MySQL。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"寫在最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲開源，大家相識；因爲 WeDataSphere，大家相聚。無論是開源大咖，還是 WeDataSphere 貢獻者和維護者，亦或是廣大社區參與者和開源愛好者，大家因熱愛技術，踐行開源之道，匯聚於此。"}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章