Apache Kyuubi PPMC燕青：爲什麼說這是開源最好的時代？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在大數據領域，Apache Spark 早已成爲最炙手可熱的計算引擎。隨着 Spark 兩年磨一劍，正式發佈 3.0 版本，帶來諸多新特性的 Spark 更是擁有了無限想象空間。不過對於用戶而言，平臺的技術門檻始終是個不小的挑戰。也正因如此，不少項目選擇直接建立在 Spark 之上，通過將平臺的能力統合，並引入新的特性，從而降低用戶使用門檻，實現大數據價值的最大化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Kyuubi 正是這樣一個擁抱 Spark、高性能的通用 JDBC 和 SQL 執行引擎，由網易數帆旗下有數大數據團隊開源。Kyuubi 提供標準化的接口，賦予用戶調動整個數據湖生態的數據的能力，使得用戶能夠像處理普通數據一樣處理大數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"6 月 21 日，Apache 軟件基金會宣佈，Kyuubi 以全票通過的表現，正式進入 Apache 基金會孵化器。這也側面證明了 Kyuubi 的受歡迎程度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"近日，InfoQ 有幸採訪到了網易數帆技術專家、Apache Kyuubi PPMC、Apache Spark Committer 燕青，和他聊了聊 Kyuubi 一路發展背後的故事，以及他對於開源的理解。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"“九尾狐”Kyuubi"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Kyuubi 的命名源自中國神話《山海經》，意爲“九尾狐”。狐會噴火，象徵 Spark；狐有九尾，類比多租戶。這個命名也體現出了 Kyuubi 系統設計之初的主要目的——在 Spark 上實現多租戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"事實上，在 Kyuubi 之前，市面上已存在具備類似能力的產品，比如 Spark ThriftServer（簡稱 STS）。這是 Spark 社區現有的、基於 HiveServer2 實現的Thrift 服務，旨在無縫兼容 HiveServer2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"雖然 STS 的性能極佳，但當前並不完善，尤其在企業場景下存在較多短板。比如，單 Spark 應用實現的 STS 並不能完整支持多租戶，因爲 STS 本質上是一個 Spark Application，整個 Application 只有全局唯一的用戶名，並同時包括 Driver 端和 Executor 端。而對於像網易這樣有多條產品線的互聯網公司來說，每條產品線的數據在一定程度上是隔離的。因此，只有支持多租戶才能滿足公司對於數據安全、資源隔離、高可用以及高併發的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這也就促使網易內部開發了 Kyuubi。Kyuubi 在統一接口基礎上，拓展了 STS 在多租戶模式下的使用場景，並依託多租戶概念獲得了完善的資源隔離共享能力和數據安全隔離的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在 2018 年上線之初，Kyuubi 的定位只是在 Spark 上實現多租戶，再引入一個比較細粒度的權限控制，做一個小而美的系統。因此，Kyuubi 的第一代架構主要面向的是 BI 產品。但是正式上線並開源後，團隊發現用戶並不在意系統本身的設計初衷以及使用場景是什麼，所以當一些用戶使用 Kyuubi 做 ETL 等工作時，用起來磕磕絆絆，反饋也不是特別好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"苦苦掙扎一年後，團隊決定深入 Spark 社區去貢獻，從中找到設計一套比較通用的、面向更多用戶的架構方案靈感。這期間，Spark 也正在醞釀一個大版本的跨越式升級，從 2.4 直接來到 3.0 版本。Kyuubi 團隊在其中參與了很多工作，並一同推動 Spark 社區發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“在這個過程中，我們對 Spark 的內核機制有了一個更加充分的瞭解。在這個基礎上，我們意識到我們對於 Kyuubi 原本的構想是不太可持續的，Kyuubi 的第一代架構視野比較小，應用場景也很少。因此，我們覺得是時候對 Kyuubi 的架構進行一番革新了。”燕青回憶道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"2020 年，團隊重新設計了 Kyuubi 第二版的架構，新架構的使用場景更加豐富。具體來看，Kyuubi 的使用場景主要包括以下三個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"1.替換 HiveServer2，輕鬆獲得 10~100 倍性能提升。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Kyuubi 高度兼容 HiveServer2 接口及行爲，支持無縫遷移；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Kyuubi 分層架構，消除客戶端兼容性問題，支持無感升級；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Kyuubi 支持 Spark SQL 全鏈路優化及再增強，性能卓著；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"高可用、多租戶、細粒度權限認證各種企業級特性統統都有。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"2.構建 Serverless Spark 平臺。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Serverless Spark 目標絕對不是讓用戶調用 Spark 的 API、繼續寫 Spark 作業；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過 Kyuubi 預置的 Engine 模塊，用戶無需理解 Spark 邏輯，入門門檻極低；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶只需通過 JDBC 及 SQL 操作數據專注自身業務開發即可，資源彈性伸縮，0運維；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持資源管理器（Kubernetes, YARN等），Engine 生命週期，Spark 動態資源分配 3 級不同粒度全方位的資源彈性策略；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持 YARN\/Kubernetes 多種資源管理器同時調度，保障歷史作業安全遷移上雲；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Spark 自適應查詢引擎（AQE）及 Kyuubi AQE plus，提供澎湃動力。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"3.構建統一數據湖探索分析管理平臺。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持 Spark 所有官方數據源及第三方數據源；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持 Spark DSv2 元數據管理，直觀進行數據湖構建及管理；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持 Apache Iceberg\/Hudi, DeltaLake等所有主流數據湖框架；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一個接口一個引擎一份數據，提供統一的分析查詢、數據攝取、數據湖管理平臺；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"批流一體，支持流式作業（Upcoming）。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當前，Kyuubi 不僅在網易內部承接了大量工作，在業內也有多家大型公司採用 Kyuubi 解決問題。“Kyuubi 新架構做出來之後，很多其他公司的小夥伴也開始活躍地參與到這個項目中來，慢慢地參與的人多了以後，我們就想着是不是可以去 Apache 軟件基金會進行孵化。”燕青說道。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"獨行者速，衆行者遠"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"2021 年 3 月份，團隊開始正式籌備將 Kyuubi 捐贈給 Apache 軟件基金會。事實上，擁抱 Apache 軟件基金會的念頭一早就紮在 Kyuubi 團隊成員心中了。在燕青看來，做下這一決定背後的原因主要有以下 3 點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第一，團隊以及公司都有意願去做這件事情。網易內部本身始終貫行開源開放的策略，所以當團隊和公司提出這個想法後，公司內部也給予了一定的支持，比如爲項目配套了一些運營力量等等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第二，隨着第二版架構的登場，Kyuubi 發展路線日益清晰，也有越來越多的開發者參與到項目中來，社區規模逐步擴大。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第三，團隊成員發現一些潛在的 Kyuubi 用戶或是開發者產生 IP 產權方面的顧慮。此前即便 Kyuubi 已經開源，但依舊是屬於網易的產品，將項目捐贈給 Apache 軟件基金會後，可以消除這些開發者的顧慮，進一步吸引更多的開發者參與其中。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"6 月 21 日，Apache 軟件基金會宣佈，Kyuubi 以全票通過的表現，正式進入 Apache 基金會孵化器。根據投票結果，Kyuubi 獲得了 13 個約束性投票(binding votes)和 8 個無約束性投票(non-binding votes)，投票全部持贊同意見，無棄權票和反對票。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在走向 Apache 軟件基金會的過程中，燕青坦言 Kyuubi 很幸運地得到了來自公司內外的不少幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“有些幫助是無形的，有些幫助是有形的”，燕青說，“Kyuubi 能夠成功進入 Apache 基金會孵化器，很大程度上是因爲那些開源前輩們已經把路鋪好，很多前輩在國內開源領域深耕多年，比如姜寧老師等等，他們在前期做的一些努力讓我們後人能夠乘涼。理論上來說，現在的項目要想進入 Apache 軟件基金會孵化，比他們那個年代要更加容易一些。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“另外，姜寧老師發起並創立的 ALC Beijing 也會幫助我們國內開發者或企業去孵化一些本土項目，比如他們會把一些 Apache 軟件基金會的文檔翻譯成中文，這對於英文不太好的開發者來說幫助非常大。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在這個過程中，Kyuubi 也得到了不少來自 Apache 軟件基金會的支持與幫助。“Apache 的指導文檔本身就對我們產生很大的幫助。如果沒有這個文檔，你根本不知道這一步做什麼，下一步做什麼。Apache 的指導文檔寫得非常詳細，就算沒有其他人的幫助，只要你耐心地把這些文檔閱讀下來，也能更好地完成項目捐獻。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在 Kyuubi 項目官宣進入 Apache 孵化器的那天，Kyuubi 特別感謝了很多給予過自己幫助的人，比如給 Kyuubi 提供指導的 Champion 和 Mentors 姜寧，Mentors 章劍鋒、張鐸、Akira Ajisaka。也有爲 Kyuubi 提出 issue 和建議的夥伴們，以及爲 Kyuubi 做出貢獻與支持的國內外數十家企業用戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"獨行者速，衆行者遠。而這，也是開源一貫傳承的精神。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"發展社區的關鍵在於多傾聽開發者的聲音"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"現在，距離 Kyuubi 進入 Apache 大家庭已有月餘，至於進入 Apache 之後發生了哪些變化，燕青認爲主要體現在項目管理和社區管理兩個方面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在過去，Kyuubi 歸屬於網易，項目管理者相當於擁有超級管理員的權限，可以直接對項目做決策，發版時間也不固定。進入 Apache 孵化器後，項目有發版等重大決策時，需要在郵件列表裏一起討論，併發起投票。內部溝通討論結束後，還需要在孵化器郵件列表裏面再去發起討論、投票，此外，Apache 軟件基金會方面也會幫忙做項目審查，看下在法律或是其他方面是否合規。相較過去，雖然整個決策鏈路長了一些，但這樣的層層討論對項目本身來說更負責，藉助這些外腦，項目的發展路線也愈加清晰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在社區管理方面，燕青坦言“以前我們比較隨意，現在進入 Apache 軟件基金會後，我們定了一個目標就是要把社區建立起來。”在燕青看來，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"社區是由人構成的，要想讓社區建立並發展起來，需要多傾聽社區的聲音，多去了解這些個人開發者或是公司的需求"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"至於如何提高社區的活躍度，燕青認爲最重要的一點是要"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"儘量避免犯一些錯誤"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，要以開放友好且包容的態度去接納開發者。對於社區新人，要更加有耐心，給他們時間去成長。此外，在宣傳方面還需要加大力度，社區運營者也需要積極地去配合宣傳。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最後在項目上，要更加註重品控的把握。“如果 Release 版本不太穩定的話，從用戶層面來說，會勸退很多用戶”，燕青解釋道。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kyuubi 的終極目標：讓大數據平民化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於 Kyuubi 的未來，燕青也有着很高的期待：希望建立在 Apache Spark 和數據湖技術之上，統一門戶，打造一個理想的數據湖管理平臺，讓用戶處理大數據能像處理普通數據一樣輕鬆。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"具體來說，第一要增強 Kyuubi 對 Kubernetes 雲原生的支持，讓 Kyuubi 提供的服務以及計算資源都可以在容器中進行；第二要增強 Kyuubi 對數據湖的支持，讓用戶能夠更簡單地管理、使用和建設數據湖，實現所見即所得；第三要繼續深入優化引擎側，增加流式場景，打造批流一體的平臺。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“未來，希望 Kyuubi 可以讓 Spark、讓大數據平民化”，燕青憧憬道。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"“這是開源最好的時代”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了是 Apache Kyuubi 的作者，燕青還有多個身份，比如 Apache Spark Committer、Apache Submarine Committer。從業多年，燕青始終在開源大數據領域深耕，對他來說，“開源是可以做一輩子的事情”，而他也始終熱愛開源，信仰開源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在燕青看來，開源非常利於個人學習和提升自己。“很多開源大神把自己的代碼或 idea 放在 GitHub 或其他開源平臺上面，如果你選擇在某一技術領域深耕的話，可以在這些平臺中找到很多學習資料，邊學邊做”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過在開源社區的不斷學習，燕青也實現了從醫學信息學博士到頂級開源社區核心貢獻者的身份轉變。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"提及自己積累下來的經驗，燕青覺得"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"在持續學習之外，也要擁有良好的心態，以及明確且專注的技術方向"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。“開源是一個圈子，這個圈子是開放的，但從個人的角度來看，每個人的想法不同，圈子外的人或多或少會感覺到有一堵牆的存在。如果你能夠擁有良好的心態，謙虛、友善地去和社區溝通，自然也能換來同樣友好的迴應，從而形成良好的迴路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"此外，從技術角度來講，方向一定要明確，專注在一個領域內深耕，從而慢慢從不擅長變得擅長。比如我自己就是一直圍繞着 Spark 這個項目去深耕，當我在這個項目裏做得差不多的時候，我並不是離開它，而是繼續在這個項目裏去幫助別人，包括我做的 Kyuubi 也是圍繞着 Spark 去做的。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當前，雖然國內開源發展存在進步與亂象並生，機遇與挑戰共存的狀態，但燕青堅信“這是開源最好的時代”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在其看來，很多開源前輩已經在這個領域做出了長時間的鋪墊和積累，很多熱愛開源的開發者也都投入其中，大家都在爲開源貢獻出自己的一份力量，幫助它朝着更好的方向去發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“從某種程度上來講，開源氛圍好與壞是在描述一個圈子，是圈子就會有一堵無形的牆，有時牆裏面的人會翻出去把牆外的人引進來，有時牆外的人會主動地進到牆裏。其實只要你學會融入它，幫助它去改善，就一定會朝着好的方向去發展。另外，從開源角度來講，有些項目正確地衰落，其實正說明我們在往更正確的方向走去”，燕青說道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於國內開源的未來，燕青期望可以有越來越多的優秀開源項目誕生自高校，讓開源氛圍更加“年輕化”。監管層面，也希望有一些相應的規範，幫助開源更好地持續發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"嘉賓介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"燕青，Apache Kyuubi PPMC，Apache Spark Committer，Apache Submarine Committer。目前就職於網易數帆有數大數據團隊，專注於開源大數據領域。"}]}]}