{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"連數據湖都開始放棄 Hadoop 了？ "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"十年前，Hadoop 是解決大規模數據分析的“白熱化”方法，如今卻被企業加速拋棄。曾經頂級的 Hadoop 供應商都在爲生存而戰，Cloudera 於本月完成了私有化過程，黯然退市。MapR 被 HPE 收購，成爲 HPE Ezmeral 平臺的一部分，該平臺尚未在調查中顯示所佔據的市場份額。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從數據湖方向發力的 Databricks，卻逃脫了“過時”的命運，於今年宣佈獲得 16 億美元的融資。另一個大數據領域的新星——雲數倉 Snowflake，去年一上市就創下近 12 年來最大 IPO 金額，成爲行業領跑者。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"行業日新月異，十年時間大數據的領導勢力已經經歷了一輪更替。面對新的浪潮，我們需要做的是將行業趨勢和技術聯繫起來，思考技術之間的關聯和背後不變的本質。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Databricks 和 Snowflake 做對了什麼？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop 和數據湖都是 2006 年開始興起的概念。爲什麼同時期興起，經歷十多年發展，Hadoop 逐漸衰落，數據湖反而迎來了熱潮？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡上有個說法：“公有云玩家”以零成本贈送 Hadoop 產品，加速了 Hortonworks 和 Cloudera 等廠商的衰落。但像 Snowflake 這樣的新興企業，它最大的合作伙伴卻是 AWS 等雲廠商。作爲雲廠商的生態系統合作伙伴，Snowflake 推動了大量 Amazon EC2 \/S3 的銷售。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們看來，Hadoop 只是數據湖的"},{"type":"text","marks":[{"type":"strong"}],"text":"一種實現"},{"type":"text","text":"，而新一代數據湖通過擁抱雲計算和開源社區，經歷了新生。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Databricks 和 Snowflake 都抓住了 OLAP 的數據分析場景，基於興起的雲技術在數據存儲和數據消費之間構建了新的中間數據抽象層（Data Virtualization），即屏蔽了底層系統的異構性，又提供了"},{"type":"text","marks":[{"type":"strong"}],"text":"遠超 Hadoop 生態系統的用戶體驗"},{"type":"text","text":"。這是他們能夠成功的根本原因。或者用一句話描述，就是利用公有云的 Infrastructure，來做自己的“cloud on cloud”商業模式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雲計算的背景下，"},{"type":"text","marks":[{"type":"strong"}],"text":"計算存儲相分離"},{"type":"text","text":"的設計概念逐漸清晰，促進了現代數據湖和數據倉庫的架構在數據存儲和數據消費端的進一步解耦以及業界標準接口的規範化，這使得開源社區通過這些標準接口貢獻新技術的發展成爲可能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，"},{"type":"text","marks":[{"type":"strong"}],"text":"公有云計算平臺的出現"},{"type":"text","text":"，某種程度上加速了數據的壟斷和計算需求的集中，推動業界對於數據以及數據處理做出更明確的需求定義，針對性地投入開源項目，以社區這種更靈活開放的方式促進技術發展，再反哺公有平臺的進化和發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由雲計算和開源合作推動相輔相成，才孵化出一系列有利於數據湖和數據倉庫發展的突破性技術，"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"表示 "},{"type":"text","marks":[{"type":"strong"}],"text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的關係型數據庫，如 Oracle、DB2、MySQL、SQL SERVER 等採用行式存儲法，而一些新興分佈式數據庫所採用的列式存儲相較於行式存儲能加速 OLAP 工作負載的性能，這已經是衆所周知的事實。但在我們看來，更加革命性的變化是列式存儲格式的標準化。Parquet 和 ORC 的列式存儲格式都是 2013 年發明的，隨着時間的推移，它們已經被接受爲業界通用的列式存儲格式。數據是有慣性的，要對數據進行遷移和格式轉換都需要算力來克服慣性；而數據的標準化格式意味着用戶不再被某一特定的 OLAP 系統所綁定（locked in），而是可以根據需要，選擇最合適的引擎來處理自己的數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二大突破性技術是分佈式查詢引擎的出現，如 SparkSQL、Presto 等。隨着數據存儲由中心式向分佈式演進，如何在分佈式系統之上提供快速高效的查詢功能成爲一大挑戰，而衆多 MPP 架構的查詢引擎的出現很好地解決了這個問題。SQL 查詢不再是傳統數據庫或者數據倉庫的獨門祕籍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在解決了分佈式查詢的問題之後，下一個問題是，對於存儲於數據湖中的數據，很多是非結構化的和半結構化的，如何對它們進行有效地組織和查詢呢？在 2016 到 2017 年之間，Delta Lake、Iceberg、Hudi 相繼誕生。這些類似的產品在相近的時間同時出現，表明它們都解決了業界所亟需解決的問題。這個問題就是，傳統數據湖是爲大數據、大數據集而構建的，它不擅長進行真正快速的 SQL 查詢，並沒有提供有效的方法將數據組織成表的結構。由此，在缺乏有效的數據組織和查詢能力的情況下，數據湖就很容易變成數據沼澤（data swamp）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"利用雲基礎架構，是成功關鍵"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果仔細瞭解一下 Databricks 和 Snowflake 的發展歷程，可以發現兩者的出發點有所不同。Databricks 是立足於數據湖，進行了向數據倉庫方向的演化，提出了湖倉一體的理念；而 Snowflake 在創建之初就是爲了提供現代版的數據倉庫，近些年來也開始引入數據湖的概念，但本質上說它提供的還是一個數據倉庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Snowflake 利用雲技術革新了傳統數據倉庫。它提供了一個基於公有云的、完全託管的數據倉庫，把傳統的軟硬件一體的消費模式改造爲了軟件服務的模式（Software as a Service）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論是存儲還是計算，Snowflake 都利用了公有云提供的基礎設施，從而使任何人都可以在雲端使用數據倉庫服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，傳統的數據湖在數據分析上存在不足，不能很好地提供 OLAP 場景的支持。因此，Databricks 通過 Delta Lake 提供的表結構和 Spark 提供的計算引擎，構建了一套完整的基於數據湖的 OLAP 解決方案。Databricks 的願景是基於數據湖提供包括 AI 和 BI 在內的企業數據分析業務的一站式解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與 Snowflake 相似的是，Databricks 也充分利用了雲基礎架構提供的存儲和計算服務，在其上構建了入門成本低、定價隨使用而彈性擴展的軟件服務方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"存儲正在經歷新一輪革命：從 Hadoop 到對象存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據湖和 Hadoop 並不是競爭關係。作爲一種架構，數據湖會將其它技術整合到一起，而 Hadoop 則成爲了一種可以用來構建數據湖的組件。換句話說，Hadoop 和數據湖的關係是互補的，在可預見的未來，隨着數據湖繼續流行，Hadoop 還將繼續存在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，數據湖會拋棄 Hadoop 轉而支持其他技術嗎？這最終有可能會發生，因爲作爲一種綜合性技術架構，除了 Hadoop HDFS 之外，數據湖還可以選擇“對象存儲”作爲它的核心存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在越來越多的，像 Databricks、Snowflake 這樣的數據平臺類創業公司選擇採用對象存儲作爲存儲的核心。從頭開始搭建一個分佈式存儲很難，其中的坑只有踩過的人才知道。所謂“計算出了問題大不了重試，而數據出了問題則是真出了問題”。所以很多數據平臺類創業公司如 Databricks、Snowflake 等都會藉着"},{"type":"text","marks":[{"type":"strong"}],"text":"計算存儲分離的趨勢"},{"type":"text","text":"，選擇公有云提供的存儲服務作爲它們的數據和元數據存儲，而公有云上最通用的分佈式存儲就是對象存儲。例如 Databricks 的論文標題就是“Delta Lake: High Performance ACID Table Storage over Cloud Object Stores”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b3\/b3fdd9648f92eb3d1933bf807c325c89.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"典型的數據湖架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"爲什麼他們會選擇採用對象存儲作爲存儲的核心？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"表示採用對象存儲作爲存儲的核心主要因素有四個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從技術角度來說，首先，對象存儲即爲非結構化存儲，數據以原始對象的形式存在。這點貼合數據湖對於先存儲原始數據，再讀取完整數據信息後續分析的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，對象存儲擁有更先進的分佈式系統架構，在可擴展性和跨站點部署上，比傳統存儲更具優勢。由於對象存儲簡化了文件系統中的一些特性，沒有原生的層級目錄樹結構，對象之間幾乎沒有關聯性，因此對象存儲的元數據設計能更爲簡單，能夠提供更好的擴展性。此外，數據湖業務往往也需要底層存儲提供多站點備份和訪問的功能，而絕大部分對象存儲原生支持多站點部署。通常用戶只要配置數據的複製規則，對象存儲就會建立起互聯的通道，將增量和 \/ 或存量數據進行同步。對於配置了規則的數據，你可以在其中任何一個站點進行訪問，由於跨站點的數據具備最終一致性，在有限可預期的時間內，用戶會獲取到最新的數據。AWS 在 Storage Day 上剛剛宣佈的“S3 Multi-Region Access Points”就是一例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，在協議層面，由 AWS 提出的 S3 協議已經是對象存儲事實上的通用協議，這個協議在設計之初就考慮到了雲存儲的場景，可以說對象存儲在協議層就是雲原生的協議，在數據接口的選擇和使用上更具靈活性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，對象存儲本身就是應雲存儲而生，一開始起家的用戶場景即爲二級存儲備份場景，本身就具備了低價的特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，對象存儲是雲時代的產物，支持原始數據存儲、分佈式可擴展、高靈活性、低價，都是對象存儲之所以被選擇的原因。可以預計在未來會有更多的數據業務完全基於對象存儲而構建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"數據湖存儲架構落地實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從技術上說，無論是數據倉庫、數據湖還是 S3 存儲，都是數據存儲的一種方式，所以從某種程度上來說，存儲行業正在迅速演變爲數據業務行業。在數據湖的探索上，"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"也分享了一些戴爾科技集團 OSA（Object, Stream and Analytics）團隊的實踐經驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"市場上的很多數據平臺公司還是利用 s3a 協議而非原生的 s3 協議來訪問對象存儲的服務。例如 Databricks 雖然在他們的論文中提到他們在對象存儲之上提供了一層表結構的服務，即 Delta Lake。但實際上 Delta Lake 與底層對象存儲交互的協議仍然用的是 s3a，即對象存儲版本的 HDFS 協議。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，從對象存儲的角度來看，s3a 協議由於多了一層協議轉換，並不能完全發揮高性能對象存儲的延時和吞吐量的優勢。因此，OSA 研發團隊認爲基於原生 s3 協議構建數據平臺的存儲訪問層應該能大大提升存儲層的訪問能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，對象存儲的部署方式也需要從單一的公有云變成混合雲 \/ 多雲的混合架構。“Latency matters”，如何在儘可能接近數據產生端進行實時、近實時處理是下一代數據湖需要直面的問題。更靈活的架構也會幫助企業級用戶打消對於數據平臺鎖定的擔憂（locked in\/lock down）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/58\/5835694d962a6dca0a4386312bb67a15.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新一代數據平臺的基本架構都是存算分離，即計算層和存儲層是松耦合的。計算層無狀態，所有的數據、元數據以及計算產生的中間數據都會存儲於存儲層之中。這一架構的優勢包括更好的擴展性（計算、存儲獨立擴展），更好的可用性（計算層的失效不影響存儲，因此能夠很快恢復），以及更低的成本。爲了適應存算分離的架構，對象存儲本身也需要進一步發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這個發展趨勢中 OSA 的產品研發也經歷了從軟件定義（Software Defined）到雲原生（Cloud Native）的改造，來適應數據平臺的需要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“儘管我們從一開始就使用了 Scale-out 架構，基於標準 x86 服務器構建，所以很早我們就認爲自己已經做到了軟件定義。但是在實踐中，我們發現部署在服務器上的存儲產品還有複雜的運維問題、動態擴展問題等等，帶來的用戶體驗並不完美。與此同時，以容器、Kubernetes 爲代表的技術的出現，顛覆了原來的軟件定義存儲，可以讓存儲做到雲原生，從提供存儲產品轉型爲提供存儲服務（Storage as a Service）。Kubernetes 提供了強大的集羣管理功能，而容器提供了低開銷的資源隔離和控制方案。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"藉助於這兩大技術，OSA 研發團隊徹底革新了對象存儲的技術棧，用微服務的方式拆分巨型單體服務，將存算分離的架構應用到對象存儲內部。通過劃分各個存儲服務的邊界，制訂服務合約，將需要直接訪問磁盤 IO 的功能集中到一兩個服務中，實現了絕大多數服務的無狀態化。改造之後，對象存儲能夠輕鬆運行在資源相對有限的邊緣節點之上。元數據服務可以根據計算負載單獨擴容，而不受限於底層存儲資源。改造之後，對象存儲本身也是 Kubernetes 所託管的諸多服務之一，只要集羣中有足夠的計算和存儲資源，對象存儲就能根據 IO 負載來動態地 Scale up 或者 Scale down，達到完全的彈性，而這也是雲原生存儲的最終目標之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"認爲，"},{"type":"text","marks":[{"type":"strong"}],"text":"想要適應存算分離的大趨勢，不是簡單地把現有存儲對接到計算層就可以完成的，存儲本身要經歷新一輪架構革命才能更好地服務於計算層"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"技術並非完美，當前對象存儲存在的四個挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在架構之外，數據平臺型業務也給對象存儲的特性提出了若干新的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一個挑戰"},{"type":"text","text":"是數據分析型業務所需要的性能要遠高於數據備份的場景，對象存儲需要能夠提供與計算需求相匹配的大帶寬與低延時。閃存技術的發展、NVMe over Fabrics 標準的提出使得高性能對象存儲成爲可能。在選擇網絡的時候，建議選擇基於以太網的 RoCE 或者 TCP 協議作爲網絡傳輸層，能同時兼顧性能、成本和部署的靈活性。同時，要完全發揮 NVMe 閃存的高併發性，也需要對數據鏈路做徹底的異步化和無鎖化改造，並且 OSA 研發團隊傾向於選擇用戶態編程模型，因爲它對內核的依賴小，對雲原生環境下的複雜部署更友好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，在寫入模式上，雖然追加寫是在硬盤時代提出的思想，但在閃存時代依然是一個優秀的選擇。由於閃存的物理特性，閃存的固件需要對內部的塊做後臺的垃圾掃描和回收，一旦閃存找不到完全空閒的塊，那再次寫入就會需要經過讀取 - 修改 - 寫入的路徑，會大大影響閃存的寫入性能。追加寫的方式能夠減少碎片，減輕閃存後臺垃圾回收的壓力。同時，對於新興的低成本、高密度的 QLC 閃存而言，追加寫也是絕佳的搭檔。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，對象存儲還需要根據業務場景來優化性能。例如，高帶寬指的不僅僅是性能測試時能達到的帶寬，因爲傳統性能測試往往使用 GB 級別甚至更大的對象，達到高帶寬相對容易。然而，在數據分析的場景中，所要處理的是中等或者較大的對象（32Kb -1024Kb 級別），在這樣的業務場景中，很可能帶寬瓶頸不在底層的 NVMe 吞吐率上，而在元數據服務或者讀寫流水線的銜接上，這就需要依據數據分析的業務特徵在讀寫全鏈路上優化對象存儲的帶寬和延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二個挑戰"},{"type":"text","text":"來自於數據分析所包含的衆多元數據操作。因此對象存儲不僅要能夠提供大帶寬，還要在處理小對象和元數據操作如 list 時提供足夠的性能。這就比較考驗對象存儲的元數據管理能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"認爲，“根據我們的經驗，基於 B-tree 的某些高效變種來存儲元數據在寫放大、讀延時等方面依然要優於 LSM tree。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工業界更常用的 Bw-tree 就是一例。Bw-tree 整體上仍然是一棵 B-tree，但是其中的每個節點則借鑑了 LSM tree 的 log structured 設計，每個節點有一個 base record 和一組 delta log。Bw-tree 還引入了 logical page 和 physical page 的概念，兩者通過一張 mapping table 進行關聯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這些設計，Bw-tree 能夠避免過於頻繁的 page split\/merge 操作以及 dirty page 向上的傳導性，提高了寫入的效率，又避免了 LSM tree 在後臺進行的大量分層 merge 操作。同時，整棵樹在點查詢和區域查詢的效率上依然接近傳統 B-tree，優於 LSM tree。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三個挑戰"},{"type":"text","text":"是對象存儲如何兼顧性能和成本。數據湖中存儲了龐大的企業數據，但在任一時間點，可能只有一小部分數據是被數據分析業務所需要的。如果所有數據都放在性能最優的物理介質上（比如非易失性內存），那麼成本將變得過高，失去了雲存儲的經濟性，而如果在對象存儲的前端再加一層 cache 層，無疑也會增加整個系統的複雜度。因此如何有效識別冷熱數據，並將它們分區放置是對象存儲需要解決的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種簡單的方式，是把選擇權交給用戶。用戶通過設置一些固定的規則（如根據寫入時間、最後訪問時間等）觸發數據在不同介質之間的遷移。第二種方式，是通過對歷史訪問數據的分析，讓存儲系統來預測單個對象的生命週期，實現某種程度上的智能分層。更高級的方式，則是讓存儲系統能夠識別計算負載的模式特徵，根據模式特徵識別不同對象之間的關聯性，根據一組對象的訪問特點來決定分層策略。例如，如果某個工作負載的特徵是依次讀取某一組對象，那麼存儲層就可以提前預取將要被訪問的對象，來進一步加速計算。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第四個挑戰"},{"type":"text","text":"是對象存儲如何與開源生態相結合。現階段比較成熟的在數據湖之上提供表結構的開源產品是 Delta Lake、Iceberg 和 Hudi。對這三款產品 OSA 研發團隊做了一些預研和實驗。Hudi 基於 Hadoop 生態系統，特別是和 Hive 有深度綁定。Delta Lake 的設計非常優秀，不過由於是 Databricks 的產品，它還有一個不開源的商業版，許多高級特性只有在商業版上才提供。同樣由於 Databricks 的關係，在計算層上，Delta Lake 和 Apache Spark 深度綁定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與前兩者相比，Iceberg 的接口抽象特別清晰，靈活性也可以完美適配，在計算層也可以跟 Apache Flink、Apache Spark 等計算層解決方案無縫集成。因此，作爲長期企業級對象存儲解決方案的提供者，在對這些開源方案進行對比研究後，在現階段 OSA 研發團隊選擇了 Apache Iceberg 作爲其開源解決方案的組成部分，未來也不排除將和更多的開源產品集成，爲客戶提供完整的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好地適配底層的對象存儲，OSA 研發團隊爲 Iceberg 做了一個通用的 S3 表管理組件（S3 Catalog）。在 S3 的標準 API 中，上傳數據需要預先知道對象的大小，因此在追加上傳的場景下，其調用方法無法像 HDFS 那樣簡潔。所以在具體實現中，追加寫的操作需要在本地預先處理，並以整體上傳。第二，利用第三方鎖的實現，解決了對象存儲中不支持原子重命名操作的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除此之外，OSA 研發團隊還提供了一個基於 Dell EMC 商業對象存儲 ECS 的專屬表管理組件（ECS Catalog）。ECS 支持 Append 語義，使用 Append 的操作可以完美應對順序寫入未知長度文件的場景。ECS 還支持類 compare-and-swap (CAS) 語義。在併發提交的場景下，ECS 支持使用 If-Match 和 If-None-Match 對對象進行 CAS 操作，來實現原子化重命名的操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"超越數據湖，展望新一代數據平臺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在"},{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"看來，下一代數據平臺有更廣闊的範疇和視角，超越當前人們對數據湖的理解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它將不光涵蓋傳統的數據中心或者雲端部署的範疇，結合日益發展的邊緣計算，將來的數據平臺將會涵蓋從傳感器到邊緣節點到數據中心到雲端一整套生態系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/7e\/7e86adeb5f2a09ba8b5b754b6b95228c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Source: VMware 調研白皮書《基於全棧雲基礎架構，構建企業級業務創新平臺》"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這套生態系統上，數據是主角。平臺的使用者會慢慢弱化對底層系統的依賴，而更多地關注於數據處理管道和基於業務邏輯的數據處理邏輯需求的定義。基於此，平臺對於使用者的專業性要求也會逐漸降低，越來越多的人能夠利用數據平臺從數據中獲取信息價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存算分離將是下一代數據平臺的標準架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲層將更統一，而計算層根據負載也將擁有更多的靈活性。和傳統數據湖相似，數據依舊會統一存儲在數據湖中，先存儲後消費。但是，對於數據的結構化定義要求會越來越不重要，數據清洗或結構化的轉化將更多地由平臺智能化或者根據更人性化的配置定義完成。至於數據在各個環節中的存儲，也將隱藏在平臺內部。數據孤島將逐漸消亡，原因很簡單，如果數據湖本身能夠提供足夠好的 OLAP 支持，爲什麼還要把數據拷貝一份再放到數據倉庫中呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在計算層面，數據平臺應該是完全開放的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據具體的業務需要，用戶可以自行選擇靈活度更高的 Spark 或者 Flink 計算框架，或者是集成度更好的一體化方案。下一代數據平臺也應該提供強大的跨表查詢能力。無論數據是直接存儲在對象存儲中、存儲在 Iceberg 等表結構中、還是存儲在外部的數據庫中，數據平臺都支持對這些表進行聯合查詢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存儲和計算之間會進化出新的數據抽象層（現在正在發生）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多統一的業界標準如 Iceberg 等，會越來越多地出現。數據產生者和數據消費者基於這層抽象層協同合作。通過這層數據抽象層，數據平臺會慢慢將各個角色的數據消費者從系統部署與理解的細節中解脫出來，以關注業務邏輯本身。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如 Iceberg 創始人之一 Ryan Blue 所說，數據平臺的使用者應該更多關心自己的數據和數據產品，把剩下的問題交給數據平臺基礎架構去解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時從應用場景上來說，在傳統的離線數據分析場景之外，實時數據分析的業務場景正在增加。以 Spark micro batch 爲代表的近實時框架可以解決一部分的業務需求，但對於延時要求更高的場景，實時分析的框架還有待發展。以流式存儲爲依託的實時分析框架依然是個尚待開發的領域。另外，目前的數據平臺對結構化和半結構化數據已經有了比較好的支持，但對於非結構化數據，如何爲它們建立高效的索引來加速查詢和分析，依然是業界懸而未決的難題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"認爲，未來的數據平臺應該是涵蓋傳感器、邊緣節點、數據中心和雲端，存儲計算相分離，依賴數據抽象層，爲多種多樣的數據產生和消費者，提供集成了離線分析、近實時分析和實時分析數據處理鏈路的統一平臺，提供更簡單和標準化的接口供用戶使用，讓用戶聚焦於從數據中獲取價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉賓簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"，Pravega 中國社區創始人，戴爾科技集團 OSA（Object, Stream and Analytics）軟件開發總監。自 2007 年加入戴爾科技集團中國研發集團以來，一直專注於分佈式系統領域，先後參與了前後 2 代 Dell EMC 對象存儲產品的研發及商業化工作，正在積極擁抱新的混合雲 \/ 多雲和實時流處理時代。從 2016 年開始，參與 Pravega 開源項目，活躍在多個開源社區。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"，技術團隊傑出成員（Distinguished Member of Technical Staff），負責戴爾科技集團分佈式對象存儲系統架構設計和開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"雷璐"},{"type":"text","text":"，高級主管工程師（Senior Principal Engineer），專注於戴爾科技集團流處理平臺 SDP（Streaming Data Platform）系統架構的設計和開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文選自《"},{"type":"link","attrs":{"href":"https:\/\/www.infoq.cn\/minibook\/LxX7bFUwKH17bzxQkSKt","title":"xxx","type":null},"content":[{"type":"text","text":"中國卓越技術團隊訪談錄"}]},{"type":"text","text":"》（2021 年第五季），掃描二維碼查看更多獨家專訪↓"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/e1\/e126ae89435157ceb3e1614581d39cca.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"《中國頂尖技術團隊訪談錄》品牌升級，現正式更名爲《中國卓越技術團隊訪談錄》"},{"type":"text","text":"，這是 InfoQ 打造的重磅內容產品，以各個國內優秀企業的 IT 技術團隊爲線索策劃系列採訪，希望向外界傳遞傑出技術團隊的做事方法 \/ 技術實踐，讓開發者瞭解他們的知識積累、技術演進、產品錘鍊與團隊文化等，並從中獲得有價值的見解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你身處傳統企業經歷了完整的數字化轉型過程或者正在互聯網公司進行創新技術的研發，並希望 InfoQ 可以關注並採訪你所在的技術團隊，可以添加微信：whitecrow-tina，請註明來意及公司名稱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
拋棄Hadoop，數據湖才能重獲新生
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"連數據湖都開始放棄 Hadoop 了？ "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"十年前，Hadoop 是解決大規模數據分析的“白熱化”方法，如今卻被企業加速拋棄。曾經頂級的 Hadoop 供應商都在爲生存而戰，Cloudera 於本月完成了私有化過程，黯然退市。MapR 被 HPE 收購，成爲 HPE Ezmeral 平臺的一部分，該平臺尚未在調查中顯示所佔據的市場份額。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從數據湖方向發力的 Databricks，卻逃脫了“過時”的命運，於今年宣佈獲得 16 億美元的融資。另一個大數據領域的新星——雲數倉 Snowflake，去年一上市就創下近 12 年來最大 IPO 金額，成爲行業領跑者。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"行業日新月異，十年時間大數據的領導勢力已經經歷了一輪更替。面對新的浪潮，我們需要做的是將行業趨勢和技術聯繫起來，思考技術之間的關聯和背後不變的本質。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Databricks 和 Snowflake 做對了什麼？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop 和數據湖都是 2006 年開始興起的概念。爲什麼同時期興起，經歷十多年發展，Hadoop 逐漸衰落，數據湖反而迎來了熱潮？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡上有個說法：“公有云玩家”以零成本贈送 Hadoop 產品，加速了 Hortonworks 和 Cloudera 等廠商的衰落。但像 Snowflake 這樣的新興企業，它最大的合作伙伴卻是 AWS 等雲廠商。作爲雲廠商的生態系統合作伙伴，Snowflake 推動了大量 Amazon EC2 \/S3 的銷售。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們看來，Hadoop 只是數據湖的"},{"type":"text","marks":[{"type":"strong"}],"text":"一種實現"},{"type":"text","text":"，而新一代數據湖通過擁抱雲計算和開源社區，經歷了新生。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Databricks 和 Snowflake 都抓住了 OLAP 的數據分析場景，基於興起的雲技術在數據存儲和數據消費之間構建了新的中間數據抽象層（Data Virtualization），即屏蔽了底層系統的異構性，又提供了"},{"type":"text","marks":[{"type":"strong"}],"text":"遠超 Hadoop 生態系統的用戶體驗"},{"type":"text","text":"。這是他們能夠成功的根本原因。或者用一句話描述，就是利用公有云的 Infrastructure，來做自己的“cloud on cloud”商業模式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雲計算的背景下，"},{"type":"text","marks":[{"type":"strong"}],"text":"計算存儲相分離"},{"type":"text","text":"的設計概念逐漸清晰，促進了現代數據湖和數據倉庫的架構在數據存儲和數據消費端的進一步解耦以及業界標準接口的規範化，這使得開源社區通過這些標準接口貢獻新技術的發展成爲可能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，"},{"type":"text","marks":[{"type":"strong"}],"text":"公有云計算平臺的出現"},{"type":"text","text":"，某種程度上加速了數據的壟斷和計算需求的集中，推動業界對於數據以及數據處理做出更明確的需求定義，針對性地投入開源項目，以社區這種更靈活開放的方式促進技術發展，再反哺公有平臺的進化和發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由雲計算和開源合作推動相輔相成，才孵化出一系列有利於數據湖和數據倉庫發展的突破性技術，"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"表示 "},{"type":"text","marks":[{"type":"strong"}],"text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的關係型數據庫，如 Oracle、DB2、MySQL、SQL SERVER 等採用行式存儲法，而一些新興分佈式數據庫所採用的列式存儲相較於行式存儲能加速 OLAP 工作負載的性能，這已經是衆所周知的事實。但在我們看來，更加革命性的變化是列式存儲格式的標準化。Parquet 和 ORC 的列式存儲格式都是 2013 年發明的，隨着時間的推移，它們已經被接受爲業界通用的列式存儲格式。數據是有慣性的，要對數據進行遷移和格式轉換都需要算力來克服慣性；而數據的標準化格式意味着用戶不再被某一特定的 OLAP 系統所綁定（locked in），而是可以根據需要，選擇最合適的引擎來處理自己的數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二大突破性技術是分佈式查詢引擎的出現，如 SparkSQL、Presto 等。隨着數據存儲由中心式向分佈式演進，如何在分佈式系統之上提供快速高效的查詢功能成爲一大挑戰，而衆多 MPP 架構的查詢引擎的出現很好地解決了這個問題。SQL 查詢不再是傳統數據庫或者數據倉庫的獨門祕籍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在解決了分佈式查詢的問題之後，下一個問題是，對於存儲於數據湖中的數據，很多是非結構化的和半結構化的，如何對它們進行有效地組織和查詢呢？在 2016 到 2017 年之間，Delta Lake、Iceberg、Hudi 相繼誕生。這些類似的產品在相近的時間同時出現，表明它們都解決了業界所亟需解決的問題。這個問題就是，傳統數據湖是爲大數據、大數據集而構建的，它不擅長進行真正快速的 SQL 查詢，並沒有提供有效的方法將數據組織成表的結構。由此，在缺乏有效的數據組織和查詢能力的情況下，數據湖就很容易變成數據沼澤（data swamp）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"利用雲基礎架構，是成功關鍵"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果仔細瞭解一下 Databricks 和 Snowflake 的發展歷程，可以發現兩者的出發點有所不同。Databricks 是立足於數據湖，進行了向數據倉庫方向的演化，提出了湖倉一體的理念；而 Snowflake 在創建之初就是爲了提供現代版的數據倉庫，近些年來也開始引入數據湖的概念，但本質上說它提供的還是一個數據倉庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Snowflake 利用雲技術革新了傳統數據倉庫。它提供了一個基於公有云的、完全託管的數據倉庫，把傳統的軟硬件一體的消費模式改造爲了軟件服務的模式（Software as a Service）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論是存儲還是計算，Snowflake 都利用了公有云提供的基礎設施，從而使任何人都可以在雲端使用數據倉庫服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，傳統的數據湖在數據分析上存在不足，不能很好地提供 OLAP 場景的支持。因此，Databricks 通過 Delta Lake 提供的表結構和 Spark 提供的計算引擎，構建了一套完整的基於數據湖的 OLAP 解決方案。Databricks 的願景是基於數據湖提供包括 AI 和 BI 在內的企業數據分析業務的一站式解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與 Snowflake 相似的是，Databricks 也充分利用了雲基礎架構提供的存儲和計算服務，在其上構建了入門成本低、定價隨使用而彈性擴展的軟件服務方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"存儲正在經歷新一輪革命：從 Hadoop 到對象存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據湖和 Hadoop 並不是競爭關係。作爲一種架構，數據湖會將其它技術整合到一起，而 Hadoop 則成爲了一種可以用來構建數據湖的組件。換句話說，Hadoop 和數據湖的關係是互補的，在可預見的未來，隨着數據湖繼續流行，Hadoop 還將繼續存在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，數據湖會拋棄 Hadoop 轉而支持其他技術嗎？這最終有可能會發生，因爲作爲一種綜合性技術架構，除了 Hadoop HDFS 之外，數據湖還可以選擇“對象存儲”作爲它的核心存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在越來越多的，像 Databricks、Snowflake 這樣的數據平臺類創業公司選擇採用對象存儲作爲存儲的核心。從頭開始搭建一個分佈式存儲很難，其中的坑只有踩過的人才知道。所謂“計算出了問題大不了重試，而數據出了問題則是真出了問題”。所以很多數據平臺類創業公司如 Databricks、Snowflake 等都會藉着"},{"type":"text","marks":[{"type":"strong"}],"text":"計算存儲分離的趨勢"},{"type":"text","text":"，選擇公有云提供的存儲服務作爲它們的數據和元數據存儲，而公有云上最通用的分佈式存儲就是對象存儲。例如 Databricks 的論文標題就是“Delta Lake: High Performance ACID Table Storage over Cloud Object Stores”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b3\/b3fdd9648f92eb3d1933bf807c325c89.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"典型的數據湖架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"爲什麼他們會選擇採用對象存儲作爲存儲的核心？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"表示採用對象存儲作爲存儲的核心主要因素有四個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從技術角度來說，首先，對象存儲即爲非結構化存儲，數據以原始對象的形式存在。這點貼合數據湖對於先存儲原始數據，再讀取完整數據信息後續分析的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，對象存儲擁有更先進的分佈式系統架構，在可擴展性和跨站點部署上，比傳統存儲更具優勢。由於對象存儲簡化了文件系統中的一些特性，沒有原生的層級目錄樹結構，對象之間幾乎沒有關聯性，因此對象存儲的元數據設計能更爲簡單，能夠提供更好的擴展性。此外，數據湖業務往往也需要底層存儲提供多站點備份和訪問的功能，而絕大部分對象存儲原生支持多站點部署。通常用戶只要配置數據的複製規則，對象存儲就會建立起互聯的通道，將增量和 \/ 或存量數據進行同步。對於配置了規則的數據，你可以在其中任何一個站點進行訪問，由於跨站點的數據具備最終一致性，在有限可預期的時間內，用戶會獲取到最新的數據。AWS 在 Storage Day 上剛剛宣佈的“S3 Multi-Region Access Points”就是一例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，在協議層面，由 AWS 提出的 S3 協議已經是對象存儲事實上的通用協議，這個協議在設計之初就考慮到了雲存儲的場景，可以說對象存儲在協議層就是雲原生的協議，在數據接口的選擇和使用上更具靈活性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，對象存儲本身就是應雲存儲而生，一開始起家的用戶場景即爲二級存儲備份場景，本身就具備了低價的特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，對象存儲是雲時代的產物，支持原始數據存儲、分佈式可擴展、高靈活性、低價，都是對象存儲之所以被選擇的原因。可以預計在未來會有更多的數據業務完全基於對象存儲而構建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"數據湖存儲架構落地實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從技術上說，無論是數據倉庫、數據湖還是 S3 存儲，都是數據存儲的一種方式，所以從某種程度上來說，存儲行業正在迅速演變爲數據業務行業。在數據湖的探索上，"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"也分享了一些戴爾科技集團 OSA（Object, Stream and Analytics）團隊的實踐經驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"市場上的很多數據平臺公司還是利用 s3a 協議而非原生的 s3 協議來訪問對象存儲的服務。例如 Databricks 雖然在他們的論文中提到他們在對象存儲之上提供了一層表結構的服務，即 Delta Lake。但實際上 Delta Lake 與底層對象存儲交互的協議仍然用的是 s3a，即對象存儲版本的 HDFS 協議。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，從對象存儲的角度來看，s3a 協議由於多了一層協議轉換，並不能完全發揮高性能對象存儲的延時和吞吐量的優勢。因此，OSA 研發團隊認爲基於原生 s3 協議構建數據平臺的存儲訪問層應該能大大提升存儲層的訪問能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，對象存儲的部署方式也需要從單一的公有云變成混合雲 \/ 多雲的混合架構。“Latency matters”，如何在儘可能接近數據產生端進行實時、近實時處理是下一代數據湖需要直面的問題。更靈活的架構也會幫助企業級用戶打消對於數據平臺鎖定的擔憂（locked in\/lock down）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/58\/5835694d962a6dca0a4386312bb67a15.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新一代數據平臺的基本架構都是存算分離，即計算層和存儲層是松耦合的。計算層無狀態，所有的數據、元數據以及計算產生的中間數據都會存儲於存儲層之中。這一架構的優勢包括更好的擴展性（計算、存儲獨立擴展），更好的可用性（計算層的失效不影響存儲，因此能夠很快恢復），以及更低的成本。爲了適應存算分離的架構，對象存儲本身也需要進一步發展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這個發展趨勢中 OSA 的產品研發也經歷了從軟件定義（Software Defined）到雲原生（Cloud Native）的改造，來適應數據平臺的需要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“儘管我們從一開始就使用了 Scale-out 架構，基於標準 x86 服務器構建，所以很早我們就認爲自己已經做到了軟件定義。但是在實踐中，我們發現部署在服務器上的存儲產品還有複雜的運維問題、動態擴展問題等等，帶來的用戶體驗並不完美。與此同時，以容器、Kubernetes 爲代表的技術的出現，顛覆了原來的軟件定義存儲，可以讓存儲做到雲原生，從提供存儲產品轉型爲提供存儲服務（Storage as a Service）。Kubernetes 提供了強大的集羣管理功能，而容器提供了低開銷的資源隔離和控制方案。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"藉助於這兩大技術，OSA 研發團隊徹底革新了對象存儲的技術棧，用微服務的方式拆分巨型單體服務，將存算分離的架構應用到對象存儲內部。通過劃分各個存儲服務的邊界，制訂服務合約，將需要直接訪問磁盤 IO 的功能集中到一兩個服務中，實現了絕大多數服務的無狀態化。改造之後，對象存儲能夠輕鬆運行在資源相對有限的邊緣節點之上。元數據服務可以根據計算負載單獨擴容，而不受限於底層存儲資源。改造之後，對象存儲本身也是 Kubernetes 所託管的諸多服務之一，只要集羣中有足夠的計算和存儲資源，對象存儲就能根據 IO 負載來動態地 Scale up 或者 Scale down，達到完全的彈性，而這也是雲原生存儲的最終目標之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以"},{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"認爲，"},{"type":"text","marks":[{"type":"strong"}],"text":"想要適應存算分離的大趨勢，不是簡單地把現有存儲對接到計算層就可以完成的，存儲本身要經歷新一輪架構革命才能更好地服務於計算層"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"技術並非完美，當前對象存儲存在的四個挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在架構之外，數據平臺型業務也給對象存儲的特性提出了若干新的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第一個挑戰"},{"type":"text","text":"是數據分析型業務所需要的性能要遠高於數據備份的場景，對象存儲需要能夠提供與計算需求相匹配的大帶寬與低延時。閃存技術的發展、NVMe over Fabrics 標準的提出使得高性能對象存儲成爲可能。在選擇網絡的時候，建議選擇基於以太網的 RoCE 或者 TCP 協議作爲網絡傳輸層，能同時兼顧性能、成本和部署的靈活性。同時，要完全發揮 NVMe 閃存的高併發性，也需要對數據鏈路做徹底的異步化和無鎖化改造，並且 OSA 研發團隊傾向於選擇用戶態編程模型，因爲它對內核的依賴小，對雲原生環境下的複雜部署更友好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，在寫入模式上，雖然追加寫是在硬盤時代提出的思想，但在閃存時代依然是一個優秀的選擇。由於閃存的物理特性，閃存的固件需要對內部的塊做後臺的垃圾掃描和回收，一旦閃存找不到完全空閒的塊，那再次寫入就會需要經過讀取 - 修改 - 寫入的路徑，會大大影響閃存的寫入性能。追加寫的方式能夠減少碎片，減輕閃存後臺垃圾回收的壓力。同時，對於新興的低成本、高密度的 QLC 閃存而言，追加寫也是絕佳的搭檔。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，對象存儲還需要根據業務場景來優化性能。例如，高帶寬指的不僅僅是性能測試時能達到的帶寬，因爲傳統性能測試往往使用 GB 級別甚至更大的對象，達到高帶寬相對容易。然而，在數據分析的場景中，所要處理的是中等或者較大的對象（32Kb -1024Kb 級別），在這樣的業務場景中，很可能帶寬瓶頸不在底層的 NVMe 吞吐率上，而在元數據服務或者讀寫流水線的銜接上，這就需要依據數據分析的業務特徵在讀寫全鏈路上優化對象存儲的帶寬和延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第二個挑戰"},{"type":"text","text":"來自於數據分析所包含的衆多元數據操作。因此對象存儲不僅要能夠提供大帶寬，還要在處理小對象和元數據操作如 list 時提供足夠的性能。這就比較考驗對象存儲的元數據管理能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"認爲，“根據我們的經驗，基於 B-tree 的某些高效變種來存儲元數據在寫放大、讀延時等方面依然要優於 LSM tree。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工業界更常用的 Bw-tree 就是一例。Bw-tree 整體上仍然是一棵 B-tree，但是其中的每個節點則借鑑了 LSM tree 的 log structured 設計，每個節點有一個 base record 和一組 delta log。Bw-tree 還引入了 logical page 和 physical page 的概念，兩者通過一張 mapping table 進行關聯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這些設計，Bw-tree 能夠避免過於頻繁的 page split\/merge 操作以及 dirty page 向上的傳導性，提高了寫入的效率，又避免了 LSM tree 在後臺進行的大量分層 merge 操作。同時，整棵樹在點查詢和區域查詢的效率上依然接近傳統 B-tree，優於 LSM tree。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第三個挑戰"},{"type":"text","text":"是對象存儲如何兼顧性能和成本。數據湖中存儲了龐大的企業數據，但在任一時間點，可能只有一小部分數據是被數據分析業務所需要的。如果所有數據都放在性能最優的物理介質上（比如非易失性內存），那麼成本將變得過高，失去了雲存儲的經濟性，而如果在對象存儲的前端再加一層 cache 層，無疑也會增加整個系統的複雜度。因此如何有效識別冷熱數據，並將它們分區放置是對象存儲需要解決的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種簡單的方式，是把選擇權交給用戶。用戶通過設置一些固定的規則（如根據寫入時間、最後訪問時間等）觸發數據在不同介質之間的遷移。第二種方式，是通過對歷史訪問數據的分析，讓存儲系統來預測單個對象的生命週期，實現某種程度上的智能分層。更高級的方式，則是讓存儲系統能夠識別計算負載的模式特徵，根據模式特徵識別不同對象之間的關聯性，根據一組對象的訪問特點來決定分層策略。例如，如果某個工作負載的特徵是依次讀取某一組對象，那麼存儲層就可以提前預取將要被訪問的對象，來進一步加速計算。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"第四個挑戰"},{"type":"text","text":"是對象存儲如何與開源生態相結合。現階段比較成熟的在數據湖之上提供表結構的開源產品是 Delta Lake、Iceberg 和 Hudi。對這三款產品 OSA 研發團隊做了一些預研和實驗。Hudi 基於 Hadoop 生態系統，特別是和 Hive 有深度綁定。Delta Lake 的設計非常優秀，不過由於是 Databricks 的產品，它還有一個不開源的商業版，許多高級特性只有在商業版上才提供。同樣由於 Databricks 的關係，在計算層上，Delta Lake 和 Apache Spark 深度綁定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與前兩者相比，Iceberg 的接口抽象特別清晰，靈活性也可以完美適配，在計算層也可以跟 Apache Flink、Apache Spark 等計算層解決方案無縫集成。因此，作爲長期企業級對象存儲解決方案的提供者，在對這些開源方案進行對比研究後，在現階段 OSA 研發團隊選擇了 Apache Iceberg 作爲其開源解決方案的組成部分，未來也不排除將和更多的開源產品集成，爲客戶提供完整的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好地適配底層的對象存儲，OSA 研發團隊爲 Iceberg 做了一個通用的 S3 表管理組件（S3 Catalog）。在 S3 的標準 API 中，上傳數據需要預先知道對象的大小，因此在追加上傳的場景下，其調用方法無法像 HDFS 那樣簡潔。所以在具體實現中，追加寫的操作需要在本地預先處理，並以整體上傳。第二，利用第三方鎖的實現，解決了對象存儲中不支持原子重命名操作的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除此之外，OSA 研發團隊還提供了一個基於 Dell EMC 商業對象存儲 ECS 的專屬表管理組件（ECS Catalog）。ECS 支持 Append 語義，使用 Append 的操作可以完美應對順序寫入未知長度文件的場景。ECS 還支持類 compare-and-swap (CAS) 語義。在併發提交的場景下，ECS 支持使用 If-Match 和 If-None-Match 對對象進行 CAS 操作，來實現原子化重命名的操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"超越數據湖，展望新一代數據平臺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在"},{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"看來，下一代數據平臺有更廣闊的範疇和視角，超越當前人們對數據湖的理解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它將不光涵蓋傳統的數據中心或者雲端部署的範疇，結合日益發展的邊緣計算，將來的數據平臺將會涵蓋從傳感器到邊緣節點到數據中心到雲端一整套生態系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/7e\/7e86adeb5f2a09ba8b5b754b6b95228c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Source: VMware 調研白皮書《基於全棧雲基礎架構，構建企業級業務創新平臺》"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這套生態系統上，數據是主角。平臺的使用者會慢慢弱化對底層系統的依賴，而更多地關注於數據處理管道和基於業務邏輯的數據處理邏輯需求的定義。基於此，平臺對於使用者的專業性要求也會逐漸降低，越來越多的人能夠利用數據平臺從數據中獲取信息價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存算分離將是下一代數據平臺的標準架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲層將更統一，而計算層根據負載也將擁有更多的靈活性。和傳統數據湖相似，數據依舊會統一存儲在數據湖中，先存儲後消費。但是，對於數據的結構化定義要求會越來越不重要，數據清洗或結構化的轉化將更多地由平臺智能化或者根據更人性化的配置定義完成。至於數據在各個環節中的存儲，也將隱藏在平臺內部。數據孤島將逐漸消亡，原因很簡單，如果數據湖本身能夠提供足夠好的 OLAP 支持，爲什麼還要把數據拷貝一份再放到數據倉庫中呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在計算層面，數據平臺應該是完全開放的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據具體的業務需要，用戶可以自行選擇靈活度更高的 Spark 或者 Flink 計算框架，或者是集成度更好的一體化方案。下一代數據平臺也應該提供強大的跨表查詢能力。無論數據是直接存儲在對象存儲中、存儲在 Iceberg 等表結構中、還是存儲在外部的數據庫中，數據平臺都支持對這些表進行聯合查詢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存儲和計算之間會進化出新的數據抽象層（現在正在發生）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多統一的業界標準如 Iceberg 等，會越來越多地出現。數據產生者和數據消費者基於這層抽象層協同合作。通過這層數據抽象層，數據平臺會慢慢將各個角色的數據消費者從系統部署與理解的細節中解脫出來，以關注業務邏輯本身。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如 Iceberg 創始人之一 Ryan Blue 所說，數據平臺的使用者應該更多關心自己的數據和數據產品，把剩下的問題交給數據平臺基礎架構去解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時從應用場景上來說，在傳統的離線數據分析場景之外，實時數據分析的業務場景正在增加。以 Spark micro batch 爲代表的近實時框架可以解決一部分的業務需求，但對於延時要求更高的場景，實時分析的框架還有待發展。以流式存儲爲依託的實時分析框架依然是個尚待開發的領域。另外，目前的數據平臺對結構化和半結構化數據已經有了比較好的支持，但對於非結構化數據，如何爲它們建立高效的索引來加速查詢和分析，依然是業界懸而未決的難題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"認爲，未來的數據平臺應該是涵蓋傳感器、邊緣節點、數據中心和雲端，存儲計算相分離，依賴數據抽象層，爲多種多樣的數據產生和消費者，提供集成了離線分析、近實時分析和實時分析數據處理鏈路的統一平臺，提供更簡單和標準化的接口供用戶使用，讓用戶聚焦於從數據中獲取價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉賓簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"滕昱"},{"type":"text","text":"，Pravega 中國社區創始人，戴爾科技集團 OSA（Object, Stream and Analytics）軟件開發總監。自 2007 年加入戴爾科技集團中國研發集團以來，一直專注於分佈式系統領域，先後參與了前後 2 代 Dell EMC 對象存儲產品的研發及商業化工作，正在積極擁抱新的混合雲 \/ 多雲和實時流處理時代。從 2016 年開始，參與 Pravega 開源項目，活躍在多個開源社區。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"孫驁"},{"type":"text","text":"，技術團隊傑出成員（Distinguished Member of Technical Staff），負責戴爾科技集團分佈式對象存儲系統架構設計和開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"雷璐"},{"type":"text","text":"，高級主管工程師（Senior Principal Engineer），專注於戴爾科技集團流處理平臺 SDP（Streaming Data Platform）系統架構的設計和開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文選自《"},{"type":"link","attrs":{"href":"https:\/\/www.infoq.cn\/minibook\/LxX7bFUwKH17bzxQkSKt","title":"xxx","type":null},"content":[{"type":"text","text":"中國卓越技術團隊訪談錄"}]},{"type":"text","text":"》（2021 年第五季），掃描二維碼查看更多獨家專訪↓"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/e1\/e126ae89435157ceb3e1614581d39cca.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"《中國頂尖技術團隊訪談錄》品牌升級，現正式更名爲《中國卓越技術團隊訪談錄》"},{"type":"text","text":"，這是 InfoQ 打造的重磅內容產品，以各個國內優秀企業的 IT 技術團隊爲線索策劃系列採訪，希望向外界傳遞傑出技術團隊的做事方法 \/ 技術實踐，讓開發者瞭解他們的知識積累、技術演進、產品錘鍊與團隊文化等，並從中獲得有價值的見解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你身處傳統企業經歷了完整的數字化轉型過程或者正在互聯網公司進行創新技術的研發，並希望 InfoQ 可以關注並採訪你所在的技術團隊，可以添加微信：whitecrow-tina，請註明來意及公司名稱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章