騰訊雲基於Alluxio優化計算存儲分離架構的最佳實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、當前大數據挑戰    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近年來，隨着大數據規模的增長，以及大數據應用的發展，大數據技術的架構也在持續演進。早期的技術架構是計算資源和存儲資源高度融合，計算和存儲資源一體化存在以下明顯的挑戰："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1. 數據孤島"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今，企業擁有PB級數據已經成爲常態，EB級數據時代也將很快到來。企業需要面向結構化數據、非結構化數據、實時數據等多種類型的數據提供高擴展且統一的數據管理和數據存儲能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 剛性擴容"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在數據空間持續增長的背景下，大數據應用場景不斷增加，對企業算力的需求也在加劇提升。而同時，新品發佈、熱點事件等帶來的業務浪湧，也需要企業大數據系統擁有極致的彈性能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 利用率低"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大數據行業技術棧迭代迅速，企業自行構建IDC中心和自行部署軟件，一次性投資大，且折舊成本高，運營運維負擔沉重。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4. 作業擁塞"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着業務的發展，在數據量巨大的背景下，單次分析作業常需要讀取TB-PB級的數據，多任務併發下，極易出現作業擁塞。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面對以上挑戰，傳統的以私有數據中心爲基礎的存算一體大數據架構，已無法滿足企業海量數據分析的需求。業界知名分析機構IDC在最新的報告中明確指出：企業上雲已成必然趨勢。因此，在公有云上部署更靈活高效的大數據分析平臺，將成爲企業的必然選擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、騰訊雲彈性MapReduce(EMR)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前越來越多的企業開始選擇使用計算和存儲分離的架構，以應對更低成本的要求，和兼顧資源擴展的靈活性。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9d334dbb5bafc7fa57dbf5e7cd6393e0.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"傳統計算存儲一體架構"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/65\/65d0c6b74d2ead074e5c8ef905a9878e.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"計算存儲分離架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前"},{"type":"text","marks":[{"type":"italic"}],"text":"騰訊雲彈性MapReduce(EMR)[1]"},{"type":"text","text":"支持了三種存儲系統：EMR-HDFS、"},{"type":"text","marks":[{"type":"italic"}],"text":"EMR-COS[2]"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"italic"}],"text":"EMR-CHDFS[3]"},{"type":"text","text":"，其中EMR-COS和EMR-CHDFS在EMR中都是開箱即用的原生支持計算存儲分離的方案，其具體應用場景及特點如下："}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

特點	EMR-HDFS	EMR-COS	EMR-CHDFS
存儲空間	集羣規模相關	海量	海量
可靠性	高	高	高
元數據效率	快	慢	快
彈性效率	中	高	高
數據本地化	高	低	低
帶寬成本	低	高	高
網絡風暴	低	高	中"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"元數據操作效率高，能夠與HDFS相當，能夠有效規避COS文件系統元數據操作耗時以及高頻訪問下可能引發不穩定的問題。但在實際使用場景中，因爲可能存在多個數據存儲源管理複雜，部分業務場景對數據源的IO訪問密集造成網絡壓力大，訪問不穩定等問題。所以我們基於Alluxio進一步優化計算和存儲架構，更好的滿足業務應用上的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、基於Alluxio優化計算存儲分離架構 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統計算存儲分離，解決了計算量和存儲量不匹配問題， 實現了算力的按需使用，大幅節省了運維規劃時間以及閒置的算力成本。但直接使用計算存儲分離架構，也引入了新的問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在IO密集型的場景下，網絡帶寬會成爲瓶頸, 可能導致計算 & 存儲資源利用不充分；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據本地化不夠，導致很多shuffle過程的重複計算，造成部分浪費計算資源的浪費；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可能存在多種甚至異構的存儲源，增加了管理難度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲此，騰訊雲EMR團隊與Alluxio社區合作，引入最新alluxio2.3.0 Release版本進行深度優化，推出開箱即用的計算存儲分離優化版本：EMR2.5.0\/EMR3.1.0\/EMR-TianQiong-1.0，解決上述問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1. 提供內存級 I\/O能力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Alluxio能夠用作分佈式共享緩存服務，這樣與Alluxio通信的計算應用程序可以透明地緩存頻繁訪問的數據（尤其是從遠程位置），以提供內存級 I\/O 吞吐率。此外，Alluxio的層次化存儲機制能夠充分利用內存、固態硬盤或者磁盤，降低具有彈性擴張特性的數據驅動型應用的成本開銷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 提高數據本地性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用Alluxio提供的分佈式緩存服務，在部署Alluxio數據節點(Alluxio-Worker)時和計算節點部署在一起，可以直接從數據節點中以內存級IO速度檢索讀取數據，而不是從底層雲存儲或對象存儲中檢索讀取，提高了數據本地性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 簡化雲存儲和對象存儲接入"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與傳統文件系統相比，雲存儲系統和對象存儲系統使用不同的語義，這些語義對性能的影響也不同於傳統文件系統。在雲存儲和對象存儲系統上進行常見的文件系統操作（如列出目錄和重命名）通常會導致顯著的性能開銷。當訪問雲存儲中的數據時，應用程序沒有節點級數據本地性或跨應用程序緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4. 簡化數據管理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Alluxio 提供對多數據源的單點訪問。除了連接不同類型的數據源之外，Alluxio還允許用戶同時連接同一存儲系統的不同版本，如多個版本的HDFS以及雲上COS\/CHDFS，只需基於EMR配套的簡單配置下發和管理管理功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在引入Alluxio後，EMR基於Alluxio的存算分離的整體架構變成了："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/76\/76445db1bde083a65d1be17537fb7946.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣，EMR的計算引擎(Spark，MapReduce，Presto等)就可以統一通過Alluxio來提升性能，降低網絡峯值帶寬，以及簡化數據管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、性能評估及調優"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了分析理解使用Alluxio存儲在主流查詢引擎Spark性能上差異，我們使用大數據壓測工具TPC-DS進行了一些性能壓測。我們使用的環境及配置如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EMR版本：EMR-2.5.0；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"選擇組件：zookeeper-3.6.1,hadoop-2.8.5,hive-2.3.7,spark_hadoop2.8-3.0.0,tez-0.9.2,alluxio-2.3.0,knox-1.2.0；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"壓測配置，使用了1個EMR的Master節點和25個CORE節點，具體如下："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/80\/80bce91e8bfde2d94ea7426e97d007c9.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1. 帶寬評估"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/87\/8765f5d314ef874443a0781f2a900097.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a7\/a713446eaed95c74c569d41b785ce9c6.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從壓測結果可以看到，能大幅優化計算存儲分離網絡帶寬，節省峯值帶寬(削峯)20%-50%，節省總帶寬(10%-50%)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2. 查詢性能評估"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/af\/af746ad26f9c4a2576516806dbd1c153.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/04\/043937bb20d2c32ee4a0f1eb4e520546.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從壓測結果可以看到,在大部分場景下能優化性能，特別是IO密集型，優化性能5%-40%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3. 性能調優及專項優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好滿足計算存儲分離場景，EMR團隊針對Alluxio做了專項調優，具體包括："}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"（1）數據本地性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好滿足數據本地，EMR在部署Alluxio時，在core節點把alluxio-worker同計算節點部署在一起，這樣yarn等計算服務節點可以在同一個節點中與alluxio-worker節點通信，大量提升了效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，結合alluxio已經提供的讀寫策略，結合存算分離場景優化了block.read.location.policy，writetype.default等策略，讓alluxio的緩存能力更好滿足本地性。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"（2）元數據優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Alluxio基於Presto實現了Catalog Service，並且實現了計算框架端的Connector，Alluxio可以感知並管理結構化數據的元數據，大大簡化表級別的使用成本。同時，騰訊內部在大規模使用Alluxio時，我們發現Alluxio本身的inode元數據也面臨着膨脹的風險。爲此結合Alluxio提供的Catalog Service和Path緩存能力，優化了path.caching.thread和path.cache.capacity等策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多meta具體優化可參考，社區"},{"type":"text","marks":[{"type":"italic"}],"text":"meta優化[4]"},{"type":"text","text":"及"},{"type":"text","marks":[{"type":"italic"}],"text":"catalog介紹[5]"},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"（3）Java GC的影響"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Alluxio作爲Java的進程，其GC的經常影響其性能表現，爲此，EMR團隊引入了 Tencent Kona，經過了內部大數據和AI等業務場景的驗證，爲JAVA生態提供專業持續的保障。Kona在GC線程調度優化，物理內存釋放優化等方面有優秀表現，更多功能特性可見："},{"type":"text","marks":[{"type":"italic"}],"text":"Kona JDK[6]"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述的這些能力和優化，在存算分離場景下，騰訊雲EMR產品針對這種場景都已經直接提供了開箱即用的能力，直接在騰訊雲EMR產品購買頁創建，或者在已有支持了alluxio的EMR版本上安裝，即可達到性能評估中效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上述的壓測結果看到，一方面有效的降低了帶寬峯值和總帶寬，從而降低帶寬成本，加速訪問；另一方面，IO密集型場景下的性能也有不少提升，能更好的支持IO密集型場景下的業務。此次基於Alluxio的優化，讓騰訊雲彈性MapReduce(EMR)產品更好的支持存儲計算分離架構，爲用戶更好的滿足業務需求的同時，降低成本，且保持資源擴展的靈活性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"頭圖：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者：騰訊雲大數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：https:\/\/mp.weixin.qq.com\/s\/WUcuGRJwOnlfNW4XjS0JiA"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：騰訊雲基於Alluxio優化計算存儲分離架構的最佳實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來源：雲加社區 - 微信公衆號 [ID：QcloudCommunity]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"轉載：著作權歸作者所有。商業轉載請聯繫作者獲得授權，非商業轉載請註明出處。"}]}]}