機器學習平臺統一化分佈式存儲Ceph的進階優化

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自網易雲音樂機器學習平臺上線以來，就承擔了音樂內部推薦、搜索、直播、社交、算法工程等各個業務團隊機器學習場景的需求， 這其中也遇到了很大的挑戰，尤其是在分佈式存儲這塊上，團隊花費大量時間、精力，解決其中的核心問題。本文是"},{"type":"text","marks":[{"type":"strong"}],"text":"網易數帆存儲團隊與網易雲音樂機器學習平臺與框架團隊"},{"type":"text","text":"聯合創作，向各位看官描述下，在機器學習場景，如何利用Ceph作爲統一化的分佈式存儲，並基於此進行的相關的優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/8b\/fc\/8bb3259d3b4b64d46361d62652cd53fc.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、業務背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着互聯網業務的飛速發展，大數據場景下的數據挖掘、算法模型，爲業務發展提供精準的個性化能力，在這個過程中，ML Infrastructure再度被人提起，以支持數以千億計的個性化請求。其中，分佈式存儲在ML Infrastructure充當十分重要的角色，並且在不同場景中，爲保證ML Infrasturecture的順暢運行，面臨不同挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以網易雲音樂（以下簡稱雲音樂）爲例，考慮到算法業務飛速發展，我們在2019年開始籌建的Goblin機器學習平臺以降本增效爲目標，整合了公司的相關資源，通過引入K8s系統提供彈性計算能力，引入CephFS提供共享分佈式存儲，爲各部門的業務工程師們提供統一的ML Infrastructure。我們基於CephFS實現核心功能如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲開發環境"},{"type":"text","marks":[{"type":"strong"}],"text":"提供彈性存儲"},{"type":"text","text":"。算法、工程人員在計算平臺上申請存儲資源存儲代碼、模型、訓練樣本，同時可以將相同的卷掛載到調試環境進行調試，而無需數據拷貝；特別是分佈式開發調試，在引入CephFS之前，模型調試，特別是分佈式訓練的調試是十分痛苦的，需要將模型、樣本數據分發到不同機器，訓練結束還需要從各個開發機上將日誌，模型等收集到開發機進行分析；引入CephFS之後，只需要爲分佈式訓練任務掛載相同的數據卷，就可以直接進行調試，而訓練過程中產生的日誌、模型在開發機上都是即時可見的，這極大節省調試時間；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"簡化數據治理"},{"type":"text","text":"。在模型開發流程中，有很大一部分時間在進行數據治理；比如大量視頻進行抽幀；引入CephFS，可以將這些治理好的數據，直接作爲數據卷掛載到訓練任務，避免了繁瑣的數據分發流程，並可以實現數據治理和訓練分離，並行處理；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"極大簡化調度系統"},{"type":"text","text":"。在ML Pipeline中，算法人員在模型開發調試結束後，可以直接將模型所在數據卷配置到調度系統進行調度，極大簡化模型上線流程。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、痛點分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CephFS爲機器學習平臺提供了彈性的、可共享的、支持多讀多寫的存儲系統，但開源CephFS在性能和安全性上還不能完全滿足真實場景需求："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據安全性是機器學習平臺重中之重功能，雖然CephFS支持多副本存儲，但當出現誤刪等行爲時，CephFS就無能爲力了；爲防止此類事故發生，要求CephFS有回收站功能；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集羣有大量存儲服務器，如果這些服務器均採用純機械盤，那麼性能可能不太夠，如果均採用純SSD，那麼成本可能會比較高，因此期望使用SSD做日誌盤，機械盤做數據盤這一混部邏輯，並基於此做相應的性能優化；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲統一開發環境，便隨着大量的代碼編譯、日誌讀寫、樣本下載，這要求CephFS既能有較高的吞吐量，又能快速處理大量小文件。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對以上關鍵性問題，雲音樂Goblin機器學習團隊和網易數帆（以下簡稱數帆）存儲團隊進行了多次探討與溝通，開始共建基於CephFS的ML Dev&Ops實踐。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、深度優化實踐"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"改進一：設計並實現基於CephFS的防誤刪系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前CephFS原生系統是沒有回收站這一功能的，這也就意味着一旦用戶刪除了文件，那麼就再也無法找回該文件了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"衆所周知，數據是一個企業和團隊最有價值的無形核心資產，有價值的數據一旦遭到損壞，對一個企業和團隊來說很可能是滅頂之災。2020年，某上市公司的數據遭員工刪除，導致其股價大跌，市值蒸發幾十億港元，更嚴重的是，合作伙伴對其信任降到了冰點，其後續業績也遭到了巨大打擊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，如何保障數據的可靠性是一個關鍵問題。但是，CephFS這一開源明星存儲產品恰恰缺少了這一環。同時，雲音樂Goblin平臺項目組也多次反饋他們有誤操作後導致文件丟失的經歷。於是，防誤刪功能作爲數帆存儲團隊與雲音樂共建項目中的重點被提上了日程。經過團隊的攻堅，最終實現了回收站這一防誤刪功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新開發的回收站在CephFS中初始化了trashbin目錄，並將用戶的unlink\/rmdir請求通過後端轉換成了rename請求，rename的目的地就是trashbin目錄。保證了業務使用習慣的一致性和無感。 回收站保持逾期定期清理的機制。恢復上，通過構建回收站內相關文件的目錄樹，然後rename回收站內的文件至目標位置來進行恢復。該問題的難點在於如何在分佈式場景下保證文件系統的目錄樹。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"改進二：混合存儲系統的性能優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過長時間觀察分析雲音樂Goblin平臺集羣的io狀態，我們發現該集羣的IO模型存在短時間的壓力突增情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，對於用戶來說，其最關注的就是成本以及AI任務訓練時長（存儲IO時延敏感）。所以基於此我們與Goblin平臺深入合作，並基於目前現狀：對於公司內外部用戶，如果是追求性能的用戶，數帆存儲團隊這邊提供的是全閃存盤的存儲系統；如果是追求成本的用戶，數帆存儲團隊這邊提供的是全機械盤的存儲系統。因此欠缺一種兼具成本與性能的存儲系統方案，爲此我們研發了這套基於CephFS的成本與性能領先業界水平的混合存儲系統。該系統架構如下："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b4\/b47c2fb1b29e50a2700fa422146debc9.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該架構也算是業界較常用的架構之一，但是有一個問題制約該混部架構的發展，即直接基於Ceph社區原生代碼使用該架構，性能只比純機械盤的集羣好一倍不到。因此，數帆存儲團隊對Ceph代碼進行了深度分析與改造，最終攻克了影響性能的兩個關鍵瓶頸點：重耗時模塊影響上下文以及重耗時模塊在IO核心路徑，如下圖標紅所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1b\/1b94abe01b553a099871172957983ca3.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過"},{"type":"link","attrs":{"href":"https:\/\/link.zhihu.com\/?target=https%3A\/\/my.oschina.net\/u\/4565392\/blog\/5130265","title":null,"type":null},"content":[{"type":"text","text":"數帆存儲團隊的性能優化"}],"marks":[{"type":"underline"}]},{"type":"text","text":"之後，該混部系統性能相較於社區原生版本有了顯著提升，在資源充足的情況下，IO時延以及IOPS等性能指標有七八倍的提升，當資源不足且達到限流後，性能也有一倍以上的提升。具體性能指標如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/66\/6630191d6a155f6931875126c3a23577.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d7\/d730bf2bc77c7535550e1a8a1792840a.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"改進三：設計並實現了基於CephFS的全方位性能優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CephFS作爲基本的分佈式存儲，簡單易用。但是在很多場景下存在着性能問題。比如業務代碼、數據管理、源碼編譯造成的卡頓、延遲過高；比如用戶刪除大目錄耗時非常久，有時候甚至要達到數天；又比如因多用戶分佈式寫模型導致的共享卡頓問題。這些問題嚴重影響着用戶的使用體驗。因此，數帆存儲團隊對性能問題進行了深入研究與改進，除了上面提到的在混合盤場景下的性能優化，我們在CephFS元數據訪問以及大文件刪除等多方面都進行了性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在大目錄刪除方面"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們開發了大目錄異步刪除功能：用戶在日常業務中，經常會遇到需要刪除大目錄情況。這些目錄一般包含幾千萬個文件，總容量在數個TB級別。現在用戶的常規方式是使用Linux下的rm -rf 命令，整個過程耗時非常久，有時甚至超過24小時，嚴重影響使用體驗。因此，用戶希望能提供一種快速刪除指定目錄的功能，且可以接受使用定製化接口。基於此，我們開發了大目錄異步刪除功能，這樣使得大目錄的刪除對用戶來說可以秒級完成。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在大文件IO方面"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們優化了大文件寫性能，最終使得寫帶寬可以提升一倍以上，寫延時可以下降一倍以上，具體性能指標如下："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f3\/f3bd066451fa0ce31f5c5b05bd7a991b.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/33\/33825b0dfc705b999c59011b2933b71c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在優化用戶開發環境git和make編譯等都很慢方面"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶在容器源碼目錄中使用git status非常慢，耗時數十秒以上，同時，使用make編譯等操作也異常慢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於該問題，杭州存儲組對該問題進行了細緻分析：通過strace跟蹤簡單的git status命令發現，流程中包含了大量的stat, lstat, fstat, getdents等元數據操作，單個syscall的請求時延一般在百us級別，但是數千個（對於Ceph源碼項目，大概有4K個）請求疊加之後，造成了達到秒級的時延，用戶感受明顯。橫向對比本地文件系統（xfs，ext4），通常每個syscall的請求時延要低一個數量級（十us級別），因此整體速度快很多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"進一步分析發現，延時主要消耗在FUSE的內核模塊與用戶態交互上 ，即使在元數據全緩存的情況下，每個syscall耗時依然比內核態文件系統高了一個數量級。接下來數帆存儲團隊通過把用戶態服務轉化爲內核服務後，性能得到了數十倍的提升，解決了用戶卡頓的這一體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"元數據請求時延方面"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經數帆存儲團隊分析發現，用戶的很多請求時延較高原因是open，stat等元數據請求時延較高，因此，基於該問題我們採用了多元數據節點的方案，最終使得元數據的平均訪問時延可以下降一倍以上，下圖分別是優化前和優化後的元數據訪問時延："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/73\/7311fc528a5f74bf074d07d1b8361f36.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/25\/2516997e291d42dbda1e73bdd68a2ae7.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、業務應用效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5d\/5de744edd93d142badb9156562c307b1.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖爲網易雲音樂整體數據架構，而云音樂機器學習平臺Goblin作爲其中一個重要的組件服務於網易雲音樂內部包括推薦、搜索、直播、社交、播客等核心業務場景，提供包括開發鏡像環境定製、模型代碼開發、模型服務開發、模型任務調度、模型相關文件線上服務推送等在內機器學習端對端開發流程，如下圖所示：CephFS作爲其核心分佈式存儲，提供平臺統一的文件存儲、訪問、共享的功能，具體使用場景包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"統一集成開發環境：通過自定義鏡像提供涉及到算法生命週期中的模型開發、線上服務開發等核心過程，Ceph與用戶以及所屬團隊綁定提供數據存取功能，通過優化Ceph小文件性能，能很好地支持大型項目編譯、版本管理等功能；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"開發與調度共享文件存儲：通過用戶以及所屬團隊與CephFS的用戶卷的綁定，支持一鍵調度開發任務，保證運行環境、運行代碼、數據的一致性；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"大規模分佈式任務中，不同角色的容器可掛載統一Ceph卷，達到運行日誌、代碼、數據的統一管理。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/79\/793809fe37d3c4e8514de3420a932be8.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前網易雲音樂Goblinlab機器學習平臺，平均每天處理2000+個線上調度任務，存儲池容量1PB+, 數百個存儲卷；每日平均IOPS數萬，帶寬GB+ 。經過優化，累積節省模型訓練時間上千小時。除此之外， CephFS混合盤場景的使用在降本增效方面具有重要價值，按照當前ML集羣PB級別的存儲量，可以爲業務節省數十萬的成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多Ceph相關文章見網易數帆存儲團隊《"},{"type":"link","attrs":{"href":"https:\/\/www.zhihu.com\/column\/c_1267088333848641536","title":null,"type":null},"content":[{"type":"text","text":"聊聊Ceph(分佈式存儲)"}]},{"type":"text","text":"》"}]}]}]}