存儲大師班 | NFS 的誕生與成長

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"​我們爲什麼需要 “網絡文件協議”","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲文件是大家日常工作生活中最常見的需求，隨着文件數量和佔用存儲空間的上升，以及在一定範圍內共享訪問文件的需求產生，我們自然需要把存儲文件的工作從單個計算機設備中剝離出來，作爲一個單獨的服務資源(或物理硬件)來對外提供存儲功能，提供更大的容量的同時，爲多個終端通過網絡共享訪問。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏提到的存儲設備就是我們常說的 NAS（Network Attached Storage：網絡附屬存儲）。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/df/df8c416be2fa93e4e692caca571c2d44.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"常見 NAS 架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而終端通過網絡來共享訪問 NAS，需要標準的協議規範。NFS 就是其中最重要也是應用最廣泛的協議標準之一（其它流行的網絡文件協議還有 SMB[1] 協議，後續會有專門的文章進行介紹，盡請期待）。今天我們就來聊聊 NFS 協議。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"不斷髮展的 NFS","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"簡單好用的無狀態協議誕生","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFS 第一次出現在我們的視野中，是在 1985 年。NFS version2 作爲 SunOS 2.0 的組件正式發佈。所以在當時把它叫做 “Sun 網絡文件系統” 可能更爲貼切。另外，你沒有看錯，第一個正式對外發布的版本就是 v2 版本（v1 版本從未對外正式發佈，這也是我們從不討論 NFS v1 的原因）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不過 NFS 並不是最早的網絡文件系統，彼時已經有一些更早的網絡文件系統存在，比如 UNIX SVR3 系統中包含的 RFS(Remote File System)[2] RFS 已經引入了 RPC（Remote Procedure Call）概念，併成爲 NFS v2 的借鑑的範本。但 RFS 也有自己明顯的問題，比如其爲每個客戶端打開文件記錄狀態（即有狀態協議），所以很難應對服務端宕機或重啓的情況。NFS v2 爲了在設計層面很好的解決 RFS 的缺陷，設計成了一個完全無狀態的協議。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1995 年，NFS v3 正式發佈。此時 NFS 協議的開發已經不再完全依賴 Sun 公司，而是多家公司共同主導完成. NFS v3 包含衆多優化，但大多數可以認爲是性能層面的優化。總體看來，NFS v3 仍然遵循無狀態協議的設計。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無狀態協議的設計，自然是降低了應對服務端宕機重啓情況的處理難度。但想要徹底掙脫“有狀態信息”的束縛並不容易。比如，作爲網絡文件協議，就需要支持 “文件鎖” 操作，但鎖信息天然就是一種“狀態信息”。所以在 NFS v2/v3 運行的環境中，NFS 將這部分負擔 “外包給”了 NLM（Network Lock Manager）。當 NFS Client 接收到一個文件鎖請求時，會產生一個 NLM 協議的 RPC 調用，而不是產生一個 NFS 協議的 RPC 調用。但這樣 NLM 就成爲了一個 “有狀態” 協議。所以它需要處理出現服務端崩潰、客戶端崩潰、網絡分區出現後的故障恢復問題。NFS v2/v3 協議與 NLM 的配合工作總體來說不夠協調（比如 NLM 協議本身會標記和識別每把鎖由哪個進程申請和持有，但 NFS server 處理讀寫請求時，卻無法區分請求來自於哪個遠端進程），這也導致了難以完美的實現鎖邏輯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即便拋開文件鎖的問題不談，無狀態協議設計本身也帶來了新的問題。NFS 服務作爲“無狀態服務”，無法記錄各個 NFS 協議客戶端打開文件的狀態，也就沒有簡單直接的辦法判斷文件內容是否已經被其它客戶端修改，即 cache 是否還有效。在 NFS v2/v3 協議中，NFS 協議客戶端通常將文件的修改時間和文件大小保存在 cache 詳細信息中，以一個時間間隔，定期對 cache 進行有效性驗證：NFS 協議客戶端獲取當前的文件屬性，和 cache 中的修改時間及文件大小做比較，如果仍一致，則假定文件沒有修改過，cache 仍然有效；如果不匹配，則認爲文件發生了了變化，cahce 不再有效。這樣的方式顯然是低效的。而更不幸運的是，由於很多文件系統保存時間戳的精度不足，NFS 協議客戶端無法探測出那些在一個較粗精度的時間單位（秒）中連續的修改，比如剛校驗過 cache 有效性後，同一秒內又發生了更改（覆蓋寫），那麼從修改時間和文件大小都觀察不出需要重新更新cache。這種情況下，只有該 cache 被 lru 驅逐，或者文件後續被再次修改，否則 NFS 協議客戶端無法感知到文件的最新變化。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"無狀態到有狀態，進化爲成熟單機網絡文件系統","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2002 年 NFS v4.0 版本發佈。此時 NFS 協議的開發完全由 IETF 主導，最大的一個變化就是 NFS 協議的設計從無狀態協議變爲有狀態協議。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從無狀態協議再次演變爲有狀態協議，並不是追求一種 old fasion，而是因爲當前已經具備了更好的工程能力，來設計開發出足以應對有狀態協議設計下複雜問題的機制（當然，這背後的動力顯然也是拜常年忍受無狀態設計中的種種缺陷之苦所賜）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFS v4.0 的有狀態設計主要體現在如下幾個方面：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. 協議自身加入了文件鎖功能，會維護鎖信息這樣的狀態信息，不需要 NLM 協助。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. 在 cache 一致性問題的處理上，NFSv4 支持了 delegation 機制。由於多個客戶端可以掛載同一個文件系統，爲了保持文件同步，NFSv4可以依靠 delegation 實現文件同步。當客戶端 A 打開一個文件時，NFS 服務端會分配給客戶端 A 一個 delegation。只要客戶端 A 持有 delegation，就可以認爲與服務端保持了一致，可以放心的的在 NFS 協議客戶端側做緩存等處理。如果另外一個客戶端 B 訪問同一個文件，則服務端會暫緩處理（即短暫阻塞）客戶端 B 的訪問請求，並向客戶端 A 發送 RECALL 請求。當客戶端 A 接收到 RECALL 請求後，會將本地緩存刷新到服務端中，然後將 delegation 歸還給服務端，這時服務端開始繼續處理客戶端 B 的請求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然，delegation 機制僅能理解爲在考慮緩存一致性的情況，以一種更加激進的方式進行讀寫處理，所以該機制更應該被理解爲是一種性能效率優化，而不是完全解決 cache 一致性問題的方案，因爲當 NFS 服務端發現多個客戶端對同一文件的競爭出現，並回收之前發放的授權後，又會回退到跟 v2/v3 版本中相似的機制去判斷 cache 的有效性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上述兩點，v4.0 版本相較之前的版本還有以下優化：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. NFSv4 增加了安全性設計，開始支持 RPC SEC-GSS[3] 身份認證。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. NFSv4 只提供了兩個請求 NULL 和 COMPOUND，所有的操作都整合進了 COMPOUND 中，客戶端可以根據實際請求將多個操作封裝到一個 COMPOUND 請求中，增加了靈活性的同時減少了交互次數，大大提高了性能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"c. NFSv4 版本中修改了文件屬性的表示方法，顯著增強對 Windows 系統的兼容性，而幾乎同時，微軟開始把 SMB 協議重塑成 CIFS（Common Internet FileSystem），這看起來絕非巧合，可見兩者的競爭意味。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過上面這些改進，可以說 NFS 已經進化成爲了成熟高效的單機網絡文件系統，但是軟件的世界已經慢慢向文件系統提出了更多擴展性的需求和更多的企業級特性需求，NFS v4.0 版本對此還未給出答案。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"具備更強擴展性，企業級集羣文件系統雛形已現","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2010 年及 2016 年, NFS v4 的演進版本 v4.1 和 v4.2 陸續發佈。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2010 年，NFS v4.1 的問世，讓 NFS 向集羣文件系統的方向邁出了重要一步 --- 因爲其引入了並行文件系統的概念（Parallel NFS/pNFS）：即在協議層面將元數據與數據分離，創造出元數據節點和數據節點的角色，對數據的訪問具備了一定擴展性。並行訪問數據的設計也讓整體吞吐提升到新的高度，這與很多現代分佈式文件系統思路相似。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但需要指出的是，在此設計中，元數據的處理擴展性仍未得到解決。另外作爲有狀態協議，在用來保證高可用性的主備架構中，由於備節點中並沒有主節點中維護的狀態信息，所以故障切換過程很難做到足夠平滑。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/01/01a48f43447bc2aa1df3da34598a3741.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Parallel NFS基本架構圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除此之外，NFS 協議開始加入了更多的數據中心級企業級特性：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFSv4.1 開始支持 RDMA（Remote Direct Memory Access）[4]，並在 NFS v4.2 中開始支持稀疏文件（sparse file）以及支持 server 側拷貝（Server-Side Copy）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這都幫助 NFS 協議可以更好地支撐更加嚴肅的數據中心/企業級應用。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"NFS 的繼續進化和工程層面的探索：內核態 vs 用戶態","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFS 協議不斷的發展，在複雜性不斷提高的同時，也在集羣擴展性/可用性方面不斷探索，但這無疑也給工程實現層面提出了新的挑戰。在 Linux 世界中，最常使用的就是內核態的 nfsd 服務，但是隨着機制和架構的複雜性的增加，在用戶態去實現一個 NFS 服務似乎成爲了一個工程更合理的方案。作爲 NFS 在 Linux 世界中的競爭對手，Samba[5] 服務就是基於用戶態打造了一套較爲完整的集羣方案。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這一問題，開源社區也給出了一些探索，其中最具影響力同時也是應用最廣泛的要數 nfs-ganesha[6] 項目，該項目目前由 RedHat 維護，其在用戶態實現了完整的 NFS 服務功能。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/87/87a28a59afbf49131ace335b9cc56534.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"NFS Ganesha 架構圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖 nfs-ganesha 架構圖所示，該項目本身專注於協議處理邏輯（支持所有的 NFS 協議版本），同時設計獨立的 SAL（State abstraction layer）抽象層，和 FSAL（File-System abstraction layer）抽象層。前者有助於在處理有狀態協議的“狀態”時擴展一些集羣邏輯，而後者方便接入包括本地 VFS 文件系統和各種開源/商業分佈式存儲。這樣的設計有利於藉助外部中間件和服務來打造更好的集羣邏輯，爲更多開放性的設計提供了空間，同時也通過對接更多的後端存儲系統來保證社區活力和擴大應用範圍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在未來相當長的一段時間裏，NFS 協議仍會承擔重要的角色，而 NFS 的工程實踐想要進一步獲得發展，滿足日漸膨脹及複雜的需求，仍需要在用戶態繼續探索。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"結束語","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFS 協議不但是我們平時共享文件的背後功臣，也在超算和廣電等行業支撐着各類核心業務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲一個有多年曆史的網絡文件協議，NFS 有其歷史侷限性，甚至每次迭代都有其沉重的歷史包袱捆綁手腳，但它仍可以被當作文件系統的經典範例去研究。可以說對 NFS 協議瞭解逐步深入的過程，也是對現代分佈式文件系統重新理解的過程。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"參考資料","attrs":{}}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.kernel.org/doc/ols/2006/ols2006v2-pages-59-72.pdf","title":"","type":null},"content":[{"type":"text","text":"Why NFS Sucks (Olaf KirchSUSE/Novell, Inc)","attrs":{}}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://nfsworld.blogspot.com/2006/10/review-of-why-nfs-sucks-paper-from.html","title":"","type":null},"content":[{"type":"text","text":"Review of \"Why NFS Sucks\" Paper from the 2006 Linux Symposium","attrs":{}}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch11_02.htm","title":"","type":null},"content":[{"type":"text","text":"NFS and file locking","attrs":{}}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://blog.csdn.net/ycnian/article/details/8515517","title":"","type":null},"content":[{"type":"text","text":"NFS各個版本之間的比較（ycnian）","attrs":{}}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/nfs-ganesha/nfs-ganesha/wiki/NFS-Ganesha-Architecture","title":"","type":null},"content":[{"type":"text","text":"NFS Ganesha Architecture","attrs":{}}]}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"引用鏈接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[1] ","attrs":{}},{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Server_Message_Block","title":"","type":null},"content":[{"type":"text","text":"SMB","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[2] ","attrs":{}},{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Remote_File_Sharing","title":"","type":null},"content":[{"type":"text","text":"RFS (Remote File System)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[3] ","attrs":{}},{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/RPC_SEC-GSS","title":"","type":null},"content":[{"type":"text","text":"RPC SEC-GSS","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[4] ","attrs":{}},{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Remote_direct_memory_access","title":"","type":null},"content":[{"type":"text","text":"RDMA","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[5] ","attrs":{}},{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Samba_(software)","title":"","type":null},"content":[{"type":"text","text":"Samba","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[6] ","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/nfs-ganesha/nfs-ganesha","title":"","type":null},"content":[{"type":"text","text":"nfs-ganesha","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#00a971","name":"user"}}],"text":"更多文章","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?__biz=Mzg3OTU2MzU2NA==&action=getalbum&album_id=1929382764454133767&scene=173&from_msgid=2247484203&from_itemidx=1&count=3&nolastread=1&uin=&key=&devicetype=Windows+10+x64&version=6303005d&lang=zh_CN&ascene=1&session_us=gh_811f3d01beb2&fontgear=2","title":"","type":null},"content":[{"type":"text","text":"存儲大師班","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s/Oz-5FsFRwvUEk2aYUyIWww","title":"","type":null},"content":[{"type":"text","text":"從應用接口視角看存儲系統","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s/OS-pDXEohthTloCuCWmZNg","title":"","type":null},"content":[{"type":"text","text":"如何打造雲原生時代的存儲基石","attrs":{}}]}]}]}