共享存儲設備，如何實現差異化IO分配策略

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着5G、大數據、區塊鏈、物聯網以及人工智能等技術在各行業不斷滲透，企業對於數據存儲及實時處理的需求日益提高，現代數據中心的規模也隨着應用服務需求的增長而越來越大，簡單通過採購更多服務器擴展集羣會引起更高的運營成本與電力消耗，而且數據中心資源利用率低、利用率不夠彈性已逐步成爲雲計算進一步發展的制約因素。爲了降低採購成本，目前業務廠商比較行之有效的方式是通過業務混部來提升硬件（CPU和存儲）資源利用率。這就需要NVMe SSD存儲設備具備像多命名空間和加權輪詢技術，實現在共享存儲設備資源時，根據業務的優先級，提供差異化IO分配策略。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"集羣負載和業務混部","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現代數據中心集羣的負載可以簡單分爲兩類，一類是延時敏感的在線服務，一旦服務崩潰或者延時過大，將導致用戶體驗不佳，需要留有較多的資源餘量以應對突發的請求高峯；另一類是離線批處理作業，雖然資源消耗較大，但是對延時相對不敏感，可以容忍一定範圍內執行時間的調整。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此業務廠商利用在線服務與離線服務批處理作業的差異性，將二者混合部署在同一個生產集羣，這就可能出現在同一臺物理機上，不僅有在線業務之間的混部，也有離線業務之間的混部，以及在線和離線業務的混部。這就需要對兩種負載實現差異化資源分配，在保證QoS（Quality of Service，服務質量）的基礎上提高資源利用率。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"處理IO的兩種仲裁機制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主機接口負責主機與SSD之間的通信和數據傳輸，接受和解析I/O請求，維護一條或者多條請求隊列（Submission Queue，SQ）。主機和SSD通信時，通常會建立多條隊列，每個隊列裏又可以建立多個請求。有序的處理這些請求通常有兩種仲裁方式，一種是輪詢算法（Round Robin），一種是加權輪詢算法（Weighted Round Robin，簡稱WRR）。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1c/1ca53100d5992d20b6d25e1de54a68e9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果NVMe SSD採用了輪詢算法（Round Robin），即所有命令隊列（包括管理隊列Admin Submission Queue，ASQ）按照相同優先級輪詢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但面對在線和離線業務這種混部場景，不同性能需求的應用程序共享存儲設備，需要對不同業務IO處理分出輕重緩急。這就需要SSD支持加權輪詢算法，根據應用的不同優先級提供差異化IO分配策略。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"企業級SSD支持加權輪詢算法","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持加權輪詢算法的企業級NVMe SSD，隊列分成3類嚴格優先級，優先級從高到低依次爲管理Admin類、緊急Urgent類和優先級Priority類。創建和刪除IO隊列、獲取SSD的log數據、固件升級等管理命令都是通過管理命令隊列提交，其優先級也最高（嚴格優先級1）。緊急隊列優先級低於管理隊列，處於第二階梯（嚴格優先級2）。處於第三級（嚴格優先級3）的IO隊列又被劃分3種不同權重優先級，包括高High、中Medium、低Low，這也是本文討論的重點。有了這個權重等級，SSD就可以根據主機端設置及分配的IO命令優先級，依次完成對每個隊列命令的處理，實現對上層應用優先級的處理及反饋。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f9bc47b27190ce682261385da0dd5d5a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"主機端隊列優先級","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主機內存中包括兩個命令隊列，Submission Queue （SQ）和Completion Queue（CQ）。主機通過SQ告知SSD的控制器要處理的命令，控制器通過CQ告知主機已經被處理完畢的命令以及這些命令執行的狀態。如果要使用NVMe SSD的加權輪詢功能，需要端到端支持，也就是從NVMe SSD設備端到Kernel層面都需要支持加權輪詢算法。一種常用方式是通過SPDK的軟件棧，實現每個不同優先級core綁定1個IO隊列(包含1個SQ，1個CQ)。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/87/87b7d159b306b8aa39776b75a1515d07.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果用戶希望某個應用程序擁有更高的優先級，則可以提高某個隊列的權重。SSD在執行應用命令時，命令會按照優先級權重分發，實現負載均衡。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"SSD設備端設置不同權重實現差異化IO分配策略","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Memblaze PBlaze6 NVMe SSD的實測數據分析，可以看到SSD在總帶寬不變的情況下，按照隊列的權重分配帶寬。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據某應用程序需求，主機創建隊列時，可以建一個或者多個高、中和低隊列，通過標準nvme set-feature命令設置SSD端隊列權重，SSD可以根據這個等級和權重來分配帶寬資源。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4c9832c56e957fcee2502c1973967311.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"NVMe協議設置加權輪詢規則示意圖","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d8/d8d3cf0c2201ec94fe6219a5a4e04ed8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"nvmecli set-feature設置IO隊列權重爲8:1:1示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將高中低三檔IO隊列權重比值設置爲16:1:1，8:1:1，8:4:2三組，通過get-feature驗證權重設置生效。分析PBlaze6 6920系列加權輪詢測試數據，可以看出在總帶寬一致的情況下，set-feature 設定高、中、低隊列的權重比值與帶寬正相關。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac279c7a31754cb0644023e4769a9824.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"PBlaze6 6920系列加權輪詢測試數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Memblaze PBlaze6系列NVMe SSD支持企業級功能 — 加權輪詢，能夠爲混部業務場景下共享存儲設備提供差異化IO分配策略，幫助業務廠商進一步提升存儲利用率，減少離線、在線服務器採購成本。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s/b3d4iWkSHD8YvWD2sEw74A","title":"","type":null},"content":[{"type":"text","text":"原文鏈接","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}