共享存储设备，如何实现差异化IO分配策略

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着5G、大数据、区块链、物联网以及人工智能等技术在各行业不断渗透，企业对于数据存储及实时处理的需求日益提高，现代数据中心的规模也随着应用服务需求的增长而越来越大，简单通过采购更多服务器扩展集群会引起更高的运营成本与电力消耗，而且数据中心资源利用率低、利用率不够弹性已逐步成为云计算进一步发展的制约因素。为了降低采购成本，目前业务厂商比较行之有效的方式是通过业务混部来提升硬件（CPU和存储）资源利用率。这就需要NVMe SSD存储设备具备像多命名空间和加权轮询技术，实现在共享存储设备资源时，根据业务的优先级，提供差异化IO分配策略。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"集群负载和业务混部","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现代数据中心集群的负载可以简单分为两类，一类是延时敏感的在线服务，一旦服务崩溃或者延时过大，将导致用户体验不佳，需要留有较多的资源余量以应对突发的请求高峰；另一类是离线批处理作业，虽然资源消耗较大，但是对延时相对不敏感，可以容忍一定范围内执行时间的调整。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此业务厂商利用在线服务与离线服务批处理作业的差异性，将二者混合部署在同一个生产集群，这就可能出现在同一台物理机上，不仅有在线业务之间的混部，也有离线业务之间的混部，以及在线和离线业务的混部。这就需要对两种负载实现差异化资源分配，在保证QoS（Quality of Service，服务质量）的基础上提高资源利用率。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"处理IO的两种仲裁机制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主机接口负责主机与SSD之间的通信和数据传输，接受和解析I/O请求，维护一条或者多条请求队列（Submission Queue，SQ）。主机和SSD通信时，通常会建立多条队列，每个队列里又可以建立多个请求。有序的处理这些请求通常有两种仲裁方式，一种是轮询算法（Round Robin），一种是加权轮询算法（Weighted Round Robin，简称WRR）。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1c/1ca53100d5992d20b6d25e1de54a68e9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果NVMe SSD采用了轮询算法（Round Robin），即所有命令队列（包括管理队列Admin Submission Queue，ASQ）按照相同优先级轮询。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但面对在线和离线业务这种混部场景，不同性能需求的应用程序共享存储设备，需要对不同业务IO处理分出轻重缓急。这就需要SSD支持加权轮询算法，根据应用的不同优先级提供差异化IO分配策略。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"企业级SSD支持加权轮询算法","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持加权轮询算法的企业级NVMe SSD，队列分成3类严格优先级，优先级从高到低依次为管理Admin类、紧急Urgent类和优先级Priority类。创建和删除IO队列、获取SSD的log数据、固件升级等管理命令都是通过管理命令队列提交，其优先级也最高（严格优先级1）。紧急队列优先级低于管理队列，处于第二阶梯（严格优先级2）。处于第三级（严格优先级3）的IO队列又被划分3种不同权重优先级，包括高High、中Medium、低Low，这也是本文讨论的重点。有了这个权重等级，SSD就可以根据主机端设置及分配的IO命令优先级，依次完成对每个队列命令的处理，实现对上层应用优先级的处理及反馈。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f9bc47b27190ce682261385da0dd5d5a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"主机端队列优先级","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主机内存中包括两个命令队列，Submission Queue （SQ）和Completion Queue（CQ）。主机通过SQ告知SSD的控制器要处理的命令，控制器通过CQ告知主机已经被处理完毕的命令以及这些命令执行的状态。如果要使用NVMe SSD的加权轮询功能，需要端到端支持，也就是从NVMe SSD设备端到Kernel层面都需要支持加权轮询算法。一种常用方式是通过SPDK的软件栈，实现每个不同优先级core绑定1个IO队列(包含1个SQ，1个CQ)。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/87/87b7d159b306b8aa39776b75a1515d07.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果用户希望某个应用程序拥有更高的优先级，则可以提高某个队列的权重。SSD在执行应用命令时，命令会按照优先级权重分发，实现负载均衡。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"SSD设备端设置不同权重实现差异化IO分配策略","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过Memblaze PBlaze6 NVMe SSD的实测数据分析，可以看到SSD在总带宽不变的情况下，按照队列的权重分配带宽。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据某应用程序需求，主机创建队列时，可以建一个或者多个高、中和低队列，通过标准nvme set-feature命令设置SSD端队列权重，SSD可以根据这个等级和权重来分配带宽资源。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4c9832c56e957fcee2502c1973967311.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"NVMe协议设置加权轮询规则示意图","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d8/d8d3cf0c2201ec94fe6219a5a4e04ed8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"nvmecli set-feature设置IO队列权重为8:1:1示意图","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们将高中低三档IO队列权重比值设置为16:1:1，8:1:1，8:4:2三组，通过get-feature验证权重设置生效。分析PBlaze6 6920系列加权轮询测试数据，可以看出在总带宽一致的情况下，set-feature 设定高、中、低队列的权重比值与带宽正相关。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac279c7a31754cb0644023e4769a9824.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":10}}],"text":"PBlaze6 6920系列加权轮询测试数据","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Memblaze PBlaze6系列NVMe SSD支持企业级功能 — 加权轮询，能够为混部业务场景下共享存储设备提供差异化IO分配策略，帮助业务厂商进一步提升存储利用率，减少离线、在线服务器采购成本。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s/b3d4iWkSHD8YvWD2sEw74A","title":"","type":null},"content":[{"type":"text","text":"原文链接","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}