{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"让人讨厌的 CPU 限流影响容器运行，有时人们不得不牺牲容器部署密度来避免 CPU 限流出现。我们设计的CPU Burst 技术既能保证容器运行服务质量，又不降低容器部署密度。CPU Burst 特性已合入 Linux 5.14，Anolis OS 8.2、Alibaba Cloud Linux2、Alibaba Cloud Linux3也都支持CPU Burst特性。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 K8S 容器调度中，容器的 CPU 资源上限是由 CPU limits 参数指定。设置 CPU 资源上限可以限制个别容器消耗过多的 CPU 运行时间，并确保其他容器拿到足够的 CPU 资源。"},{"type":"text","marks":[{"type":"strong"}],"text":"CPU limits 限制在 Linux 内核中是用 CPU Bandwidth Controller 实现的，它通过 CPU限流限制 cgroup 的资源消耗"},{"type":"text","text":"。所以当一个容器中的进程使用了超过 CPU limits 的资源的时候，这些进程就会被 CPU 限流，他们使用的 CPU 时间就会受到限制，进程中一些关键的延迟指标就会变差。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面对这种情况，我们应该怎么办呢？"},{"type":"text","marks":[{"type":"strong"}],"text":"一般情况下，我们会结合这个容器日常峰值的 CPU 利用率并乘以一个相对安全的系数来设置这个容器的 CPU limits ，这样我们既可以避免容器因为限流而导致的服务质量变差，同时也可以兼顾 CPU 资源的利用"},{"type":"text","text":"。举个简单的例子，我们有一个容器，他日常峰值的 CPU 使用率在 250% 左右，那么我们就把容器 CPU limits 设置到 400% 来保证容器服务质量，此时容器的 CPU 利用率是 62.5%（250%\/400%）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而生活真的那么美好吗？显然不是！CPU 限流的出现比预期频繁了很多。怎么办？似乎看上去我们只能继续调大 CPU limits 来解决这个问题。很多时候，"},{"type":"text","marks":[{"type":"strong"}],"text":"当容器的 CPU limits 被放大 5~10 倍的时候，这个容器的服务质量才得到了比较好的保障，相应的这时容器的总 CPU 利用率只有 10%~20%。"},{"type":"text","text":"所以为了应对可能的容器 CPU 使用高峰，容器的部署密度必须大大降低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"历史上人们在 CPU Bandwidth Controller 中修复了一些 BUG 导致的 CPU 限流问题，我们发现当前非预期限流是由于100ms级别CPU突发使用引起，并且提出 CPU Burst 技术允许一定的 CPU 突发使用，避免平均 CPU 利用率低于限制时的 CPU 限流。"},{"type":"text","marks":[{"type":"strong"}],"text":"在云计算场景中，CPU Burst 技术的价值有："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"不提高 CPU 配置的前提下改善 CPU 资源服务质量；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"允许资源所有者不牺牲资源服务质量降低CPU资源配置，提升CPU资源利用率；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"降低资源成本（TCO, Total Cost of Ownership）。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"你看到的CPU利用率不是全部真相"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"秒级 CPU 利用率不能反映 Bandwidth Controller 工作的 100ms 级别 CPU 使用情况，是导致非预期 CPU 限流出现的原因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Bandwidth Controller 适用于 CFS 任务，用 period 和 quota 管理 cgroup 的 CPU 时间消耗。若 cgroup 的 period 是 100ms quota 是 50ms，cgroup 的进程每 100ms 周期内最多使用 50ms CPU 时间。当 100ms 周期的 CPU 使用超过 50ms 时进程会被限流，cgroup 的 CPU 使用被限制到 50%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"CPU 利用率"},{"type":"text","text":"是一段时间内 CPU 使用的平均，以较粗的粒度统计 CPU 的使用需求，CPU 利用率趋向稳定；当观察的粒度变细，CPU 使用的突发特征更明显。以 1s 粒度和 100ms 粒度同时观测容器负载运行，当观测粒度是 1s 时 CPU 利用率的秒级平均在 250% 左右，而在 Bandwidth Controller 工作的 100ms 级别观测 CPU 利用率的峰值已经突破 400% 。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/8a\/d4\/8a653f0c7fd8d6d30dc6b643bcdb54d4.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据秒级观察到的 CPU 利用率 250% 设置容器 quota 和 period 分别为 400ms 和 100ms ，容器进程的细粒度突发被 Bandwidth Controller 限流，容器进程的 CPU 使用受到影响。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何改善"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们用 CPU Burst 技术来满足这种细粒度 CPU 突发需求，在传统的 CPU Bandwidth Controller quota 和 period 基础上引入 burst 的概念。当容器的 CPU 使用低于 quota 时，可用于突发的 burst 资源累积下来；当容器的 CPU 使用超过 quota，允许使用累积的 burst 资源。最终达到的效果是将容器更长时间的平均 CPU 消耗限制在 quota 范围内，允许短时间内的 CPU 使用超过其 quota。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/fa\/b2\/fa083e98a70e33d34842dc0dceb5d5b2.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果用 Bandwidth Controller 算法来管理休假，假期管理的周期（period）是一年，一年里假期的额度是 quota ，有了 CPU Burst 技术之后今年修不完的假期可以放到以后来休了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在容器场景中使用 CPU Burst 之后，测试容器的服务质量显著提升"},{"type":"text","text":"。观察到 RT 均值下降 68%（从 30+ms 下降到 9.6ms ）；99% RT 下降 94.5%（从 500+ms 下降到 27.37ms ）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/1c\/a0\/1c81c21459ff0a0029a920973c2dc5a0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"CPU Bandwidth Controller的保证"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用CPU Bandwidth Controller可以避免某些进程消耗过多CPU时间，并确保所有需要CPU的进程都拿到足够的CPU时间。之所以有这样好的稳定性保证，是因为当Bandwidth Controller设置满足下述情况时，"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4d\/5e\/4d24b4f841314123b2feb8b2b2598d5e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有如下的"},{"type":"text","marks":[{"type":"strong"}],"text":"调度稳定性约束："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/fd\/0b\/fdfa855d986065a156e5459a0a6d190b.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9e\/a9\/9e862c72e6d29f008a9716cafef0yya9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"是第i个cgroup的quota，是一个period内该cgroup的CPU需求。Bandwidth Controller对每个周期分别做CPU时间统计，调度稳定性约束保证在一个period内提交的全部任务都能在该周期内处理完；对每个CPU cgroup而言，这意味着任何时候提交的任务都能在一个period内执行完，即"},{"type":"text","marks":[{"type":"strong"}],"text":"任务实时性约束："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/27\/75\/27eb3d88b5319dd1d381dbef199fbd75.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不管任务优先级如何，最坏情况下任务执行时间（WCET, Worst-Case Execution Time）不超过一个period。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假如持续出现"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/5d\/76\/5d1bdf16fbc2fa958d2ee0c17dba0a76.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"调度器稳定性被打破，在每个period都有任务积攒下来，新提交的作业执行时间不断增加。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用CPU Burst的影响"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"出于改善服务质量的需要，我们使用CPU Burst允许突发的CPU使用之后，对调度器的稳定性产生什么影响？"},{"type":"text","marks":[{"type":"strong"}],"text":"答案是当多个cgroup同时突发使用CPU，调度器稳定性约束和任务实时性保证有可能被打破"},{"type":"text","text":"。这时候两个约束得到保证的概率是关键，如果两个约束得到保证的概率很高，对大多数周期来任务实时性都得到保证，可以放心大胆使用CPU Burst；如果任务实时性得到保证的概率很低，这时候要改善服务质量不能直接使用CPU Burst，应该先降低部署密度提高CPU资源配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"于是下一个关心的问题是，怎么计算特定场景下两个约束被打破的概率。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"评估影响大小"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定量计算可以定义成经典的排队论问题，并且用蒙特卡洛模拟方法求解。"},{"type":"text","marks":[{"type":"strong"}],"text":"定量计算的结果表明"},{"type":"text","text":"，判断当前场景是否可以使用CPU Burst的主要影响因素是平均CPU利用率和cgroup数目。CPU利用率越低，或者cgroup数目越多，两个约束越不容易被打破可以放心使用CPU Burst。反之如果CPU利用率很高或者cgroup数目较少，要消除CPU限流对进程执行的影响，应该降低部署提高配置再使用CPU Burst。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"问题定义是：一共有"},{"type":"text","marks":[{"type":"strong"}],"text":"m"},{"type":"text","text":"个cgroup，每个cgroup的quota限制为1\/m"},{"type":"text","marks":[{"type":"strong"}],"text":"，"},{"type":"text","text":"每个cgroup在每个周期产生的计算需求（CPU利用率）"},{"type":"text","marks":[{"type":"strong"}],"text":"服从某个具体分布"},{"type":"text","text":"，这些分布是相互独立的。假设任务在每个周期的开始到达，如果该周期内的CPU需求超过100%，当前周期任务WCET超过1个period，超过的部分累积下来和下个周期新产生的CPU需求一起在下个需求处理。输入是cgroup的数目m和每个CPU需求满足的具体分布，输出是每个周期结束WCET > period的概率和WCET期望。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以输入的CPU需求为帕累托分布、m=10\/20\/30的结果为例进行说明。选择帕累托分布进行说明的原因是它产生比较多的长尾CPU突发使用，容易产生较大影响。表格中数据项的格式为"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/c2\/90\/c25e6472cdb28511a67446c6134af790.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/39\/7f\/3963e14b8174c0697c601403yy18267f.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"越接近1越好，"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/64\/e7\/645e75b664b241c9147d6c6fa4784fe7.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"概率越低越好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"u_avg m=10 m=20 m=30 10% 1.0000\/0.00% 1.0000\/0.00% 30% 50% 70% 90% 参数 描述 值 distribution 计算需求产生的分布 负指数、帕累托 u_avg 平均产生的计算需求 10%-90% m cgroup（容器）个数 10、20、30 b 令牌桶的buffer大小（相对于其quota的倍率） 100%、200%、∞"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"负指数分布是排队论模型中最常见、最多被使用的分布之一。其密度函数为"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ac\/ae\/ac294f675bf4e8ebfe17b5c3fd36f8ae.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/3a\/8d\/3a20ef84f59befdcba38833862ed558d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"帕累托分布是计算机调度系统中比较常见的分布，且它能够模拟出较大的延迟长尾，从而体现 CPU Burst 的效果。其密度函数为："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/39\/39\/39d4f204f40ff64783e496847bcac739.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了抑制尾部的概率分布使其不至于过于夸张，我们设置了："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7e\/cb\/7e6f104d2956fe20a9c5f52b54ef28cb.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此时当 u_avg=30% 时可能产生的最大计算需求约为 500%。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数据展示"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按上述参数设置进行蒙特卡洛模拟的结果如下所示。我们将第一张（WCET 期望）的图表 y 轴进行颠倒来更好地符合直觉。同样地，第二张图表（WCET 等于 1 的概率）"},{"type":"text","marks":[{"type":"strong"}],"text":"表示调度的实时性得到保证的概率，以百分制表示"},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"负指数分布"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/72\/bf\/7279cdf5d4d387397f232e9458de5ebf.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/31\/1a\/31b9aee077cf13c70546f60a6220c91a.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"帕累托分布"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/66\/yy\/66e3102c46f8a3dbcd0156fc42351cyy.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/80\/54\/805d91798b8e9ca6958eaa4a3302ed54.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"结论"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"一般来说，u_avg（计算需求的负荷）越高，m（cgroup数量）越少，WCET 越大"},{"type":"text","text":"。前者是显然的结论，后者是因为独立同分布情况下任务数量越多，整体产生需求越趋于平均，超出 quota 需求的任务和低于 quota 从而空出 cpu 时间的任务更容易互补。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"提高 buffer 会使得 CPU Burst 发挥更好的效果，"},{"type":"text","text":"对单个任务的优化收益更明显；但同时也会增大 WCET，意味着增加对相邻任务的干扰。这也是符合直觉的结论。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在设置 buffer 大小时，我们建议根据具体业务场景的计算需求（包括分布和均值）和容器数量，以及自身需求来决定。"},{"type":"text","marks":[{"type":"strong"}],"text":"如果希望增加整体系统的吞吐量，以及在平均负荷不高的情况下优化容器性能，"},{"type":"text","text":"可以增大 buffer；反之如果希望保证调度的稳定性和公平性，在整体负荷较高的情况下减少容器受到的影响，可以适当减小 buffer。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一般而言，在低于 70% 平均 CPU 利用率的场景中，CPU Burst 不会对相邻容器造成较大影响。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"模拟工具与使用方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"说完了枯燥的数据和结论，接下来介绍可能有许多读者关心的问题：CPU Burst 会不会对我的实际业务场景造成影响？为了解决这个疑惑，我们将蒙特卡洛模拟方法所用工具稍加改造，从而能帮助大家在自己的实际场景中测试具体的影响～"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工具可以在这里获取："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"https:\/\/codeup.openanolis.cn\/codeup\/yingyu\/cpuburst-simulator"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"详细的使用说明也附在 README 中了，下面让我们看一个具体的例子吧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"小A想在他的服务器上部署 10 台容器用于相同业务。为了获取准确的测量数据，他先启动了一台容器正常运行业务，绑定到名为 cg1 的 cgroup 中，不设限流以获取该业务的真实表现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后调用 "},{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"sample.py "},{"type":"text","text":"进行数据采集：（演示效果只采集了 1000 次，实际建议有条件的情况下采集次数越大越好）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f8\/46\/f8c17bbd09854344a0e0683e450bff46.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些数据被存储到了"},{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":".\/data\/cg1_data.npy "},{"type":"text","text":"中。最后输出的提示说明该业务平均占用了约 6.5% 的 CPU，部署 10 台容器的情况下总的平均 CPU 利用率约为 65%。（PS：方差数据同样打印出来作为参考，也许方差越大，越能从 CPU Burst 中受益哦）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来，他利用"},{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":" simu_from_data.py "},{"type":"text","text":"计算配置 10个 和 cg1 相同场景的 cgroup 时，将 buffer 设置为 200% 的影响："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/91\/35\/91bd55fda3c84677f159038cf5322435.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据模拟结果，开启 CPU Burst 功能对该业务场景下的容器几乎没有负面影响，小A可以放心使用啦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"想要进一步了解该工具的用法，或是出于对理论的兴趣去改变分布查看模拟结果，都可以访问上面的仓库链接找到答案～"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"关于作者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常怀鑫（一斋），阿里云内核组工程师，擅长CPU调度领域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"丁天琛(鹰羽)，2021年加入阿里云内核组，目前在调度领域等方面学习研究。"}]}]} |