改善十年应用的部署体验

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 2018 年，Etsy 将它的服务基础设施从自我管理的数据中心迁移到云端配置（我们当时在博客上写了这件事）。这种改变提供了改善整个公司技术流程的机会。对于 Search 团队而言，云环境所带来的灵活扩展让我们可以完全重新评估一个有些繁琐的部署流程。在已有的金丝雀发布架构模式的启发下，我们编写了一个新的自定义工具来补充现有的部署基础设施。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在三个月的努力之后，我们最终得到了一个更具可扩展性、对开发人员更友好、最终也是更健壮的方式来滚动发布对 Search 的改变。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"蓝绿部署"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"过去，我们把堆栈部署到两套独立的主机上，也就是所谓的“蓝绿策略”。在任何时候，只有一组主机是活动的；而另一组，或 \"侧端\"则处于“黑暗”状态。这两边都是完全扩展的，并准备好为流量提供服务，但只有活动的那一组可以进入公共互联网。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"蓝绿部署策略虽然简单，但具有一些非常有用的特性："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于搜索应用，我们可以在一边做重要的改变，因为它是有状态的，同时持续地使用另一边来提供流量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在将生产流量发送到它之前，我们有地方可以手动测试更改。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们总是有一个以前的 Search 版本，在紧急情况下，我们可以轻松恢复。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"还有一个内置的机制，可以测试和产生其他突破性的变化，比如软件版本升级。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们将这两组主机称为“flip”和“flop”，这是以现代计算机的基本构件——电路命名的。在配置文件中，我们通过几行代码，将单个 PHP Web 应用指向哪一组，哪一组就应该处于活动状态。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/69\/95\/693902ef68d7a288437c9bbd8d1c2f95.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"图为我们之前的基础设施。一组（本例中的 flop）始终处于活动状态，在部署过程中，我们会将所有的流量一次性转移到另一组（本例中的 flip）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三年前，Etsy 向云端迁移时，这种蓝绿部署的方法被“提升和转移”。Search 团队将搜索应用移至谷歌 Kubernetes 引擎（Google Kubernetes Engine，GKE），flip 和 flop 成为两个独立的生产 Kubernetes 命名空间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"撇开这一变化不谈，事情的运作与以往一样：部署 Search 会立即触发所有的流量从一个命名空间——活动端——重定向到另一个命名空间中运行的同一服务。为了确保黑暗端始终准备就绪，我们将继续在任何时候都保持 200% 的容量（每个生产命名空间 100%），正如我们在企业内部时所做的那样。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这种最初的部署策略对团队来说非常有用，尤其是为我们提供了一个安全的空间，可以测试和准备主要的软件更新和基础设施变更。然而，它也并非没有痛苦。突然间，所有流量都被转移到两边，这使得我们没有余地在全部投入之前测试少量生产流量的变化。甚至在一切顺利的时候，部署也是充满压力的。如果情况出了问题，工程师们就得进行分流，来决定是否要完全恢复原状。最重要的是，长时间维持双倍容量既昂贵又低效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因为云计算提供了灵活性，一旦我们安全地进入 GKE，我们就有机会重新考虑我们的蓝绿策略，并解决这些长期存在的问题。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"金丝雀（精简版）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们的第一个想法是采用金丝雀部署策略。在“金丝雀发布”期间，在将所有流量切换到新服务之前，将一小部分流量发送到服务的新版本，以确定它是否 “安全”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为什么叫这个名字？煤矿工人曾经使用金丝雀来检测一氧化碳的浓度，这种浓度可能会伤害一只小鸟，但仍不会对人造成伤害。软件工程师们也采取了类似的模式（尽管更加人性化），以树立新软件能够安全地服务于流量的信心。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虽然 Kubernetes 的架构和灵活的扩展意味着金丝雀发布是一个非常流行的部署解决方案，但是 Etsy 的搜索系统的设计意味着我们不能使用任何现成的金丝雀发布解决方案。我们必须为自己建造一些新的东西，一种金丝雀精简版。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在考虑重新构建整合金丝雀组件的部署流程时，存在两个关键的限制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一个限制是，我们不能使用单一的负载均衡器活 API 断电来控制流入流量的数量。这样，我们就不能使用 Kubernetes 标签在单个 Kubernetes 部署上为任何搜索服务做基本的金丝雀发布，因为 Search 是由许多不同的 Kubernetes 部署组成的。我们没有地方来设置路由逻辑来检查标签，以及相应的路由到金丝雀 Pod。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，Etsy 的 PHP Web 应用是搜索应用的唯一客户端。对于 Etsy，这是一种常见模式，因此，配置负载平衡通常是直接在 Web 应用本身中进行管理。任何新的部署解决方案或者必须对 Web 应用管理流量到 web 应用内部的 Search，或者实现某种全新的网状网络（比如 Istio），捕获并引导 Web 应用到 Search 的所有流量。这两个方案在分配给该项目的时间范围内都不可行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二个限制是，搜索应用假定请求路径中所有搜索服务的相同版本都会为任何单一 Web 请求服务。所以，任何新解决方案的部署都需要确保所有搜索服务的旧版本都能满足正在进行的搜索请求。甚至像 Istio 这样复杂的金丝雀发布解决方案，也要求你的应用处理不同服务之间的版本不匹配，这是我们无法保证的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么，我们如何为所有新版本的搜索服务创建一个逐步推出的过程，同时管理从 Web 应用到滚动发布的所有部分的负载平衡，并保证搜索服务只能和其他搜索服务的相同版本对话？对于这种 Etsy 特定问题，没有现成的解决方案。因此我们开发了一种全新的工具，叫做 Switchboard。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"走入 Switchboard"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Switchboard 的主要功能是管理流量：它通过逐步增加提供给新的活动端的百分比，并按比例减少进入旧的活动端的数量，将一个部署滚动发布到生产。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具有预先定义的流量比例的部署阶段被硬编码到系统中，当在当前滚动发布阶段中添加的所有 Pod 都完全创建并正常运行时，Switchboard 就会过渡到下一个阶段。这是通过编辑和提交新的流量比例到 Web 应用中的配置文件来实现这一目标的。Web 应用对每一个心情求都重新检查这个文件，并使用这些信息在两个不同的生产 Kubernetes 命名空间之间进行负载均衡搜索流量，这两个命名空间仍然被称为 flip 和 flop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/46\/81\/463a22bf6455643540yycc0e156cb481.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"使用 Switchboard 的一端开关的例子。Smoke 测试在 16:57 和 17:07 进行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Switchboard 在部署过程中很大程度上实现了流量从一个搜索端到另一个搜索端的自动迁移。Smoke 测试在部署的不同阶段运行，向新的一端发送人工创建的和真实的历史搜索查询请求。开发人员只需监控图表，以确保滚动发布的工作顺利进行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4b\/0c\/4be7c0bdc479a2ae75fb2b361c5bc50c.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"推动部署的工程师通过显示当前滚动发布状态的用户界面来管理 Switchboard，它还可以选择暂停或者回滚部署。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Switchboard 中，我们主要依靠 Kubernetes 内置的自动伸缩功能来扩展部署期间的新集群。在开始向集群发送生产流量之前，我们已经发现，我们只需要先将集群的规模扩大到我们当前容量的 25%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kubernetes 内置的自动伸缩是被动的，因此与我们强制 Search 在需要额外容量之前进行伸缩相比，速度肯定要慢。这样，它可以帮助提前扩展新的活动端，从而在该端上线并开始接收流量时更快地响应初始转换。通过 Switchboard，Kubernetes 可以管理自己的自动伸缩功能，只需监控 Kubernetes 的滚动发布，可以确保所有服务在当前阶段是健康的，然后再决定升级。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"结果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们设计 Switchboard 是为了改善我们 Search 系统的资源消耗，它已经做到了这一点。然而，分步部署的方法也给开发者带来了许多不错的工作流程改进。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Switchboard 使我们能够将搜索虚拟机的总容量控制在或接近 100%，而非我们以前所支持的 200% 的容量，随着 Search 流量的增加，我们再也不必提供双倍的容量了。如今，适应流量的不安华变得更加容易，因为任何额外的的被动（自动）或主动（手动）扩展只需要为我们的真实容量保留计算服务，而不是两倍。所以，我们在发布 Switchboard 期间，云虚拟机的利用率有了明显的提高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/40\/ba\/40861a3a1b61eca63c08d74baac280ba.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"几个月来，每个搜索请求的云成本（云计费总额 \/ 请求数）显示，我们在 Switchboard 之后，利用效率有所提高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Switchboard 的第二大成功在于，它使得我们在 Staging 环境中的部署速度不断提高。我们第一次尝试放弃传统的双重配置方法，就是在两次部署之间完全缩减未使用的搜索集群，然后预先对其进行重新配置，作为下一个部署的第一步。这种方法的一个问题就是，开发人员必须等待我们的 Search 系统内的所有服务被扩展到足以接受流量，然后他们才能在我们的 Staging 环境中进行测试。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/c7\/ee\/c7e89fb5e9dce64a235af51f7fb2d0ee.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"每个 Staging 环境部署所用的时间。每个单独的行是一个单独的部署。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"总的来说，在实施 Switchboard 后，我们看到利用率的提高与中间解决方案相似，但却不必在较慢的部署时间上做出妥协。Switchboard 甚至还提高了中间解决方案的利用效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在，在部署过程中发现和响应问题也变得更加容易。从技术上讲，Search 部署的时间比我们维护两个完全扩展的集群时要长，但是这个额外的时间是由于自动流量滚动发布过程的渐进性造成的。人类搜索部署人员通常是被动地监控滚动发布阶段，根本没有交互。但是，如果他们需要的话，可以暂停滚动发布，检查当前的结果。Search 部署人员至少每月使用一次 Switchboard 来暂停滚动发布。我们以前根本就没有这个选项。由于 Switchboard 个别部署变得更加安全和可靠。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后，重新架构我们的蓝绿部署流程，包括通过 Switchboard 进行金丝雀式的逐步流量提升，使我们的系统更具可扩展性和效率，同时也为开发者提供了更好的体验设计。为了充分利用 Kubernetes 和云环境的灵活性，我们成功地调整了搜索应用的架构。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/codeascraft.com\/2021\/06\/15\/improving-the-deployment-experience-of-a-ten-year-old-application\/"}]}]}