前车之鉴：聊聊我在基础设施中掉过的坑

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着自己在职业道路上不断成长，我总在会议上找到那种奇妙的、颇有点超现实意味的“既视感”。同事们偶尔提到的小状况，不禁让我想起之前曾经就同一个问题开过的会。我也还记得，自己当初的某个糟糕选定让接下来的几个月变得如同噩梦。于是我条件反射般地蹦起来，大呼“千万别如何如何！”同事们吓了一跳，但他们不知道这种反应的背后是种深深的恐惧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我想用这篇文章聊聊自己犯过的最大错误，希望能给各位后来者一点指引。其实我知道，无论看不看本文，该犯的错都会犯、该掉的坑都要掉。但无论是给您做个提醒、还是给自己做个回顾，我都希望能用一篇文章整理自己那些曾经深以为然、甚至投入不少精力的灾难性决定。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"别贸然把应用程序从数据中心迁移到云端"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各位云支持者先别急着喷。我个人也是云服务的忠实粉丝，但咱们应该承认，运行在物理数据中心内的应用程序很少能无缝迁移至云端。我先后参与过三次尝试，内容都是把专为特定数据中心编写的应用程序大规模迁移至云端；而每一次都会冒出预想不到的状况，把原本美好的迁移期望击个粉碎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/57ce2eb98282e29e79b3681dd6d3fbfc.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"当初遇到第一个无法解决的云迁移问题时，我的心情就如图所示"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"开发人员在编写和测试应用程序的同时，已经对目标环境的运作方式做出了假设或者说预期。服务器该如何工作、我的应用程序能获得怎样的性能、网络可靠性如何、传输延迟大概处在怎样的水平等等。任何一位拥有丰富工作经历的朋友，都会在接触项目的一刹那就整理出大致的判断；但如果想把应用程序打包起来、特别是比较陈旧的应用程序，再转移到其他运行环境时，总会出现种种奇怪的现象。我们开始遭遇前所未有的错误，还得忍受一个个为了支持迁移而不得不做的架构决策。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很快，大家就会发现迁移的预期价值被消耗殆尽，而某些糟糕的尝试（例如通过直连服务将本地数据中心与AWS无缝桥接起来）还会引发意外后果。总之，我们的决策清单开始快速堆积、增长，也给云服务商带来一个又一个极端案例。在此期间，我们还会找到很多根本不可能迁移的东西，于是被迫卡在两个环境之间——需要继续维护的数据中心，外加新的云账户。当初的意气风发已经不在，如今心里只剩下后悔。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"经验分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把应用程序移植到云端。请务必为开发人员提供一个与数据中心完全隔离的环境，引导他们将应用程序移植到云端，之后再给应用程序安排4到8小时的停机时间。在此期间，我们可以分步进行持久层切换，之后再更改DNS条目以指向新的云端版本。这是必要之恶，任何想要逃避这段停机时间的尝试都会带来一个又一个更错误的决定。总之，直面成本、大步向前。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"或者更直接点，在应用程序开发之初就充分考虑到云环境的特性。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"别总想自主编写加密系统"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不知道是太倒霉了还是怎么样，我总会遇上自主编写的系统。很多组织都特别喜欢原创加密系统，有时候是把环境变量注入系统、有时候是调用基于RSA密钥的解密API。好吧，我承认自己也曾经是那种自以为是的家伙，总以为这事“不会特别困难”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有一次，我肯定是疯了心了，突发奇想要在自己负责的PostgREST应用中管理自有secrets。所以我编写了程序，能够根据各种标准生成JWT并将其返回至应用程序。如此一来，应用就能安全无忧地访问secrets了……至少理论上能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a1\/a18da7378ea4b0e75b877cdcaef5b681.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我还要为PostgREST辩护几句，它很好地完成了预期任务。但问题是，后来出现的secrets管理情况比初步预计要复杂得多。我们先是碰到了缓存问题，也就是在面对每小时上百万次的服务访问压力之下，怎么继续保留一部分服务器作为事实来源。虽然后来我们调整Nginx配置解决了这个问题，但我其实应该提前就想到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之后的问题再次狠狠打了我的脸。新版本推送倒不成问题，但secrets却往往无法在客户端上实现版本化。我对应用程序进行了身份验证，也能查看正确的secrets；但在轮替期间会有两条secrets同时处于正确状态，这一点我是真的没想到。跟之前的问题一样，这事不难修复，但随着时间推移我在服务中遇到了越来越多的极端状况，也最终承认自己犯了个大错误。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回到现实，secrets管理是那种典型的高风险、低回报服务。这类功能既不会直接提升客户体验、也不会得到管理层的认同，它只会用没完没了的调试不断浪费我的时间，同时逼着我边调边学新知识。就为了这样一项小小的原创功能，我把从多区域可用性（跨区域同步问题）到服务强化等方向试了个遍。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"经验分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直接用AWS Secrets Manager或者Vault。我个人更喜欢Secrets Manager，但大家可以随意选择。总之，选什么都比自己写要好，因为原创方案往往没考虑到那些极端状况、而且跟现成服务比也没啥优势。我要用自己的体会告诉大家，一个小小的原创应用很可能成为系统宕机、拉升运营成本的罪魁祸首。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"不要自主运行Kubernetes集群"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我知道，很多朋友都觉得自己完全有能力运行Kubernetes集群。什么etcd啦、设置各种证书啦，分分钟搞定。没错，但面对“该不该自主运行K8s集群”这个问题时，我们不妨走一遍以下决策树："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你所在的是全球财富百强企业吗？如果不是，请放弃。因为虽然可以，但真的没有必要。直接用现成的K8s集群，能让我们享受到服务商添加的一系列强大功能。AWS EKS就包含不少令人惊艳的选项，包括在kubeconfig文件中支持AWS SSO、以及使用IAM角色让ServiceAccounts for pod访问AWS资源等等。最重要的是，整个控制平面的年运行成本还不到1000美元。如果这还不能说服你那颗骚动的心，那咱们再聊点实在的。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"云服务的一大优势，就是让别人帮你做beta测试。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我不明白这么简单的道理为什么很多人老是不懂。没错，你当然可以顺畅运行自己的K8s集群，但图什么呢？AWS那边有成千上万的beta测试员帮我们保障EKS的迭代升级。更重要的是，AWS的工程师水平很高、稳定可靠；而勉强在AWS中运行自有集群的唯一回报，似乎只是让自己拥有可以随时“更换云服务商”的错觉。是的，只是种错觉，后面我会再具体讨论。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"经验分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把一切交给云服务商。交给他们，问题就归他们管了。让开发人员们过得舒服点、轻松点，不好吗？"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"别针对多家云服务商做设计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其实我一直就是这么想的，但有位经验老道的经理说服了我，强调企业应该有能力随时更换云服务商。这话听起来也有道理，所以我们就踏上了这条“不归路”。现在的我，会称这种思路为“过早优化症候群”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按照多云设计思路，我开始审查“多云兼容性”并通过自主维护的SDK替代AWS提供的预制SDK。而这一切都是为了营造某种虚假的安全感，即如果AWS这家大受欢迎的云巨头哪天突然不行了，我们就能比较顺畅地完成大规模负载转移并值回投入的成本。我猜大家这么干是想证明自己有某种厉害的前瞻性，或者是野心太大把脑回路给带偏了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而这也是我们做过的最愚蠢的尝试，大大提高了向客户交付产品的难度。既然选择了AWS，请拜托各位别假装自己的应用程序还需要面向其他云环境部署。如果AWS过几天消失了，那确实得迁移应用程序；但有几个人敢说自己的公司能开得比AWS久？既然没这个信心，干嘛要在跟当前云无关的翻译层上浪费那么多时间、投入那么多金钱？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们最终得到的只是一堆始终跟不上节奏的库，开发人员也被搞得身心俱疲、没工夫研究AWS发布的一系列最新功能。这些自定义库发挥不了AWS的云功能优势，我们自己也从来没试过更换云服务商或者搞什么双云部署。因为从经济层面讲，这么干没有任何意义。开发团队空耗许久，最终什么都没有得到。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"经验分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果各位身边还有人鼓吹“必须保证不必依赖於单一云服务商”，拜托替我骂醒这家伙。与数据中心类似，在AWS当中设计、测试并成功运行多年的应用程序往往都带有与环境相匹配的某些预期和模式。针对其他不可知设计做出的优化，都必然在牺牲当前云服务商功能价值的同时、给开发团队带来本不必要的额外工作压力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"别当那种站关说话不腰疼的人，跟所有人对着干往往当不成力挽狂澜的英雄、反而让我们沦为众目睽睽下的小丑。另外，就算大家基本认定转向其他云服务商具有可靠的经济意义，也请至少留出3个月时间开展应用程序测试和移植，之后再考虑到底要不要具体实施。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"云服务商当然会产生依赖性，就像编程语言一样。我们不可能随随便便就决定更换，甚至全面移植也不太现实。所以大家不妨把迁移议案当成一种演习偶尔试试，同时把大部分精力集中在产品与当前环境的充分融合上。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"别没完没了地增加警报"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相信大家在工作中都见过这类情境。办公室里摆着一台显示器，专门展示图表或者CloudWatch警报之类。某些警报会隔段时间就定期被触发，但同事们已经习以为常，直接当作没看见。继续追问，他们的回答也只是“我们只想看看到底会报多少次警，没突然增多就行。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这种不在意的态度终究会体现在真正紧急的警报当中，导致我们没能把握时机解决关键问题。那能不能减少警报数量呢？不行，服务管理团队觉得还是应该提醒一下，哪怕没人看。所以随着时间推移，似乎没人能说清到底哪些警报重要、哪些不重要；只有新人报道时会再提起这事，然后同事们把前面的话重新解释一遍。最终，系统一定会发生大崩溃，因为某条“不要紧”的警报里藏着非常要紧的信息，但却根本没人关注。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我就这么干过，而且捏造出了“设置警报肯定有设置的道理”来搪塞新员工。我错了，我当初应该支持“干翻一切、推倒重来”的观点。希望多年前让我痛心疾首的决定，不要在当下成为各位读者朋友的现实隐患。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"经验分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"只要警报响起，就一定代表着系统无法自行恢复、必须马上介入的状况。警报应该具有严肃性，我们也不该把故障内置到应用程序的设计之内。比如“有时候我们的服务需要重新启动，通过SSH连接并重启就行了”，这属于常态、不该被设置成警报。如果说重启失败，那又是另一个问题，咱们不要把二者搞混淆了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"别让垃圾警报慢慢污染我们的生活。如果平台上的所有警报几乎都没有实际意义，请果断推倒重来。如果系统每天要发出600封提醒邮件，那跟一封都不发没有任何区别。过量的警报只会让我们神经麻木，代表着我们当前根本得不到警报系统的保护。人类的大脑结构就是这样，我们不能指望一个人在持续收到垃圾警报之后，还能敏锐地从中找到真正重要的那条。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"不要用Python编写内部CLI工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这事我只简单谈两句。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"没人知道怎么正确安装和打包Python应用程序。如果你决定用Python编写内部工具，那面前只有两条路：要么保证应用具有完全可移植性，要么请选择Go或者Rust语言。相信我，否则在安装过程中遇到麻烦的用户很可能提着刀来追杀你。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/matduggan.com\/mistakes\/","title":"","type":null},"content":[{"type":"text","text":"https:\/\/matduggan.com\/mistakes\/"}]}]}]}