抓住那只捣乱的猴子，中国移动的混沌工程实践

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"睡梦正酣，你被一阵密集的电话铃声吵醒，匆匆抹了把脸赶到公司，加入正在进行的讨论。夜幕退去，天色渐亮，你的思路越发清晰，看着电脑屏幕上畅通运行的程序，松了口气。紧接着，疲惫感如潮水般向你涌来。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这是对技术人员而言并不陌生的场景，也是威胁年轻程序员毛囊健康的主因之一。脱落的是毛发，迸发的是灵感。中国移动对于混沌工程有了最初的想法，也是在这样一个被叫醒的深夜。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"从故障中来"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中国移动信息技术中心磐基PaaS平台运营运维团队在一个处理故障到五六点的凌晨，向自己发出疑问：平时总结在流程、运维等方面准备了很多，为什么关键时刻，这些东西起不到作用？磐基PaaS平台目前只是支持中国移动的上百个系统，未来要支持更多系统，怎么才能真正做到“乘舟上云，稳如磐基”？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中国移动磐基PaaS平台是中国移动信息技术自主研发的云原生平台，目前已建设K8S集群144个，生产节点规模近1.1万个，容器总实例数超过20万个，提供微服务框架支撑能力、服务治理能力，解决微服务的分布式特点带来的管理复杂性。而混沌工程，是专门为主动理解并应对复杂系统而创建的学科，如今普遍被用于分布式架构和云原生环境中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"混沌工程的诞生，是为了建立对系统抵御生产环境中失控条件的能力以及信心。2010年，Netflix为了在避免迁移时可能出现的故障影响用户体验，开发了Chaos Monkey，用于提前测试系统。在实践过程中，这个想法和测试工具被不断完善，衍生出“混沌工程师”的角色，在工程社区推广，并在《Principles Of Chaos》中被正式总结为“混沌工程”。随着越来越多的大公司开始了解和实践混沌工程，2018年，不断迭代的混沌工程（Chaos Engineering）成为CNCF一个新的技术领域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中国移动解决故障的思路是“自己折腾自己”——探知风险、预测风险、挖掘风险。 "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"到“混沌”中去"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于业务场景、人员结构、实施方式等指标的差异性，系统稳定性的评判标准往往难以统一。此前对业务系统的上云架构评审以及高可用测试方案，都是由工程师根据经验主观推断出来的。系统稳定性、功能合理性、服务完备性如何证明？传统方案上我们只能证明系统可能会出现问题，而不能证明系统一定不会出现问题，无穷性出发点使我们没有办法证明系统的正确性。所以中国移动磐基PaaS平台引入了混沌工程，用混沌工程在系统上的逻辑性来证明系统的正确性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"故障注入、故障测试和混沌工程，是业内容易被混淆的话题。在磐基团队看来，混沌工程并不是在制造混沌，制造故障，而是将系统固有的混沌进行可视化。在整个混沌工程领域下包含了反脆弱性、故障注入、故障测试等相关范畴，所以它不仅仅是测试，其方法可以理解为探索性测试与可观测技术的结合。“混沌工程与反脆弱的一个关键区别是混沌工程让人认识到混沌为系统固有，从而提高团队韧性，而反脆弱目的是让系统在响应混沌时变得更强大。但混沌可以通过上述方法去辅助系统的理解，所以认为它是大小集合的概念。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"混沌工程的黄金标准是："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"建立关于稳态行为的假说"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多样化引入真实故障事件"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在生产环境中进行实验"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"持续运行自动化实验"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最小化爆炸半径"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磐基混沌能力发展路径，包括接触和使用混沌工程的能力，混沌平台建设和深化应用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磐基团队表示，混沌工程建设方面，最开始是利用工具实现高可用场景进行场景侧的小范围试点，在平台构建的过程中进行业务规划研讨、混沌团队建设和用户试用，平台发展的过程包括：用户业务研讨、稳态防御、应用推广、流程细化等。工具选型方面，基于支持不同的应用架构与部署架构，支持不同资源的故障场景，以及部署方便，中国移动磐基PaaS平台选定阿里开源的ChaosBlade作为混沌工程能力的底座。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"中国移动的混沌工程实践经验"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中国移动在混沌工程的实践已经经历了多次迭代。“我们其实最初的平台建设，是先从故障注入的那种角度先进行，然后根据混沌工程的理念，往里面再增加一些监控。举个例子，对一个演练进行混沌工程的测试，可能某些指标上升了，如果我们对这个系统的认知不够全面，故障一旦注入，可能会导致故障的蔓延式发展。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"什么样的企业需要做混沌工程？从行业属性上来看，磐基团队比较建议对用户感知比较高的企业做混沌工程，比如金融证券行业。在中国移动，用户感知高的业务，比如电话停机、卡券售卖，一旦出现故障，会立刻反馈道用户端，降低用户体验。毕竟，业务端的业务量可能有上线，但变更无止境，故障也不会完全消失。混沌工程的意义在于可以减少故障发生的机率，并且让系统在面对故障时拥有快速响应、恢复的可能性。所以，距离用户最近的业务、用户感知度高的业务，需要考虑进行混沌工程实验。"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采访嘉宾简介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"赵淳，磐基PaaS运营维护总监，负责磐基PaaS在集、省、专推广、交付与后期运维，有多项国际认证，熟悉容器、微服务以及数据库、中间件等云原生技术，有多年省BOSS、CRM建设运维经验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"严俊，磐基PaaS平台开发组成员，多年深耕于大数据、云原生等领域，主导中国移动磐基PaaS平台混沌工程能力的研发和建设工作。目前主要负责磐基PaaS平台在边缘云、混沌工程、AIOPS等领域的技术研究与落地工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"袁庆彬，中国移动磐基平台运营运维团队成员，IT运维专家，长期致力于IaaS、PaaS领域。负责磐基平台服务连续性治理、平台架构优化和能力组件提升工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"晁元宁，磐基PaaS平台运营维护组成员，PaaS交付专家、混沌工程专家，长期致力于云原生、PaaS、混沌工程等领域。负责磐基PaaS平台项目上云的运维和运营工作，主要对接集团公司、专业公司、省公司的上云项目，负责项目对接、项目交付、项目维护，具备丰富的交付、运维、应急演练等经验。"}]}]},{"type":"horizontalrule"},{"type":"heading","attrs":{"align":"center","level":4},"content":[{"type":"text","text":"扫描下方二维码，进入有奖问答"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"参与国内首个混沌工程调研报告"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/78\/b4\/78ca90e0bd194c1745abb38a57f3c4b4.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了解我国混沌工程发展全貌，中国信通院联合混沌工程实验室启动《中国混沌工程调查报告》问卷征集活动，深入探索我国系统稳定性现状及混沌工程使用情况、行业采纳度、技术成熟度及未来发展趋势，以期推动混沌工我国的概念普及，提升国内系统稳定性，促进软件质量发展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次调查问卷由中国信通院联合混沌工程实验室、infoQ、VCEC、中国云原生社区共同发起，参与问卷的用户有机会获得电脑包、文化衫等精美礼品，扫描上方二维码进入问卷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"混沌工程实验室成员包括："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/05\/38\/059b75d168a41a99d5aeef0a116ee838.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}