零故障上云全过程再现，PB级数据迁移如何保障一致性？

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、美图秀秀介绍"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1b\/1b63f8125cb008db917277776bdffed3.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们来看一组数据，截止今年上半年，美图应用月活跃用户数超过3亿，独立设备数超过20亿，每个月美图秀秀要处理的图片以及视频数超过30亿。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这些数据背后还有一些看不见的数据，在全国各地我们有6大自建机房，数以百计的服务器，十几款软硬件产品，PB级的业务数据。在支撑这些数据的过程中，我们逐渐的发现了一些问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是运维管理复杂，因为有这么多的服务器，随着一些服务器过保老化，故障率逐步上升。为了业务的稳定性，需要投入大量的研发人力去监控这些机器的监控状况。每隔一段时间，我们就会看到公司大群里，内购一些硬盘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"美图秀秀的业务流量有极强的周期性，周末比平时峰值流量高，元旦春节峰值会是平时的很多倍。为了支撑每年元旦春节流量，业务部门首先会预估流量，然后采购部门都会提前按照预估峰值流量采购服务器，然而春节过后流量下来，一部分服务器资源会闲置，变成一种浪费。需要付出的成本较高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"美图内部多个部门服务及组件重复建设的情况也比较普遍。但是由于有的服务组件与业务绑定的过于紧密，使得美图内部孵化的一些创新产品无法快速使用，导致创新产品的发布周期较长。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们看到了这么些问题。应该怎么解决？那换个角度思考，从这些问题，看看我们需要的是什么？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/53\/533c09caf90a35112fcc308cc5513fae.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其实我们需要的是运维简单的弹性计算和存储资源，开箱即用的服务和组件共同支撑起应用的快速开发。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过什么样的方式能够实现我们想要的这些能力？经过公司仔细调研，我们给出的答案是云服务。公司在19年下半年提出了上云的计划。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、上云的挑战与应对"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那对于这么庞大体量的产品要上云，应该怎么做呢？又会遇到些什么问题呢？接下来就给大家带来的是上云的什么挑战，以及我们应该怎么去应对呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可能有人会说上云有什么难的？把服务部署上去不就好了。一开始我们接到上云任务的时候也是这么想的。然后在讨论这个事情的时候，组内的资深技术专家给大家提了这样的一个问题？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6f\/6fd4c6d3b358c4e14f4a5f1d0640fd2e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把大象装进冰箱里，总共分几步？当然我们都知道答案。3步，第一步把冰箱门打开，第二步把大象装进去，第三步把冰箱门关上。上云就相当于把大象装进冰箱。简单来说上云也分3步："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一步在云上创建对应的服务和资源"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二步把数据同步上去"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三步把流量切换到云上。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每一步听起来都很简单，对不对？但是具体每一步该怎么做，细节就是魔鬼？我们有3亿的月活，海外用户的占比也不小，我们需要为用户提供的是7X24小时的不间断服务，在上云的过程中也不能中断。另外是我们有PB级数据，用户数据是我们最核心的资产，不管在哪个环境下，我们都需要保证用户数据的完整性和一致性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单来说就是用户需要完全无感知，这就要求我们整个上云的过程需要做到平滑和稳定。接下来主要围绕着如何做到平滑和稳定来跟大家分享上云的一些方案设计。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/50\/50c4e1b096095f2945912bfd96dbd382.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"去掉一些细枝末节，我们可以将业务系统的架构简化为3层，最上层的负载均衡，中间的web服务，下面是资源层，对应着3个层面的迁移，流量的切换，应用的迁移和资源的迁移。另外还有依赖的外部服务。如果外部依赖不迁移，跨IDC的访问，专线有1ms的延迟，公网有几毫秒的时延。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们将依赖服务按照跨IDC的时间敏感度分为强依赖服务和弱依赖服务。对于强依赖服务，需要在我们业务上云之前部署在云上，弱依赖服务可以按照服务提供方自身的上云顺序进行，但是我们仍然需要关注的是依赖服务跨IDC访问的一个带宽的情况。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每一个迁移操作我们都会按照“准备”—>\"执行”—>\"验证”—>”观察”的方式来操作，确保每一个操作都没有问题才进行下一个操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来我会按照大象装冰箱的一个步骤，以应用、资源和流量的顺序跟大家分享各部分内容具体是如何迁移的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、把冰箱门打开：应用迁移"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是应用的迁移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a7\/a71f2f0550d699efc5a8cca874a3b9de.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在云下我们有自己的一套容器化平台，可以支撑业务的容器化需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此应用迁移，我们会将其划分为两种类型，一种是容器化应用，另一种是非容器化的应用。由于容器化平台的建设和改造提前上云，可以让业务在云上部署应用的时候采用跟IDC相同的方式，所以大大简化了应用上云的复杂性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于另一类非容器化应用，因为我们的应用都是无状态的服务，所以上云主要考虑的是在云上部署的问题。那是部署在虚机上，还是各方面性能都更好的裸金属服务器？另外在选择组件的时候，我们是应该自建，还是使用云上的开源组件，或者云上自研的组件？其实这里涉及到一个云上服务及组件的选型问题。那我们遵循的一个选型原则是：最大限度的复用已有的流程和系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因为我们的目的是服务上云，不是做架构或者应用优化。上云本身是一个复杂的工程，我们希望在上云的过程中尽量控制变量的发生。在上云的过程中不会刻意去改动架构和流程，因为这样会进一步的增加上云的复杂度和时间，导致故障的发生。当然也不是绝对的不变，为了解决上云中的一些问题，该变的还是要变。就是不要为了“顺便”做个架构优化而去变。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、把大象装进去：数据迁移"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"说完了应用迁移，相当于打开了冰箱门，接下来是数据迁移，该装大象了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d48cb0bac45587cb12322f0206e09266.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据分为两部分，一个是业务数据，一个是媒体数据。业务数据也包括了很多，比如MySQL的数据，缓存Memcache，Redis，Elasticsearch等等。今天业务数据我们只跟大家介绍MySQL和Memcached。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在下面的分享中，大家可以看到MySQL中的持久化数据，Memcache中的缓存和媒体文件作为3种常见的数据，分别采取了不同的迁移方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/73\/739b61a9c6c91bf9137a4cadcb354103.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要迁移PB级的数据量，仅仅依靠公网带宽做数据传输是远远不够的。那应该怎么办？很多人会想到搭专线，没错。我们也是使用专线来进行数据的传输。但其实公有云还提供了一种方式，就是数据快递。使用方式是，你预约一个数据快递服务，然后云厂商会给你邮寄一个几十或者几百T容量的硬件设备，真的是邮寄，你把这个设备插到服务器上开始同步数据，同步完成后寄给云厂商，云厂商会把这些数据在云上用对应的存储恢复出来。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们有PB级的数据，好几百个数据快递在路上跑，听起来总有一种不靠谱的感觉。当然我们也没有采用这种方案。说回专线。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在各个机房之间提早搭建了专门用于数据传输的专线。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"业务跨IDC访问流量走业务专线，业务数据传输通过数据传输专线，然后媒体文件走存储专线。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在右边公有云上，针对不同的集群，我们会划分vpc来做网络隔离，vpc之间通过虚拟路由来连接。另外还会搭建一套云上的测试环境。主要用于服务组件的压测和迁移方案的演练。后面绝大多数方案中涉及到的一些问题都是通过在测试环境压测和演练发现的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在专线带宽搭建好之后，我们开始进入mysql的数据迁移。在数据迁移前，我们会在云上搭建一套跟IDC配置一样的mysql的主从集群。因为本身业务做了读写分离，所以主从节点都分配有各自的域名。一开始公有云上的读写域名都指向IDC。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们依靠云上提供的数据复制服务来做数据同步。将IDC的数据同步到云上。同步分为两个阶段，刚开始是存量数据同步，之后是增量数据同步。云上数据复制服务会告诉你什么时候存量数据同步完了，什么时候开始增量数据同步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3b47b373b01cf27dceab4bc8b954cd1b.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么MySQL数据迁移的完整性如何保证？我们从两方面来看，首先是数据复制服务的设计，我们可以将数据复制服务理解为也是MySQL的实例，使用的同步机制跟MySQL的主从同步一样，理论上来说就是相当于是挂载了一个MySQL的从库，由MySQL的主从机制来保证数据的完整性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外我们还可以从两个维度来验证数据的完整性，一个是数据复制服务提供了按照表行数和表内容的对比功能，同时，从业务层面我们也提供了相应的接口可以抽样对比两个数据库中的数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在数据同步完成以及校验完成之后，下一步就是读写流量的切换？切换顺序是先切读再切写。流量切换只需要变更对应域名的DNS配置。当域名的DNS配置变更完成后，并且经过验证变更已经生效后，mysql的流量应该切换到云上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是在验证环境，我们通过流量监测脚本发现IDC的mysql集群仍然有来自云上的读写流量？通过ip对比，确认这些流量来自一部分服务，通过在这些服务的机器上去查看tcp连接，我们并没有发现来自云上服务的ip。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那我们首先想到的是不是DNS的缓存？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在对应的机器上执行nslookup域名解析，域名确实已经解析到云上了。机器上域名解析正常。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再一个怀疑的是不是服务里面有dns缓存的机制？当我们在服务里面去做域名解析的操作发现依然是解析到云上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"都不是，那会是什么问题？重启大法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们重启一部分服务，发现服务起来之后连接全部到云上。没有IDC的连接。说明不是域名解析的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后我们在另外的没有重启的服务上发现了一个现象，就是新建的连接都来自云上，老的一些连接来自IDC。从这里发现我们不同服务使用的数据库组件，有的有dns感知，有的没有dns感知功能。当域名变化时，没有dns感知功能的服务不会主动释放老的连接创建新连接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因为本身服务使用的组件都比较老，变更版本带来了很多依赖兼容性问题，所以我们放弃在服务本身上变更。转而在方案上做文章。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b9\/b99509e665ec6e276162dd6856b254de.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在计算机系统设计领域有这样一句经典语录：”计算机系统设计里面的任何问题都可以通过增加一个中间层来解决\"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们通过增加一个VIP来解决域名变更后长连接不断开的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"切完写之后，流量全部切换到云上，此时IDC没有写入流量，这时候IDC的MySQL主从集群已经不能用了。整个切换操作就完成了。如果方案到这里就结束了，那这只是一个正常的切换操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0d\/0d35b6992976c46d3e0191c013626bcb.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那再问一个问题，如果这个时候云上服务出现故障，需要回滚怎么办？已经回滚不回去了，数据已经不一致了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么我们该怎么解决切写之后的回滚问题呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bd\/bda2f8195199c57251a792cc976f3487.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"想一想这里回滚不了是因为什么？因为IDC没有完整的数据了。那现在完整的数据在哪里在云上，那如果我把云上的数据同步到IDC一份不就又是一份完整的数据了吗？但是同步的时机肯定不是出问题之后，而是应该在出问题之前。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/55\/5513dead9dcbd8964190baf3b2001d17.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顺着这个思路，在一开始的时候，我们就在IDC重新准备了一个备库集群。将公有云上的数据又同步一份到备库集群。这样可以保证在切写之后，IDC还是存在一份完整的数据，可以保证流量的回滚。在回滚后，将数据复制服务的流向变更为从最右边的集群到共有云。然后重复前面的过程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对MySQL的数据迁移，这里有些小Tips跟大家分享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4e\/4e8aa4729bd343c4ac89683df03c00fb.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一点，是数据迁移需要做数据对比，对比从两方面入手，一个是迁移工具的验证，一个是业务层面的验证。另外要注意迁移源库的内存使用率，因为在做数据对比的时候，源库的内存使用率会上升，如果源库本身内存容量不足，有可能造成源库oom。在实际迁移过程中，我们就遇到了源库oom的情况。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二点，针对一些按年，按月建表的业务，不要忘记了数据库表创建的脚本要同步到云上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三点是数据库复制服务的限速，一个是复制的过程中不要给源库造成太大的压力，另外迁移过程中有多个应用多种数据迁移，需要协调专线的使用率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四点是 在做切换操作时，需要将切换操作脚本话，一键切换，只需要执行一条命令。可以减少出错的机率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介绍完MySQL的数据迁移，下面介绍Memcached。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5c\/5c622497ce5f0f782313a0b594f0c474.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看下作为缓存的memcache是如何迁移的。美图秀秀重度依赖mc作为缓存。mc的同步采用的是双写+就近读的方案。云上和IDC会互相写数据，但是读流量只会读本数据中心的mc。另外在流量切换之前需要有一个预热的过程，预热既可以在业务层面做，也可以通过灰度的流量自动预热，利用灰度预热需要我们实现预估好穿透的量，避免给mysql造成太大的压力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在测试环境使用mc的过程中发现了不少的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3d\/3dfc7d6d2dcd1b0b4f648402d39e6613.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在稳定运行期间，mc容量写满之后性能下降比较明显。而且数据写得越多，性能下降越厉害。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过云上监控发现，容量满后，对应虚机swap开始上涨。发现云上mc实例默认开启了swap。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们使用mc的姿势就是申请一定容量的mc实例，然后一直写数据，写满后依赖mc自身内存管理机制来淘汰数据。有一些业务方跟我们使用姿势不一样，会开启mc的swap。开启swap，当访问到磁盘上的数据时性能就下降的比较厉害。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mc平稳运行一段时间后会突然挂掉？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个问题跟前面关闭swap有一定关系，因为mc创建在虚机上。比如我们申请的mc实例是8G，那虚机内存8G，因为操作系统会占用一定的内存，所以mc实际使用的内存可能不会超过6.4G，当我们不断写入数据之后，由于关闭了swap没法做数据交换，在内存分配和销毁的过程中不断产生内存碎片。当整个mc的内存+操作系统内存超过8G后，虚机会重启，导致mc挂掉。这里需要的是设置mc的max_memory的值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mc的穿透率不平稳，时高时低？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"穿透率与mc的淘汰策略有关系，虽然业务也会设置mc的过期时间，但是在业务设置的mc过期之前，就存在穿透率不平稳的问题。然后发现云上mc的淘汰策略设置的是随机淘汰。互联网应用大多是读多写少的模式。因为我们首页请求量很大，需要缓存来抗读的量，越热的数据越需要有被缓存住，如果是随机淘汰可能会因为淘汰掉热数据带来一定的穿透量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于我们重度依赖mc作为缓存，mc的查询性能直接影响到上云后系统的稳定性。因此在上云前会在压测平台上拷贝线上真实环境的流量对云上的mc进行了压测。但是压测的结果显示，mc的读性能跟idc相比有一定的差异，差异大得有点令人诧异。具体现象是压测流量到达某个点后，mc会出现大量的数据淘汰，同时伴随着有大量的新连接创建，CPU利用率升高，ops降低，然后5-10分钟后又自动恢复的现象。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跟云上工程是沟通发现，云上mc不是原生的mc实现，是使用redis+mc协议实现的。当然云上有自己的考虑，使用redis主要是为了数据持久化以及HA。但是我们知道， redis内存管理基于Jemalloc内存分配器，Jemalloc频繁分配和释放内存会导致内存碎片。而原生的mc使用slab的内存管理，内存满后，淘汰同等大小的slab，直接覆盖数据，不会造成内存碎片。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当碎片率到达一定程度会触发内存的主动淘汰，由于主动淘汰时间过长，影响了mc的读写性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一个我们业务上使用的mc客户端，当发现mc时延超过一定阈值后，会重新创建连接。由于服务数过多，导致单个mc上会看到大量的新建连接过来。每一个连接又会占用一定的mc内存，从而进一步恶化整个mc的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对这个问题，云上进行了mc的性能优化攻关，最终将mc的性能提升到跟idc的一个量级。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在mc的性能这点，其实前面的参数都不重要，重要的是我们对于云上服务和组件应该保持的一个态度。引用一句当年美苏中程核武器谈判时，美国总统里根对戈尔巴乔夫说的一句话俄语谚语：“Trust，But Verify”。信任，但要核实。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有点儿奇怪的一个场景，一个中国人演讲里面引用的是美国人对俄国人说的俄语谚语。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"媒体文件的迁移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/15\/15fef0e949e180e8a3b98ef8623c2b84.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"特点数据量大，迁移时间长，缓存数据双写同步按天记，数据库同步按周，媒体要按月。数据迁移依然使用云上的obs迁移工具，支持断点续传。同时还需要我们在业务层面进一步做数据对比。全量数据做etag对比，确保数据完整迁移到云上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"切读依赖CDN更改域名。域名指向变更后读流量就逐步迁移到云上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"切写流量依赖上传SDK组件变更上传域名。我们可以下发需要上传的域名给到sdk组件。但是客户端会有一段时间的缓存，因此始终都会存在一部分客户端将数据上传到IDC，一部分上传到公有云。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，对于媒体数据的迁移，我们并没有做什么快速切换的方案。过渡态是一定会出现的。因此主要解决过渡状态下，数据访问的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e5\/e5bf233a30e2d6806ecf5c842116a4aa.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里有一个302系统，302系统是干什么用的呢？302的含义来自于HTTP 状态码302，是页面跳转的意思。这里也一样。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"302系统在这里可以解决什么问题呢？当我们在切换的过程中，首先读肯定是读本数据中心的obs，比如读公有云上的obs，当公有云上的obs返回数据不存在时，经过302系统，将请求跳转到IDC的obs上去查询，如果有就返回，如果没有就返回404。在IDC也一样。有了302系统就不担心流量在切换的过程中找不到媒体文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6c\/6cb80587222dd66aa4d6ccc8d6ecfaa8.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、把冰箱门关上：流量切换"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后我们看一下如何关闭冰箱门。流量是如何切换的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/08\/089f8d726af585caa6342a22dc02bda0.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从图中可以看到，我们对外的API域名有两种配置，一个是A记录，一个是CDN。A记录流量占比大约30%，CDN70%。我们会将A记录的流量当做灰度流量来操作。先将A记录的一部分流量切到云上，指向云上的LB。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/95\/950721f68a68b9d6b47b281148380d67.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"稳定灰度一段时间，观察服务状态没问题后，然后开始切CDN，一家一家的切。cdn切完，全部流量都到云上。此刻检查，IDC是否还有流量访问。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a8\/a87d48438119f68e8f5f0207696bcb4e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家可能也发现了，这里存在一个问题，切换DNS和CDN的生效时间比较长，同时回滚的话也没办法快速回滚。那应该怎么解决？这时候想起了一个计算机系统设计的经典语录：计算机系统里面的所有问题都可以通过增加一个中间层来解决。通过增加一层LB来解决切换生效时间比较长的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面我们做了应用迁移，数据迁移，当这里流量迁移完成后。整个上云的步骤就结束了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、云上稳定性建设"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其实不管是在云上，还是云下，我们都会因为下面这3个因素导致系统故障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1d\/1daa75562e53b6a5c387f0b95b44c99e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自然灾害，物理损坏，逻辑错误。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上云的时候，我们的服务都是部署在同一个机房的，上云完成后，为了进一步提升服务的稳定性，我们进行了一些云上的容灾建设。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在云上我们采用的是两地三中心的方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/88\/880ffa2d91bdbb908ddbd5568d031cf7.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对机房级的故障，我们会在云上的RegionA部署两个AZ。一个AZ相当于是一个机房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用云上同步组件同步资源数据。通过云上的DNS解析，将流量划分到不同的AZ下。对于AZ我们也有一个主次的划分，AZ1作为主机房承担70%的流量，AZ2承担30%的流量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"帮助我们压测的时候可以将流量全部切到一个AZ，对于两一个AZ进行压测。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外为了避免Region级的故障，我们还在Region B的一个机房里部署了另外一套服务，这套服务通过云专线跟Region A联通。在RegionB我们只做核心服务的业务容灾，非核心服务只负责数据容灾。regionB可以保证我们遇到Region级的故障的时候，可以将核心服务切到RegionB，保证用户的主要功能可用，另外可以通过RegionB的数据恢复非核心服务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二个想跟大家分享的是全链路的监控体系。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/80\/8086f98bf13aa65a48d872fcadc2f4aa.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们的监控体系是包括客户端和服务端的一体化监控。客户端的监控体系在整个上云过程中发挥了重要的作用。在前面的方案中，多次提到了灰度，回滚。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么在灰度操作之后，是应该继续灰度还是回滚，我们的判断依据是什么？依据就是观察到的客户端监控体系里面的网络质量，错误率，错误率，慢请求这些指标。监控触达到离用户越近的地方，越能体现每个操作的真实影响范围，影响程度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们内部对于故障的定级有一套定级标准，针对影响的功能，用户人数，时长分别对应到不同的故障等级。那回滚标准就可以简单的依据客户端监控的指标来判断。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那上云后，服务监控和基础监控这两块更多的是利用云上的监控数据来构建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"监控报警就像我们人的一双眼睛，可以帮助我们更快的发现系统的一些问题。对于监控报警，我们做了如下的一些优化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c6\/c6e0c7367aa70794680d51c0dac571b7.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是通过grafana统一了所有的监控报表，每个业务的所有相关报表都统一到grafana，当系统出现问题的时候，我们不再需要到处去找监控页面查看。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，使用grafana的flowcharting插件，我们将各业务从客户端到服务端的整个链路的健康状况展现在一张监控大盘上。用红黄绿来代表服务的健康程度，绿色代表健康，黄色代表不健康，需要修复，红色代表异常，需要立即人工介入。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后从大盘上我们就可以清晰的看到哪些服务是正常的，哪些服务目前处于非正常状况。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时，针对那些不正常的服务，我们利用grafana的另外一个插件Grafana image Renderer将当时的服务状况发送到企业微信报警群里，实现图文报警。业务开发和运维就能及时介入处理问题。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、避坑指南"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后跟大家分享的是上云过程中的避坑指南。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要想跟大家介绍下上云的整个流程。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0ec54b93b6abc82065d2cad7c2ff10c6.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个上云流程我们分为4步，评估调研，规划设计，实施和验收。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"评估调研分为信息收集，业务分析，风险评估和迁移策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"评估调研主要是在设计方案前的准备工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"信息收集分为两方面，一个是自身业务的信息收集，一个是公有云上的信息收集。信息收集完成之后，结合业务分析，来制定迁移策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在规划设计阶段，我们在设计迁移方案的时候需要考虑方案如何灰度，各种可能出问题后的回滚操作是什么样的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外就是准备应急预案，提前列出关键的可能发生问题的事件，针对该事件的处理方式。墨菲定律。然后需要准备非常详尽的迁移手册。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实施阶段主要想强调一下正式迁移前的演练。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后验收，验收阶段一个是需要通过监控看到上云后的性能情况。另外一个也是需要通过监控对上云后的一些问题做进一步的优化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d2\/d232197701ac5dad0ecb593e2dbff2d4.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括在迁移过程中和上云后出现的一些问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"资源实例的配额，全方位的限制。木桶效应，流量达到一定程度后，总会遇到遇到最短的那块板子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是可以通过全链路压测来触发云上一些资源的瓶颈点。另外是我们的监控有时候可能过多的关注了业务本身的情况。这就要求我们将监控的范围进一步的扩大，更完善的监控指标让我们发现问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"云上组件异常。没有什么好的方式来避免，更多的通过监控来发现问题，另外就是针对组件做好容灾、降级的服务建设。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对人为操作的问题，更多的是规范化流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"计划与现实的差异，道路坎坷，那我们是如何做到上云过程中0故障的了，我们的压舱石是什么？前面分享的具体的方案，参数，操作步骤都不重要。给大家总结的就3句话："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/60\/10\/606152e58795b76a6967dcbfd8aaf610.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过信息收集和业务分析设计可灰度可回滚的迁移方案。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"制定详细的迁移手册在测试环境提前演练。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在迁移过程中利用贴近用户的监控体系及时发现问题。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：数据迁移会不会导致数据错乱？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1："},{"type":"text","text":"数据迁移过程中可能导致数据错乱的情况主要是切写的场景。因为两个数据中心之间做数据同步有1-2ms的延迟，在做写流量的切换时，如果存在在1ms以内针对同一条数据的相同字段的更新操作，且两个更新操作分别在不同的数据中心执行，当自建机房的数据再次同步到云上的时候理论上可能存在新的更新操作被覆盖的情况。但是在我们这种应用场景下，这种数据错乱的情况出现的机率会极低。主要是因为我们的内容都是用户主动生产的，用户针对同一条内容的数据变更不会很快发生。所以在切换过程中不会存在数据错乱的情况。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：obs的地址存放在数据库中存一份吗？迁移后是否需要修改？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2："},{"type":"text","text":"obs的绝对路径地址并没有保存在数据库中，只会将obs文件的文件名保存在数据库，另外还有一些obs文件对应的bucket等属性。针对obs的迁移，迁移过程中obs文件名不会变化，bucket也不会变化。除非云上已经存在同名的bucket或者bucket名称不符合云上的规范，我们需要变更bucket的时候，需要修改对应的数据库中的bucket的值。其他的不需要修改。变化的只是访问域名，迁移后只需要修改访问域名就可以了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：迁移如何做数据一致性校验？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3："},{"type":"text","text":"分为两个维度来做一致性校验。一个是利用云上的数据复制服务，在数据迁移过程中，数据复制服务会提供数据的存量以及增量数据的对比，对比包括行数的对比和内容的对比。另一个是在业务层面，针对迁移的数据进行业务校验，可以抽样一批数据进行业务层面的数据校验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：你们迁云之后，制定了怎样的风险策略？如何规避安全漏洞呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4："},{"type":"text","text":"这里说的风险策略我可以理解为我们提前做的预案。预案主要是针对迁移过程中可能发生的情况采取的执行操作。其实主要是在每一个重要操作之前问一下如果这个操作失败了该怎么处理？针对这样的问题，逐一设计方案解决。另外就是针对一些不可控的因素做假设。比如专线断了怎么处理？专线带宽满了怎么处理等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"安全漏洞这一块儿因为不是我负责的，我也不太了解。如果同学非常感兴趣，我可以找下相关人员帮忙解答一下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：请问你们的业务迁云之后，成本是如何控制的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5:"},{"type":"text","text":"首先迁移前我们制定了架构平移的策略，所以迁移过程中不会刻意压缩上云的成本。为了保证上云服务的稳定，云上所使用的的服务会略微高于IDC的配置。即使这样的策略下，上云后成本也比IDC要低。另外在上云完成后，需要成立成本控制中心，从DBA，运维，一直到开发都会需要有成本控制的意识。优化已有架构，业务逻辑等等来降低成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：数据库迁移上云有哪些注意事项？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6: "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据库迁移要提前启动，因为数据迁移相比应用和流量迁移的时间要更久，出现问题影响面也更大；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据库迁移需要针对数据做一致性校验；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有一些月表或者年表的创建脚本不要忘记了迁移到云上；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"尽量在低峰期执行，操作要页面化，脚本化。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"嘉宾介绍："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"靳洪兵，"},{"type":"text","text":"美图技术专家。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"美图用户产品研发部技术专家，主要负责美图秀秀社区服务研发，主导参与美图秀秀调度系统、内容中台等多个项目的研发与设计。在分布式系统、消息中间件等方面具有丰富经验；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"曾就职于新浪微博，负责平台研发部消息中间件、配置服务等系统的研发与设计。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：dbaplus社群（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/b8nZgMU_jXB_KkH1BgZFzg","title":"xxx","type":null},"content":[{"type":"text","text":"零故障上云全过程再现，PB级数据迁移如何保障一致性？"}]}]}]}