分布式系统：数据一致性解决方案

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分布式系统中，随着系统架构演进，原来的原子性操作会随着系统拆分而无法保障原子性从而产生一致性问题，但业务实际又需要保障一致性，下面我从学习和实战运用总结一下分布式一致性解决方案。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. CAP & Base理论"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"CAP定理"},{"type":"text","text":"指的是在一个分布式系统中，一致性（Consistency）、可用性（Availability）、分区容错性（Partition tolerance）。这三个要素最多只能同时实现两点，不可能三者兼顾："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"一致性"},{"type":"text","text":"：在分布式系统中的所有数据备份，在同一时刻是否同样的值。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"分区容错性"},{"type":"text","text":"：可靠性，无论应用程序或系统发生错误，还是用户以意外或错误的方式使用，软件系统都能继续运行。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"可用性"},{"type":"text","text":"：在集群中一部分节点故障后，集群整体是否还能响应客户端的读写请求。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c3/c3673d6bec7976a0ec05557e3eaadc69.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CAP理论3选2是伪命题，实际上必须从A和C选择一个和P组合，更进一步基本上都会选择A，相比一致性，系统一旦不可用或不可靠都可能会造成整个站点崩溃，所以一般都会选择AP。但是不一致的问题也不能忽略，使用最终一致是比较好的办法。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"BASE理论"},{"type":"text","text":"是对CAP中一致性和可用性权衡的结果，其来源于对大规模互联网系统分布式实践的总结， 是基于CAP定理逐步演化而来的。BASE理论的核心思想是：即使无法做到强一致性，但每个应用都可以根据自身业务特点，采用适当的方式来使系统达到最终一致性。接下来看一下BASE中的三要素："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"基本可用（"},{"type":"text","text":"Basically Available"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：基本可用是指分布式系统在出现不可预知故障的时候，允许损失部分可用性。注意，这绝不等价于系统不可用。比如："}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"响应时间上的损失"},{"type":"text","text":"。正常情况下，一个在线搜索引擎需要在0.5秒之内返回给用户相应的查询结果，但由于出现故障，查询结果的响应时间增加了1~2秒"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"系统功能上的损失"},{"type":"text","text":"：正常情况下，在一个电子商务网站上进行购物的时候，消费者几乎能够顺利完成每一笔订单，但是在一些节日大促购物高峰的时候，由于消费者的购物行为激增，为了保护购物系统的稳定性，部分消费者可能会被引导到一个降级页面。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"软状态（"},{"type":"text","text":"Soft State"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：软状态指允许系统中的数据存在中间状态，并认为该中间状态的存在不会影响系统的整体可用性，即允许系统在不同节点的数据副本之间进行数据同步的过程存在延时。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"最终一致（"},{"type":"text","text":"Eventually Consistent"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：最终一致性强调的是所有的数据副本，在经过一段时间的同步之后，最终都能够达到一个一致的状态。因此，最终一致性的本质是需要系统保证最终数据能够达到一致，而不需要实时保证系统数据的强一致性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 重试"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在出现一致性问题时如果系统的并发或不一致情况较少，可以先使用重试来解决。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/7642570cfc4255c9af4d065fda33e65c.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在调用Service B超时或失败时进行重试："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"同步调用"},{"type":"text","text":"，捕获异常重新调用Service B。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"异步消息"},{"type":"text","text":"，捕获异常发送延迟消息重新调用Service B。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"异步线程"},{"type":"text","text":"，捕获异常开启异步线程重新调用Service B。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果重试还是不能解决问题，那么需要使用分布式事务来解决。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 分布式事务"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于分布式一致性问题可以采用分布式事务来解决。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1 2PC-XA协议"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA事务由一个或多个资源管理器（Resource Managers）、一个事务管理器（Transaction Manager）以及一个应用程序（Application Program）组成。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"资源管理器（RM）："},{"type":"text","marks":[{"type":"strong"}],"text":"参与者"},{"type":"text","text":"。提供访问事务资源的方法。通常一个数据库就是一个资源管理器。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事务管理器（TM）："},{"type":"text","marks":[{"type":"strong"}],"text":"协调者"},{"type":"text","text":"。分配标识符，监视事务的进度，并负责事务完成和故障恢复。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"应用程序（AP）："},{"type":"text","marks":[{"type":"strong"}],"text":"发起者"},{"type":"text","text":"。定义事务的边界，制定全局事务中的操作。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个过程分为2个阶段：准备(prepare)和提交(commit)，prepare前需要先执行对应的DML操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b9/b972299aec9cc3d6a012e3dfa6168fba.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mysql XA事务语句："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"注：执行前需要先生成全局唯一的xid\n\nXA {START|BEGIN} xid [JOIN|RESUME]\nDML操作\nXA END xid [SUSPEND [FOR MIGRATE]]\nXA PREPARE xid //1.准备\nXA COMMIT xid [ONE PHASE] //2.提交\nXA ROLLBACK xid\nXA RECOVER"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单的事例："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"//准备前阶段\nmysql> XA START 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n\nmysql> INSERT INTO mytable (i) VALUES(10);\nQuery OK, 1 row affected (0.04 sec)\n\nmysql> XA END 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n//准备\nmysql> XA PREPARE 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n//提交\nmysql> XA COMMIT 'xatest';\nQuery OK, 0 rows affected (0.00 sec)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA中的TM事务管理器一般是由应用自己实现"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA看起来很美好，但还是存在一些问题："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"commit丢失/超时导致数据不一致"},{"type":"text","text":"。A commit成功，B commit时网络超时导致数据库未收到commit请求。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"性能低"},{"type":"text","text":"。所有事务参与者在等待其它参与者响应的时候都处于同步阻塞状态。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"主备数据不一致。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"协调者单点故障，在任意阶段协调者发生故障都会导致分布式事务无法进行。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2 3PC"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3PC通过在参与者加入超时机制解决2PC中的协调者单点带来的事务无法进行的问题，但是性能和一致性仍没有解决。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.3 TCC"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这样看下来强一致是很难做了，还是最终一致把。。。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TCC是通过最终一致来达到分布式事务的效果，即：在短时间内无法保证一致性，但最终会一致。核心分为2个阶段："},{"type":"text","marks":[{"type":"strong"}],"text":"1.Try  2.Confirm or Cancel。"},{"type":"text","text":"先尝试（Try）操作数据，如果都成功则全部确认（Confirm）该修改，如果有任意一个尝试失败，则全部取消（Cancel）。简单点说就是"},{"type":"text","marks":[{"type":"strong"}],"text":"增加了中间态和可回改能力"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/56/56174f0fb8a4a12c4d13b546f1ca099b.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通过重试机制保障Confirm和Cancel一定能成功。"},{"type":"text","text":"TCC解决了性能问题，但是业务系统想要实现对业务代码的侵入性很大，可以学习一下阿里GTS是如何做的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.4 事务管理器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在实际运用中事务管理器（TM）一般是由应用程序（AP）兼职实现的，这样对业务代码侵入性大，最好是把TM做成中间层。接下来详细看一下TM的职责："},{"type":"text","marks":[{"type":"strong"}],"text":"协调者。分配标识符，监视事务的进度，并负责事务完成和故障恢复"},{"type":"text","text":"。TM需要具备持久化能力才能完成监控、故障恢复和协调，整个协调过程可以分为3个阶段："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"准备阶段"},{"type":"text","text":"：向TM注册全局事务并获取到全局唯一XID，这一步需要定义好事务的范围。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"执行阶段"},{"type":"text","text":"：应用程序AP执行业务功能操作自己的RM，全部执行完毕后向TM commit，如果无失败则整个事务成功。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"确认/回滚阶段"},{"type":"text","text":"：如果第2步出现异常会触发该阶段，TM咨询各个AP对应操作是否成功，如果成功则commit，如果失败则调用AP进行rollback。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面简单介绍几种实现方式。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.1 本地事务管理器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于简单的业务可能只要保障2个数据库之间的一致，这样在本地实现事务管理器比较快成本也不高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e31dc0013a7b93f4a142a7f419ef5538.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"在做业务逻辑之前把对应事件添加到本地event表中（记录订正时所需要的关键数据）"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"执行业务逻辑"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"本地业务逻辑，操作表数据等等"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"RPC调用其他服务"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"修改event状态为确认"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"如果第2步不成功，则event状态还是预提交，通过定时任务捞取再执行订正逻辑"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"检查业务逻辑中的操作是否完成，如未完成则执行订正逻辑"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"本地业务逻辑是否执行成功"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"RPC调用其他服务是否执行成功"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"返回订正结果"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"成功则修改event为确认提交"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"未成功则不操作，等待轮询订正"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.2 可靠消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用可靠消息一样可以实现最终一致，逻辑和本地任务基本一样，但是需要MQ支持回调服务的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3c/3ca1d3b09e49061f2411eca737e64a1f.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.3 分布式事务框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阿里GTS："},{"type":"link","attrs":{"href":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI","title":null},"content":[{"type":"text","text":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI"}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 兜底核对"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虽然有了分布式事务，但是在实际场景中可能会因为bug导致数据不一致，这时需要兜底来做最后一道防线，通过定时核对数据是否一致，如不一致手动/自动进行订正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.1 系统自核对"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果系统数量和数据量不多的情况下可以由业务系统自行核对，通过发送延迟消息"},{"type":"text","marks":[{"type":"underline"},{"type":"strong"}],"text":"自消费或监听其他系统消息"},{"type":"text","text":"做相关数据核对并进行订正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d6/d6bca7802d4d6cc11640e91a91fa6322.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.2 搭建核对系统"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在核对工作繁多的情况下，由业务系统自己核对会存在很多耦合，这时可以选择搭建独立的核对系统进行核对和订正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1.监听drc消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/81/81692e7a8eb9ae8ed3eaeeba17c3c0a4.png","alt":"image.png","title":"image.png","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2.监听业务消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e4/e4f482882d0fc398fc40ede81c638e95.png","alt":"image.png","title":"image.png","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"参考："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阿里GTS："},{"type":"link","attrs":{"href":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI","title":null},"content":[{"type":"text","text":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"饿了么DRC："},{"type":"link","attrs":{"href":"https://zhuanlan.zhihu.com/p/34958596","title":null},"content":[{"type":"text","text":"https://zhuanlan.zhihu.com/p/34958596"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mysql XA："},{"type":"link","attrs":{"href":"https://dev.mysql.com/doc/refman/8.0/en/xa-statements.html","title":null},"content":[{"type":"text","text":"https://dev.mysql.com/doc/refman/8.0/en/xa-statements.html"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}