分佈式系統：數據一致性解決方案

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分佈式系統中，隨着系統架構演進，原來的原子性操作會隨着系統拆分而無法保障原子性從而產生一致性問題，但業務實際又需要保障一致性，下面我從學習和實戰運用總結一下分佈式一致性解決方案。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. CAP & Base理論"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"CAP定理"},{"type":"text","text":"指的是在一個分佈式系統中，一致性（Consistency）、可用性（Availability）、分區容錯性（Partition tolerance）。這三個要素最多隻能同時實現兩點，不可能三者兼顧："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"一致性"},{"type":"text","text":"：在分佈式系統中的所有數據備份，在同一時刻是否同樣的值。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"分區容錯性"},{"type":"text","text":"：可靠性，無論應用程序或系統發生錯誤，還是用戶以意外或錯誤的方式使用，軟件系統都能繼續運行。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"可用性"},{"type":"text","text":"：在集羣中一部分節點故障後，集羣整體是否還能響應客戶端的讀寫請求。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c3/c3673d6bec7976a0ec05557e3eaadc69.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CAP理論3選2是僞命題，實際上必須從A和C選擇一個和P組合，更進一步基本上都會選擇A，相比一致性，系統一旦不可用或不可靠都可能會造成整個站點崩潰，所以一般都會選擇AP。但是不一致的問題也不能忽略，使用最終一致是比較好的辦法。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"BASE理論"},{"type":"text","text":"是對CAP中一致性和可用性權衡的結果，其來源於對大規模互聯網系統分佈式實踐的總結， 是基於CAP定理逐步演化而來的。BASE理論的核心思想是：即使無法做到強一致性，但每個應用都可以根據自身業務特點，採用適當的方式來使系統達到最終一致性。接下來看一下BASE中的三要素："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"基本可用（"},{"type":"text","text":"Basically Available"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：基本可用是指分佈式系統在出現不可預知故障的時候，允許損失部分可用性。注意，這絕不等價於系統不可用。比如："}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"響應時間上的損失"},{"type":"text","text":"。正常情況下，一個在線搜索引擎需要在0.5秒之內返回給用戶相應的查詢結果，但由於出現故障，查詢結果的響應時間增加了1~2秒"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"系統功能上的損失"},{"type":"text","text":"：正常情況下，在一個電子商務網站上進行購物的時候，消費者幾乎能夠順利完成每一筆訂單，但是在一些節日大促購物高峯的時候，由於消費者的購物行爲激增，爲了保護購物系統的穩定性，部分消費者可能會被引導到一個降級頁面。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"軟狀態（"},{"type":"text","text":"Soft State"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：軟狀態指允許系統中的數據存在中間狀態，並認爲該中間狀態的存在不會影響系統的整體可用性，即允許系統在不同節點的數據副本之間進行數據同步的過程存在延時。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"最終一致（"},{"type":"text","text":"Eventually Consistent"},{"type":"text","marks":[{"type":"strong"}],"text":"）"},{"type":"text","text":"：最終一致性強調的是所有的數據副本，在經過一段時間的同步之後，最終都能夠達到一個一致的狀態。因此，最終一致性的本質是需要系統保證最終數據能夠達到一致，而不需要實時保證系統數據的強一致性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 重試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在出現一致性問題時如果系統的併發或不一致情況較少，可以先使用重試來解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/7642570cfc4255c9af4d065fda33e65c.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在調用Service B超時或失敗時進行重試："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"同步調用"},{"type":"text","text":"，捕獲異常重新調用Service B。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"異步消息"},{"type":"text","text":"，捕獲異常發送延遲消息重新調用Service B。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"異步線程"},{"type":"text","text":"，捕獲異常開啓異步線程重新調用Service B。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果重試還是不能解決問題，那麼需要使用分佈式事務來解決。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 分佈式事務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於分佈式一致性問題可以採用分佈式事務來解決。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1 2PC-XA協議"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA事務由一個或多個資源管理器（Resource Managers）、一個事務管理器（Transaction Manager）以及一個應用程序（Application Program）組成。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源管理器（RM）："},{"type":"text","marks":[{"type":"strong"}],"text":"參與者"},{"type":"text","text":"。提供訪問事務資源的方法。通常一個數據庫就是一個資源管理器。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事務管理器（TM）："},{"type":"text","marks":[{"type":"strong"}],"text":"協調者"},{"type":"text","text":"。分配標識符，監視事務的進度，並負責事務完成和故障恢復。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用程序（AP）："},{"type":"text","marks":[{"type":"strong"}],"text":"發起者"},{"type":"text","text":"。定義事務的邊界，制定全局事務中的操作。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個過程分爲2個階段：準備(prepare)和提交(commit)，prepare前需要先執行對應的DML操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b9/b972299aec9cc3d6a012e3dfa6168fba.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mysql XA事務語句："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"注：執行前需要先生成全局唯一的xid\n\nXA {START|BEGIN} xid [JOIN|RESUME]\nDML操作\nXA END xid [SUSPEND [FOR MIGRATE]]\nXA PREPARE xid //1.準備\nXA COMMIT xid [ONE PHASE] //2.提交\nXA ROLLBACK xid\nXA RECOVER"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單的事例："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"//準備前階段\nmysql> XA START 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n\nmysql> INSERT INTO mytable (i) VALUES(10);\nQuery OK, 1 row affected (0.04 sec)\n\nmysql> XA END 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n//準備\nmysql> XA PREPARE 'xatest';\nQuery OK, 0 rows affected (0.00 sec)\n//提交\nmysql> XA COMMIT 'xatest';\nQuery OK, 0 rows affected (0.00 sec)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA中的TM事務管理器一般是由應用自己實現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XA看起來很美好，但還是存在一些問題："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"commit丟失/超時導致數據不一致"},{"type":"text","text":"。A commit成功，B commit時網絡超時導致數據庫未收到commit請求。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"性能低"},{"type":"text","text":"。所有事務參與者在等待其它參與者響應的時候都處於同步阻塞狀態。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"主備數據不一致。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"協調者單點故障，在任意階段協調者發生故障都會導致分佈式事務無法進行。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2 3PC"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3PC通過在參與者加入超時機制解決2PC中的協調者單點帶來的事務無法進行的問題，但是性能和一致性仍沒有解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.3 TCC"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣看下來強一致是很難做了，還是最終一致把。。。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TCC是通過最終一致來達到分佈式事務的效果，即：在短時間內無法保證一致性，但最終會一致。核心分爲2個階段："},{"type":"text","marks":[{"type":"strong"}],"text":"1.Try  2.Confirm or Cancel。"},{"type":"text","text":"先嚐試（Try）操作數據，如果都成功則全部確認（Confirm）該修改，如果有任意一個嘗試失敗，則全部取消（Cancel）。簡單點說就是"},{"type":"text","marks":[{"type":"strong"}],"text":"增加了中間態和可回改能力"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/56/56174f0fb8a4a12c4d13b546f1ca099b.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通過重試機制保障Confirm和Cancel一定能成功。"},{"type":"text","text":"TCC解決了性能問題，但是業務系統想要實現對業務代碼的侵入性很大，可以學習一下阿里GTS是如何做的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.4 事務管理器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在實際運用中事務管理器（TM）一般是由應用程序（AP）兼職實現的，這樣對業務代碼侵入性大，最好是把TM做成中間層。接下來詳細看一下TM的職責："},{"type":"text","marks":[{"type":"strong"}],"text":"協調者。分配標識符，監視事務的進度，並負責事務完成和故障恢復"},{"type":"text","text":"。TM需要具備持久化能力才能完成監控、故障恢復和協調，整個協調過程可以分爲3個階段："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"準備階段"},{"type":"text","text":"：向TM註冊全局事務並獲取到全局唯一XID，這一步需要定義好事務的範圍。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"執行階段"},{"type":"text","text":"：應用程序AP執行業務功能操作自己的RM，全部執行完畢後向TM commit，如果無失敗則整個事務成功。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"確認/回滾階段"},{"type":"text","text":"：如果第2步出現異常會觸發該階段，TM諮詢各個AP對應操作是否成功，如果成功則commit，如果失敗則調用AP進行rollback。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面簡單介紹幾種實現方式。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.1 本地事務管理器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於簡單的業務可能只要保障2個數據庫之間的一致，這樣在本地實現事務管理器比較快成本也不高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e31dc0013a7b93f4a142a7f419ef5538.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"在做業務邏輯之前把對應事件添加到本地event表中（記錄訂正時所需要的關鍵數據）"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"執行業務邏輯"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"本地業務邏輯，操作表數據等等"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"RPC調用其他服務"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"修改event狀態爲確認"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"如果第2步不成功，則event狀態還是預提交，通過定時任務撈取再執行訂正邏輯"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"檢查業務邏輯中的操作是否完成，如未完成則執行訂正邏輯"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"本地業務邏輯是否執行成功"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"RPC調用其他服務是否執行成功"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"返回訂正結果"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"成功則修改event爲確認提交"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"未成功則不操作，等待輪詢訂正"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.2 可靠消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用可靠消息一樣可以實現最終一致，邏輯和本地任務基本一樣，但是需要MQ支持回調服務的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3c/3ca1d3b09e49061f2411eca737e64a1f.png","alt":null,"title":"image.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4.3 分佈式事務框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阿里GTS："},{"type":"link","attrs":{"href":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI","title":null},"content":[{"type":"text","text":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI"}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 兜底覈對"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然有了分佈式事務，但是在實際場景中可能會因爲bug導致數據不一致，這時需要兜底來做最後一道防線，通過定時覈對數據是否一致，如不一致手動/自動進行訂正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.1 系統自核對"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果系統數量和數據量不多的情況下可以由業務系統自行覈對，通過發送延遲消息"},{"type":"text","marks":[{"type":"underline"},{"type":"strong"}],"text":"自消費或監聽其他系統消息"},{"type":"text","text":"做相關數據覈對並進行訂正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d6/d6bca7802d4d6cc11640e91a91fa6322.png","alt":null,"title":"image.png","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.2 搭建覈對系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在覈對工作繁多的情況下，由業務系統自己覈對會存在很多耦合，這時可以選擇搭建獨立的核對系統進行覈對和訂正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1.監聽drc消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/81/81692e7a8eb9ae8ed3eaeeba17c3c0a4.png","alt":"image.png","title":"image.png","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2.監聽業務消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e4/e4f482882d0fc398fc40ede81c638e95.png","alt":"image.png","title":"image.png","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阿里GTS："},{"type":"link","attrs":{"href":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI","title":null},"content":[{"type":"text","text":"https://help.aliyun.com/document_detail/157850.html?spm=a2c4g.11186623.6.554.47ae4df4Zy6hBI"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"餓了麼DRC："},{"type":"link","attrs":{"href":"https://zhuanlan.zhihu.com/p/34958596","title":null},"content":[{"type":"text","text":"https://zhuanlan.zhihu.com/p/34958596"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mysql XA："},{"type":"link","attrs":{"href":"https://dev.mysql.com/doc/refman/8.0/en/xa-statements.html","title":null},"content":[{"type":"text","text":"https://dev.mysql.com/doc/refman/8.0/en/xa-statements.html"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}