探祕RocketMQ事務機制，如何保證消息零丟失

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"真正的大師永遠懷着一顆學徒的心","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"引言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事務的概念就不用多說了，我相信閱讀文章的童鞋都是有着非常深刻的認識。我們都知道","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"MQ","attrs":{}}],"attrs":{}},{"type":"text","text":"可以實現微服務之間的異步以及解耦，那麼引入","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"MQ","attrs":{}}],"attrs":{}},{"type":"text","text":"之後，如何實現微服務之間的數據一致性是一個值得思考的問題。RocketMQ事務消息正是解決這個問題的解決方案。另外事務消息也是爲了解決消息丟失問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"哪些場景會出現消息丟失","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分析","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"事務消息之前，我們先來分析下引入消息中間件之後，整個消息鏈路在哪些場景會出現消息丟失的異常情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當我們支付訂單之後，我們賬戶的購物積分也會進行相應的積分調整。我們結合下面的訂單服務、","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"、積分服務的簡化交互圖來看，我們來分析下整個鏈路中可能會出現的消息丟失問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/aca80c6217c1b84c597f500dad584ac2.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"場景1：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在訂單服務向RocketMQ發送訂單成功生成的消息的時候，可能由於網絡抖動的問題導致訂單消息沒能正常投遞到RocketMQ，導致消息丟失。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"場景2：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼假如訂單服務以及RocketMQ之間的網絡沒問題，消息正常被RocketMQ接收到了，那麼會存在消息丟失的情況嗎？答案是肯定的，這和RocketMQ的持久化機制有關係，當消息到達RocketMQ之後，並不是立馬落盤存儲，而是存儲在page cache中的。如果此時出現服務器斷電或者宕機情況，那麼還沒來得及落盤的消息數據就有可能丟失。另外即使是落到磁盤當中，如果出現磁盤壞道的話，依然會出現消息數據的可能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"場景3：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果前面兩種場景都沒問題，積分服務拿到訂單消息了。還會出現消息丟失的問題嗎？答案依然是肯定的。即便是積分服務拿到了訂單消息，當積分服務自動提交消息offset到RocketMQ中，但是此時如果出現宕機或者積分服務掛了，沒有將本該增加的積分進行處理，此時也就出現了消息丟失的情況。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"事務消息機制原理","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"half消息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所謂的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"事務機制，其實是","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"提供了一種","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息的機制。當訂單服務接受到訂單支付信息後，訂單服務會發送","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"中，這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息是不被消費者所見的。怎麼理解這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"信息呢，按照我自己的理解，就是它實現了一半的消息功能，只在生產端可見，在消費端不可見。另外這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"信息相當於一種","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"的可用性探測，如果","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息都發送失敗的話，就不必再進行下游業務的一系列操作了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3a/3ac5f18cde2c6cc16447b7b54853afaa.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果此時用於探測RocketMQ的可用性的half消息發送失敗了，那麼說明此時訂單服務與RocketMQ存在異常，則會對之前訂單進行一系列的回滾操作。如果half消息被成功投遞，則需要進行本地事務操作，更新訂單狀態。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果本地事務執行失敗了怎麼辦，訂單服務可以發送rollback請求，將之前的half消息從RocketMQ中進行刪除，不再進行後續的消息投遞。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"half消息原理分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上文提到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息不被消費端可見，那麼這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息是怎麼實現在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"中不被積分服務所見的呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"訂單服務發送","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息，實際並不是將消息投遞到積分服務訂閱的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"topic","attrs":{}}],"attrs":{}},{"type":"text","text":"，而是將消息投遞到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"中的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RMQ_SYS_TRANS_HALF_TOPIC","attrs":{}}],"attrs":{}},{"type":"text","text":"對應的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"messeageQueue","attrs":{}}],"attrs":{}},{"type":"text","text":"。由於積分服務並沒有訂閱這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Topic","attrs":{}}],"attrs":{}},{"type":"text","text":"，所以這個消息對於積分服務是不可見的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外有個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"OP_TOPIC","attrs":{}}],"attrs":{}},{"type":"text","text":"用於記錄對應","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"commit/rollback","attrs":{}}],"attrs":{}},{"type":"text","text":"狀態。大致的交互如下如所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/12/123c165576508f20e7f29a9e66bbfeb3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果訂單服務","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息發送失敗了，由於網絡原因或者","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"掛了，那麼此時需要執行一些回滾操作，讓訂單進行關閉。因爲訂單信息無法通知到下游服務了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fb21322de4081f672343f3a0d4b66072.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼如果","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息已經寫入","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"中，但是本地事務執行失敗又該怎麼辦呢？也就是說當訂單服務接收到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息寫入成功的響應後，更新訂單信息時發生了異常，無法完成狀態更新。那麼此時訂單服務需要發送","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"rollback","attrs":{}}],"attrs":{}},{"type":"text","text":"的請求給","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"，通知其將原來的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"信息進行刪除。如果本地事務執行成功，則需要發送commit請求給","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"會將原先存在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RMQ_SYS_TRANS_HALF_TOPIC","attrs":{}}],"attrs":{}},{"type":"text","text":"中的消息重新投遞到積分服務訂閱的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"TOPIC","attrs":{}}],"attrs":{}},{"type":"text","text":"中去，這樣積分服務就可以正常消費信息進行下一步的積分操作了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再考慮一種情況，如果訂單服務發送","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"commit","attrs":{}}],"attrs":{}},{"type":"text","text":"或者","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"rollback","attrs":{}}],"attrs":{}},{"type":"text","text":"請求未正常投遞到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"中，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"不知道","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息到底是對應的本地事務到底是執行成功了還是執行失敗了。針對這種情況，訂單服務需要提供狀態回查接口，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"定時檢測是否還有沒有處理的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息，當存在這樣的消息時，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"調用回查接口確認本地事務執行情況。執行失敗的則刪除","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"half","attrs":{}}],"attrs":{}},{"type":"text","text":"消息，執行成功則重新投遞消息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a8/a8fb7fc3ac5a14b34731f1cd6c15b160.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上文的分析，訂單服務和","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"之間的交互，通過事務消息機制可以保證消息可以被可靠投遞。至少在訂單服務和","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RocketMQ","attrs":{}}],"attrs":{}},{"type":"text","text":"之間不會出現消息丟失的問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我是慕楓，感謝各位小夥伴點贊、收藏和評論，文章持續跟新，我們下期再見！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"微信搜索：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"慕楓技術筆記","attrs":{}},{"type":"text","text":"，優質文章持續更新，我們有學習打卡的羣可以拉你進，一起努力衝擊大廠，另外有很多學習以及面試的材料提供給大家。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7f/7ff6f4cf7dc92ed189149727ace4dc43.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}