乾貨 | 攜程最終一致和強一致性緩存實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、前言       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"攜程金融從成立至今，整體架構經歷了從0到1再到10的變化，其中有多個場景使用了緩存來提升服務質量。從系統層面看，使用緩存的目的無外乎緩解DB壓力（主要是讀壓力），提升服務響應速度。引入緩存，就不可避免地引入了緩存與業務DB數據的一致性問題，而不同的業務場景，對數據一致性的要求也不同。本文將從以下兩個場景介紹我們的一些緩存實踐方案："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終一致性分佈式緩存場景"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強一致性分佈式緩存場景"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注：我們DB用的是MySQL，緩存介質用的是攜程高可用Redis服務，存儲介質的選型及存儲服務的高可用不是本文重點，後文也不再做特別說明。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、最終一致性分佈式緩存場景"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1 場景描述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過幾年演進，攜程金融形成了自頂向下的多層次系統架構，如業務層、平臺層、基礎服務層等，其中用戶信息、產品信息、訂單信息等基礎數據由基礎平臺等底層系統產生，服務於所有的金融系統，對這部分基礎數據我們引入了統一的緩存服務（系統名utag），緩存數據有三大特點：全量、準實時、永久有效，在數據實時性要求不高的場景下，業務系統可直接調用統一的緩存查詢接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的典型使用場景有：風控流程、APP入口信息提示等，而對數據一致性要求高的場景依然需要走實時的業務接口查詢。引入緩存前後系統架構對比如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ca\/ca2c21cb9e1eb8fe32704aefeb91ce5d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統一緩存服務的構建給部門的整體系統架構帶來了一些優勢："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對業務系統："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"響應速度提升：相比直接調用底層高流量的基礎服務，調用緩存服務接口的系統響應時間大大減少（緩存查詢接口P98爲10毫秒）。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統一接口，降低接入成本：一部分業務場景下可以直接調用統一緩存服務查詢接口，而不用再對接底層的多個子系統，極大地降低了各個業務線的接入成本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統一緩存，省去各個服務單獨維護緩存的成本。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對基礎服務："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務壓力降低：基礎平臺的系統本身就屬於高流量系統，可減少一大部分的查詢流量，降低服務壓力。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體而言，緩存服務處於中間層，數據的寫入方和數據查詢方解耦，數據甚至可以在底層系統不感知的情況下寫入（見下文），而數據使用方的查詢也可在底層服務不可用或“堵塞”時候仍然保持可用（前提是緩存服務是可用的，而緩存服務的處理邏輯簡單、統一且有多種手段保證，其可用性比單個子系統都高），整體上服務的穩定性得到了提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在構建此統一緩存服務時候，有三個關鍵目標："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據準確性：DB中單條數據的更新一定要準確同步到緩存服務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據完整性：將對應DB表的全量數據進行緩存且永久有效，從而可以替代對應的DB查詢。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統可用性：我們多個產品線的多個核心服務都已經接入，utag的高可用性顯的尤爲關鍵。接下來先說明統一緩存服務的整體方案，再逐一介紹此三個關鍵特性的設計實現方案。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.2 整體方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的系統在多地都有部署，故緩存服務也做了相應的異地多機房部署，一來可以讓不同地區的服務調用本地區服務，無需跨越網絡專線，二來也可以作爲一種災備方案，增加可用性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於緩存的寫入，由於緩存服務是獨立部署的，因此需要感知業務DB數據變更然後觸發緩存的更新，本着“可以多次更新，但不能漏更新”的原則，我們設計了多種數據更新觸發源：定時任務掃描，業務系統MQ、binlog變更MQ，相互之間作爲互補來保證數據不會漏更新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外爲了緩存更新流程的統一和與觸發源的解耦，我們使用MQ來驅動多地多機房的緩存更新，在不同的觸發源觸發後，會查詢最新的DB數據，然後發出一個緩存更新的MQ消息，不同地區機房的緩存系統同時監聽該主題並各自進行緩存的更新。對於MQ我們使用攜程開源消息中間件QMQ 和 Kafka，在公司內部QMQ和Kafka也做了異地機房的互通。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於緩存的讀取，utag系統提供dubbo協議的緩存查詢接口，業務系統可調用本地區的接口，省去了網絡專線的耗時（50ms延遲）。在utag內部查詢redis數據，並反序列化爲對應的業務model，再通過接口返回給業務方。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了描述方便，以下異地多機房部署統一使用AB兩地部署的概念進行說明。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體框架如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/20\/2023723f8f214985eaea195aacfcb186.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來介紹一下幾個關鍵點的設計 。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.3  數據準確性設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同的觸發源，對緩存更新過程是一樣的，整個更新步驟可抽象爲4步："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step1：觸發更新，查詢DB中的新數據，併發送統一的MQ"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step2：接收MQ，查詢緩存中的老數據"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step3：新老數據對比，判斷是否需要更新"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step4：若需要，則更新緩存"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於我們業務的大部分核心系統和所有的DB都在A地機房，所以觸發源（如binlog的消費、業務MQ的接收、掃表任務的執行）都在A側，觸發更新後，第一步查詢DB數據也只能在A側查詢（避免跨網絡專線的數據庫連接，影響性能）。查詢到新數據後，發送更新緩存的MQ，兩地機房的utag服務進行消費，之後進行統一的緩存更新流程。總體的緩存更新方案如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f0\/f0a1cb65833826b3dc8f7b1b1356b896.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於有多個觸發源，不同的觸發源之間可能會對同一條數據的緩存更新請求出現併發，此外可能出現同一條數據在極短時間內（如1秒內）更新多次，無法區分數據更新順序，因此需要做兩方面的操作來確保數據更新的準確性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）併發控制  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若一條DB數據出現了多次更新，且剛好被不同的觸發源觸發，更新緩存時候若未加控制，可能出現數據更新錯亂，如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a4\/a4b5772680d254213d7be8681843cb79.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      故需要將第2、3、4步加鎖，使得緩存刷新操作全部串行化。由於utag本身就依賴了redis，此處我們的分佈式鎖就基於redis實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）基於updateTime的更新順序控制   "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即使加了鎖，也需要進一步判斷當前db數據與緩存數據的新老，因爲到達緩存更新流程的順序並不代表數據的真正更新順序。我們通過對比新老數據的更新時間來實現數據更新順序的控制。若新數據的更新時間大於老數據的更新時間，則認爲當前數據可以直接寫入緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們系統從建立之初就有自己的MySQL規範，每張表都必須有update_time字段，且設置爲ON UPDATE CURRENT_TIMESTAMP，但是並沒有約束時間字段的精度，大部分都是秒級別的，因此在同一秒內的多次更新操作就無法識別出數據的新老。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對同一秒數據的更新策略我們採用的方案是：先進行數據對比，若當前數據與緩存數據不相等，則直接更新，並且發送一條延遲消息，延遲1秒後再次觸發更新流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子：假設同一秒內同一條數據出現了兩次更新，value=1和value=2，期望最終緩存中的數據是value=2。若這兩次更新後的數據被先後觸發，分兩種情況："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"case1：若value=1先更新，value=2後更新，（兩者都可更新到緩存中，因爲雖然是同一秒，但是值不相等）則緩存中最終數據爲value=2。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"case2：若value=2先更新，value=1後更新，則第一輪更新後緩存數據爲value=1，不是期望數據，之後對比發現是同一秒數據後會通過消息觸發二次更新，重新查詢DB數據爲value=2，可以更新到緩存中。如下圖所示："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/db\/dbe9eb69df5be699a3c9bdb58c0f1ed3.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過以上方案我們可以確保緩存數據的準確性。有幾個點需要額外說明：  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼要用延遲消息？    "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實不用延遲消息也是可以的，畢竟DB數據的更新時間是不變的，但是考慮到出現同一秒更新的可能是高頻更新場景，若直接發消息，然後立即消費並觸發二次更新，可能依然查到同一秒內更新的其他數據，爲減少此種情況下的多次循環更新，延遲幾秒再刷新可作爲一種優化策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不支持db的刪除操作      "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲刪除操作和update操作無法進行數據對比，無法確定操作的先後順序，進而可能導致更新錯亂。而在數據異常寶貴的時代，一般的業務系統中也沒有物理刪除的邏輯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若當前db沒有設置更新時間該如何處理？    "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以將查DB、查緩存、數據對比、更新緩存這四個步驟全部放到鎖的範圍內，這樣就不需要處理同一秒的順序問題。因爲在這個串行化操作中每次都從DB中查詢到了最新的數據，可以直接更新，而時間的判斷、值的判斷可以作爲優化操作，減少緩存的更新次數，也可以減少鎖定的時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而我們爲何不採用該方案？因爲查詢DB的操作我們只能在一側機房處理，無法讓AB兩地系統的更新流程統一，也就降低了二者互備的可能性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其他方案"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DB數據模型上增加版本字段，可嚴格控制數據的更新時序。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將update_time 字段精度設置爲精確到毫秒或微秒，提升數據對比的準確度，但是相比增加版本字段的方案，依然存在同一時間有多次更新的可能。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些額外的方案，都需要業務數據模型做對應的支持。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.4  數據完整性設計    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述數據準確性是從單條數據更新角度的設計，而我們構建緩存服務的目的是替代對應DB表的查詢，因此需要緩存對應DB表的全量數據，而數據的完整性從以下三個方面得到保證："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）“把雞蛋放到多個籃子裏”，使用多種觸發源（定時任務，業務MQ，binglog MQ）來最大限度降低單條數據更新缺失的可能性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單一觸發源有可能出現問題，比如消息類的觸發依賴業務系統、中間件canel、中間件QMQ和Kafka，掃表任務依賴分佈式調度平臺、MySQL等。中間任何一環都可能出現問題，而這些中間服務同時出概率的可能相對來說就極小了，相互之間可以作爲互補。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）全量數據刷新任務：全表掃描定時任務，每週執行一次來進行兜底，確保緩存數據的全量準確同步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（3）數據校驗任務：監控Redis和DB數據是否同步並進行補償。   "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.5  系統可用性設計     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統一緩存服務被多個業務線的核心繫統所依賴，所以緩存服務的高可用是至關重要的。而對高可用的建設，除了集羣部署、容量規劃、熔斷降級等常用手段外，針對我們自己的場景也做了一些方案。主要有以下三點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）異地機房互備    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上所述，我們的服務在AB兩地部署，兩機房的緩存通過兩地互通的MQ同時寫入。在這套機制下，本地區的業務系統可以直接讀取本地區的緩存，如果出現了本地區utag應用異常或redis服務異常，則可以快速降級到調用另外機房的服務接口。具體方案如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1c\/1c03ad05bbd795e28bccf39e2452ce6a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本地業務系統通過dubbo調用本地的utag服務，在utag的本地處理流程中，查詢本地緩存前後分別可根據一定的條件進行服務降級，即查詢另一機房。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢本地緩存前降級：若本地redis集羣出現故障，可以在配置平臺人工快速切換到查詢另一側的服務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢本地緩存後降級：本地處理結束，若出現特定錯誤碼（\"OPERATE_REDIS_ERROR\"）則可降級到查詢另一側服務。該功能也需要手工配置開關來啓用。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了避免循環調用，在降級調用前，需要判斷當前請求是否來自本地，而此功能通過Dubbo的RpcContext透傳特定標識來實現。除此之外，還建立了兩機房的應用心跳，來輔助切換。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/19\/195ae55192784a556155887a768c087e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）QMQ和Kafka互備"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存更新流程通過MQ來驅動，雖然公司的MQ中間件服務由專人維護，但是萬一出現問題長時間不能恢復，對我們來說將是致命的。所以我們決定同時採用Kafka和QMQ兩種中間件來作爲互備方案。默認情況下對於全表掃描任務和binlog消費這類大批量消息場景使用Kafka來驅動，而其他場景通過QMQ來驅動。所有的場景都可以通過開關來控制走Kafka或者QMQ。目前該功能可通過配置管理平臺來實現快速切換。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（3）快速恢復"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在極端情況下，可能出現Redis數據丟失的情況，如主機房（A機房）突然斷網，redis集羣切換過程出現數據丟失或同步錯亂，此時很可能無法通過自動觸發來補齊數據，因此設計了全錶快速掃描的補償機制，通過多任務並行調度，可在30分鐘內將全量數據完成刷新。此功能需要人工判斷並觸發。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.6 總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上介紹了我們最終一致性分佈式緩存服務的設計思路和要點，其中的關鍵點爲數據準確性、數據完整性、系統可用性的設計。除此之外，還有一些優化點如降級方案的自動觸發、異地機房緩存之間、緩存與DB之間做旁路數據diff，可進一步確保緩存服務整體的健壯性，在後續的版本中進行迭代。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、強一致性分佈式緩存場景"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 場景描述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 強一致性分佈式緩存目前主要應用在我們攜程金融的消金貸前服務中。隨着我們用戶量和業務量的增漲，貸前服務的查詢量激增，給數據庫帶來了很大的壓力，解決此問題有幾種可選方案："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）分庫分表：成本和複雜度相對較高，我們場景下只是數據查詢流量較大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）讀寫分離：出於數據庫性能考慮，我們的MySQL大部分採用異步複製的方式，而由於我們的場景對數據實時性要求較高，因此無法直接利用讀寫分離的優勢來分擔主庫壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合來看，增加緩存是更加合適的方案，我們決定設計一套高可用的滿足強一致性要求的分佈式緩存。接下來介紹我們的具體設計實現方案。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 整體方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存的處理我們採用了較爲常見的處理思路：在更新操作中，先更新數據庫，再刪除緩存，查詢操作中，觸發緩存更新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/81\/81b73236d8e2eae2ca36754b32e8ce8b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在此過程中，若不加控制，則會存在數據不一致性問題，主要是由於緩存操作和DB更新之間的併發導致的。具體分析如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）緩存讀取和DB更新併發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示，查詢時候若緩存已經存在，則會直接返回緩存數據。若查詢緩存的操作，發生在“更新DB數據”和“刪除緩存”之間，則本次查詢到數據爲緩存中的老數據，導致不一致。當然下次查詢可能就會查詢到最新的數據。這種併發在我們服務中是存在的，比如某個產品開通後，會在更新DB（產品開通狀態）後立即發送MQ（事務型消息）告知業務，業務側處理流程中會立即發起查詢操作。此場景中數據庫的更新和數據的查詢間隔極短，很容易出現此種併發問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1b\/1bafb39d97b12d0d4343f56fc5653221.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）緩存更新與DB更新併發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示，查詢的時候，若緩存不存在，則更新緩存，流程是先查詢DB再更新Redis。若更新緩存時候，出現以下時序：查詢DB老數據（T0時刻，DB中value=1）→ 更新DB（T1時刻，更新DB爲value=2）→ 刪除Redis（T2）→ 更新Redis（T3），則會導致本次查詢返回數據及緩存中的數據與DB數據不一致，即接口返回和更新後的緩存都爲髒數據。若T2和T3互換，即更新DB後，先更新Redis，再刪除Redis ，由於緩存被刪除在下次更新可能會被正確更新，但本次返回數據依然與DB更新後的數據不一致。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0e2ddd03239a3f8c366635b7349371d3.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上分析，爲了避免併發帶來的緩存不一致問題，需要將\"更新DB\"+\"刪除緩存\"、\"查詢DB\"+\"更新緩存\"兩個流程都進行加鎖。此處需要加的是分佈式鎖，我們使用的是redis分佈式鎖實現。加鎖後的讀寫整體流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/15\/154a135d119be6c91f3bdfdc661e0400.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，有兩處加鎖，更新DB時加鎖，鎖範圍爲\"更新DB\"+\"刪除cache\"（圖中lock1），更新緩存時加鎖，鎖範圍爲\"查詢DB\" + \"更新cache\"（圖中lock2），兩處對應的鎖key是相同的。基於此方案，對於上面所說的兩種併發場景，做針對性分析如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）緩存查詢和DB更新的併發控制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 查詢操作流程中，先判斷lock是否存在，若存在，則表示當前DB或緩存正在更新，不能直接查詢緩存，在查詢DB後返回數據。之所以這麼做，還是由場景決定的，如前文所述，我們場景下的基本處理思路是，緩存僅作爲“DB降壓”的輔助手段，在不確定緩存數據是否最新的情況下，寧可多查詢幾次DB，也不要查詢到緩存中的不一致數據。此外，更新操作相對於查詢操作是很少的，在我們貸前服務中，讀寫比例約爲8:1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此處另外的一個可行方案是可在檢測到有鎖後可進行短暫的等待和重試，好處是可進一步增加緩存的命中率，但是多一次鎖等待，可能會影響到查詢接口的性能。可根據自身場景進行抉擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，爲了進行降級，在鎖判斷前也增加了降級開關判斷，若降級開關開啓，也會直接查詢DB。而降級主要是由於redis故障引起的，下文詳述。若檢測是否有鎖時發生了異常同樣也會直接查詢DB。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）緩存更新和DB更新的併發控制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢操作流程中，若緩存不存在，則進行緩存的更新，在更新時候先嚐試進行加鎖，若當前有鎖說明當前有DB或緩存正在更新，則進行等待和重試，從而可避免查詢到DB中的老數據更新到緩存中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中lock2的流程（load cache），我們是同步進行的。另外一個可行的方案是，異步發起緩存的加載，可減少鎖等待時間，但是若出現瞬時的高併發查詢，可能緩存無法及時加載產生從而頻繁產生瞬時壓力。可根據自身場景進行抉擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上爲我們的整體設計思路，接下來從實現的角度分別描述一下基於本地消息表的緩存刪除策略，緩存的降級和恢復這兩個方面的具體方案。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 緩存刪除策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在更新操作中，在鎖的範圍內，先更新DB，再刪除緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中鎖的選型，我們採用與緩存同介質的redis分佈式鎖，這樣做的好處是若因爲redis服務不可用導致的鎖處理失敗，對於緩存本身也就不可用，可以自動走降級方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，更新流程還要考慮兩點：鎖的範圍和刪除緩存失敗後如何補償。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）鎖粒度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新操作中加鎖粒度有以下三種方案："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方案一：事務提交後加鎖，只鎖定刪除緩存操作。對原事務無任何額外影響，但是在事務提交後到刪除緩存之間存在與查詢的併發可能性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方案二：在事務提交前加鎖，刪除緩存後解鎖。在滿足一致性要求的前提下，鎖的粒度可以做到最小，但是增加了DB事務的範圍，若redis出現超時則可能導致事務時間拉長，進而影響DB操作性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方案三：在事務開始前加鎖，刪除緩存後解鎖。鎖的範圍較大，但是能滿足我們一致性要求，對單個DB事務也基本無影響。且對同一個用戶來說，貸前數據的更新並不頻繁，鎖範圍稍大一些是我們可以接受的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a3\/a355795b47cfcc8fe5f482ab5ae2bb2b.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"立足自身場景，權衡一致性要求和服務性能要求，我們剔除了方案二，默認情況下使用方案三，但是若在事務開始前加鎖失敗，爲了不影響原業務流程（緩存只是輔助方案，redis故障不影響原應用功能）會自動降級到方案一，即在事務提交後刪除緩存前再加鎖。而這種降級，若出現併發的查詢操作，依然可能出現上述不一致的問題，但是是可以容忍的，原因如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通常情況下加鎖失敗是由於操作redis異常或者鎖競爭引起的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若出現redis異常，同時出現了併發的查詢，而併發的兩個操作時間間隔是極短的，因此查詢時候，鎖檢測操作通常也是異常的，此時查詢會自動降級爲查詢DB。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若極短時間內的redis集羣抖動，事務執行前redis不可用，事務執行後redis恢復，而此時在加鎖操作還沒有完成前恰巧又進行了併發的查詢操作，檢測鎖成功且鎖不存在，纔可能會出現查詢出老數據的情況。這種是極其嚴苛的併發條件。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而加鎖過程會進行重試（可動態調整配置），多次重試後可解決大部分的鎖競爭情況。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜上，在上述鎖降級的方案下，數據不一致出現的情況雖然無法完全避免，但是產生條件極其苛刻，而應對這種極其極端的情況，在系統層面做更加強的方案帶來的複雜度提升與收益是不成正比的，一般情況下做好日誌記錄、監控、報警，通過人工介入來彌補即可。從該方案上線後至今兩年多的時間內，沒有出現過該情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）刪除緩存失敗的補償"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外要考慮的問題是，如果更新DB成功但刪除緩存失敗要後如何處理，而此種情況往往因應用服務器故障、網絡故障、redis故障等原因導致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若應用服務器突然故障，則服務整體不可用，跟緩存就沒多大關係了。若是由於網絡、redis故障等原因導致的刪除緩存失敗，此時查詢緩存也不可用，查詢走DB，但需要可靠地記錄下哪些數據做了變更，待redis可用後需要進行恢復，需要將中間變更的記錄對應的緩存全部刪除。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此處的一個關鍵點在於數據變更的可靠性記錄，受到QMQ事務消息實現方案的啓發，我們的方案是構建一張簡易的記錄表（代表發生變更的DB數據），每次DB變更後，將該變更記錄表的插入和業務DB操作放在一個事務中處理。事務提交後，對應的變更記錄持久化，之後進行刪除緩存，若緩存刪除成功，則將對應的記錄表數據也刪除掉。若緩存刪除失敗，則可根據記錄表的數據進行補償刪除，而在redis的恢復流程中，需要校驗記錄表中是否存在數據，若存在則表示有變更後的數據對應的緩存未清除，不可進行緩存讀取的恢復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外刪除操作還要進行異步重試，來避免偶爾超時引起的緩存刪除失敗。此方案整體流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f0\/f0e58b1354e379e7368c6cd9ff63fdff.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中cache_key_queue表即爲我們的變更記錄表，放在業務的同DB內。其表結構非常簡單，只有插入和刪除操作，對業務DB的額外影響可以忽略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nCREATE TABLE `cache_key_queue` (\n `id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT '主鍵',\n `cache_key` varchar(1024) NOT NULL DEFAULT '' COMMENT '待刪除的緩存key',\n `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '創建時間',\n PRIMARY KEY (`id`)\n) ENGINE = InnoDB AUTO_INCREMENT = 0 CHARSET = utf8 COMMENT '緩存刪除隊列表'"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上分析，爲了鎖範圍儘可能小，且爲了儘可能降低極端的redis抖動情況下產生的影響，我們期望可以在事務提交後立即觸發緩存的刪除操作。爲了能夠對redis不可用期間發生變更的數據進行清除，我們需要可靠地記錄數據變更記錄。幸運的是，基於Spring的事務同步機制 TransactionSynchronization，可以很容易實現該方案。簡單來說，該機制提供了Spring環境中事務執行前後的AOP功能，可以在spring事務的執行前後添加自己的操作，如下所示（代碼和註釋經過了簡化）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\npublic interface TransactionSynchronization extends Flushable {\n \/**\n * 事務同步器掛起處理\n *\/\n void suspend();\n\n \/**\n * 事務同步器恢復處理\n *\/\n void resume();\n\n \/**\n * 事務提交前處理\n *\/\n void beforeCommit(boolean readOnly);\n\n \/**\n * 事務完成(提交或回滾)前處理\n *\/\n void beforeCompletion();\n\n \/**\n * 事務提交後處理\n *\/\n void afterCommit();\n\n \/**\n * 事務完成(提交或回滾)後處理\n *\/\n void afterCompletion(int status);\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於此機制，我們可以很方便且相對優雅地實現我們的設計思路，即在 beforeCommit方法中，插入cache_key_queue記錄；在 afterCommit方法中同步刪除緩存，若刪除緩存成功則同步刪除cache_key_queue表記錄；在afterCompletion方法中進行鎖的釋放處理。若同步刪除緩存失敗，則cache_key_queue表記錄也會保留下來，之後觸發異步刪除，並啓動定時任務不斷掃描cache_key_queue表進行緩存刪除的補償。需要注意的是可能存在嵌套事務，一個完整事務中，可能存在多次數據更新，可藉助ThreadLocal進行多條更新記錄的彙總。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4 緩存的熔斷和恢復"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上述鎖處理流程中討論的redis抖動問題外，還需要考慮緩存服務redis集羣不可用（網絡問題、redis集羣問題等）。按照我們的基本原則，引入的緩存服務僅做輔助，並不能強依賴。如果緩存不可用，主業務依然要保持可用，這就是我們接下來要討論的緩存的熔斷和恢復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）緩存熔斷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熔斷的目的是在redis不可用時避免每次調用（查詢或更新）都進行額外的緩存操作，這些緩存操作會進行多次嘗試，比如加鎖操作我們設置的自動重試3次，每次間隔50ms，總耗時會增加150ms。若redis不可用則每次調用的耗時都會有額外增加，這對主業務功能可能會產生影響，降低底層服務的質量和性能。因此我們需要識別出 redis不可用的情況，並進行熔斷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的熔斷判斷邏輯爲：每個redis操作都try-catch異常，並做計數統計（不區分讀寫操作），若在M秒內出現N次異常則開啓熔斷。我們的場景下設置爲10秒內出現50次異常就熔斷，可根據自身場景設置，需要注意的是如果redis請求次數比較少，則需要在配置上保證在M秒內至少出現N次請求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外熔斷開關的配置是放在應用服務器的內存中，即單機熔斷，而非集羣熔斷，這樣做的原因是，redis服務不可用有可能是單機與redis服務的連通性問題導致，而在其他機器上依然可以訪問緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）緩存恢復"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熔斷之後的恢復策略相對複雜一些，需要區分緩存的讀操作恢復和寫操作恢復。具體如下流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/91\/919b961710f96b7bf8aa702775d39e10.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step1：校驗redis是否可用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"       "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"判斷邏輯爲，連續發起特定的set操作N次，每次間隔一定時間，若都成功，則認爲redis恢復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此處需要注意的是，我們的redis集羣是Cluster模式，不同的key會散落在不同的redis主節點上，因此最保險的做法是判斷當前集羣中所有的主節點都恢復才認爲操作恢復，而簡單的做法是每次探測恢復的set操作都設置不同的key以求能儘可能散列到不同的節點去。可按照自身場景進行方案抉擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step2：恢復緩存寫操作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若redis恢復，緩存的寫操作就可以恢復了。即可在更新操作中進行加鎖、更新DB、刪除緩存。但是此時讀操作還不能立即恢復，因爲redis不可用期間發生了DB變更但是緩存並沒有變更，依然爲老數據，因此需要將這部分老數據剔除後才能恢復讀操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"step3：校驗擠壓的cache_key_queue記錄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輪訓查看cache_key_queue表中是否有記錄存在，若存在記錄則認爲當前有不一致的緩存數據，需要等待定時任務將暫存的key表記錄對應的緩存全部刪除（同時也會刪除cache_key_queue表記錄）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" step4：恢復緩存讀操作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若當前不存在cache_key_queue記錄則可恢復讀操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上闡述了redis緩存的自動熔斷和恢復方案。需要明確的是，能夠進行熔斷是有前提條件的，即應用完全去掉緩存，DB還是可以抗住一段時間壓力的，否則一旦出現緩存服務故障，流量全部走到應用，超過了應用和DB的承受能力，將服務壓垮，後果更加嚴重。所以不能強依賴熔斷機制，不能強依賴緩存，而這就需要接口限流等其他手段來從整體上保證服務的高可用。此外可進行定期壓測，來錨定服務性能上限，進而不斷優化對各種策略和資源的配置。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.5 總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上描述了我們強一致性緩存方案的設計思路及一些實現細節。基於該方案，我們核心數據庫的QPS降低了80%，緩存的命中率達到92%。而該方案的關鍵是通過加鎖來控制讀寫，從表面上看會犧牲一些性能，但是實際上高緩存命中率同樣彌補了此缺陷，緩存的建立使得我們服務查詢接口AVG響應時間降低了10%左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/38\/85\/38de7de311d49464187dc7cc1ee9bb85.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、結語"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 以上分別描述了我們的最終一致性和強一致性緩存設計和實現思路。兩套緩存方案側重點各有不同："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終一致性場景的基本思路是：讀緩存優先，數據可以容忍暫時不一致，因此重點在及時補償。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強一致性場景的基本思路是：讀DB優先，緩存僅作爲“DB降壓”的輔助手段，在不確定緩存數據是否最新的情況下，寧可多查詢幾次DB，也不要查詢到緩存中的不一致數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，我們的最終一致性緩存方案是獨立的緩存服務，而該強一致緩存方案，是需要嵌入到應用系統中去使用的。方案的選擇需要立足於自身場景，希望我們的分享能夠給大家帶來一些啓發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GSF，攜程高級技術專家，關注高可用、高併發系統建設，致力於用技術驅動業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：攜程技術（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/E-chAZyHtaZOdA19mW59-Q","title":"xxx","type":null},"content":[{"type":"text","text":"乾貨 | 攜程最終一致和強一致性緩存實踐"}]}]}]}