緩存與存儲的一致性策略：從 CPU 到分佈式系統

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在計算機系統設計實踐中，我們常常會遇到下圖所示架構："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/80/80dbea8b096f714fdb33e4fc8291b7e9.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決單個存儲器讀吞吐無法滿足要求的問題，常常需要在存儲器上面增加一個或多個緩存。但由於相同的數據被複制到一個或多個地方，就容易引發數據一致性問題。不一致的數據可能出現在"},{"type":"text","marks":[{"type":"strong"}],"text":"同級 Cache 之間 (Cache Coherence) "},{"type":"text","text":"和"},{"type":"text","marks":[{"type":"strong"}],"text":"上下級 Cache 之間"},{"type":"text","text":"。解決這些數據一致性問題的方案可以統稱爲 Cache Policies。從本質上看，所有 Cache Policies 的設計目的都可以概括爲："},{"type":"text","marks":[{"type":"strong"}],"text":"在增加一級緩存之後，系統看起來和沒加緩存的行爲一致，但得益於局部性原理，系統的讀吞吐量提高、時延減少"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文將探討三個場景："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Cache Policy In Single-core Processor"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Cache Coherence in Multi-core Processor"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Cache Policy in Cache/DB Architecture"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Cache Policy in Single-core Processor"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在單核 CPU 中，只有一套 Cache，因此只要確保寫入 Cache 中的數據也寫入到 Memory 即可。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5d/5d4363a106323959b64d29e27f0c8fe6.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"補充一些概念定義：數據在 Cache 與 Memory 之間移動的最小單位通常在 32 - 128 字節之間，Memory 中對應的最小單位數據稱爲 Cache Block，Cache 中與單個 Cache Block 對應的存儲空間稱爲 Cache Line，在 Cache 中除了存儲 Block 數據，還需要存儲 Block 對應的唯一標識 $T$ (Tag)，以及一個用於標記 Cache Line 是否有數據的有效位 $V$。完整對應關係如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a9/a929dedb193340ffa826bbb5f220821c.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單核處理器下的 Cache Policy 要解決的問題可以被概括爲："}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU 從 Cache 中讀到的數據必須是最近寫入的數據"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要滿足定義，最簡單的方式就是 Write-Through，即每次寫入 Cache 時，也將數據寫到 Memory 中。當之前寫入的某數據 $D$ 在某時刻被置換後，可以保證再次讀入的數據是最近寫入的數據。這裏有個很明顯的改進空間：只需要在數據 $D$ 被置換前將其寫入 Memory 即可。爲此我們可以爲每個 Cache Line 增加一個髒位 (Dirty Bit)，即："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/27/2783b9bf7fd1ddf78b531b4467a7cccb.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當其被寫入時置爲 1；當其被置換時，如果髒位爲 1，則寫出到 Memory，否則直接丟棄即可。以上所述的 Cache Policy 就是 Write-Back Policy，也是目前在單核處理器中被廣泛採用的 Cache Policy。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Cache Coherence in Multi-core Processor"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"‌在多核 CPU 中，如果這些核共用一套緩存，由於單套 Cache 的吞吐跟不上，無法達到最佳性能。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a8/a86e5f755d109fd6e89fa33bae19d2d0.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時候就需要在每個核上再加一級私有緩存："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bd2725ad3632393aa212a8d0afdc054d.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設在一個 4 核處理器中，內存地址 $MA$ 處最開始存儲着整數 0，這時每個核都需要完成一個 read-modify-write 的操作，如下所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bf/bfd8bd43321f26898918f95682c355d6.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果不加任何協議，當 4 個核都完成相應的操作後，內存地址 $MA$ 處可能存儲着 1、2、3、4 中的任意值，這將影響並行計算的正確性。要保證並行計算的正確性，就必須保證每個核私有緩存之間的"},{"type":"text","marks":[{"type":"strong"}],"text":"數據一致"},{"type":"text","text":"且永遠是"},{"type":"text","marks":[{"type":"strong"}],"text":"最新版本"},{"type":"text","text":"，可以想象，多核處理器上的各核之間必須遵守某種數據讀寫協議，纔可能在獲得多核計算力的同時維持計算的正確性，我們稱這種數據讀寫協議爲 Cache Coherence Protocols。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cache Coherence 的要求："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"從內存地址 $MX$ 將數據 $D$ 讀入到核 $C1$ 的 Cache 中，在其它核沒有寫入數據到 MX 的情況下，讀入的數據 $D$ 必須是 $C1$ 最近寫入的數據值。(單核 CPU 的 Cache Coherence 定義)"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"如果 $C1$ 寫入數據到 $MX$ 中，經過足夠長的一段時間後，在其它核沒有寫入數據的情況下，$C2$ 必須能夠讀入 $C1$ 寫入的數據值。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"針對地址 $MX$ 中的來自於各個核的寫入操作必須被序列化，即在每個核眼中，數據的寫入順序相同。"}]}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"How To Get Coherence"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要在多核 CPU 中實現 Cache Coherence，需要解決的根本問題是："},{"type":"text","marks":[{"type":"strong"}],"text":"讓每個讀操作在執行前能夠獲得所有最近的寫操作歷史"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"從寫操作傳遞信息的內容出發"},{"type":"text","text":"，可以將 Cache Coherence Protocols 劃分爲兩類："},{"type":"text","marks":[{"type":"strong"}],"text":"Write-Update"},{"type":"text","text":" 和 "},{"type":"text","marks":[{"type":"strong"}],"text":"Write-Invalidate"},{"type":"text","text":"。Write-Update 就是在寫入數據時，將所有其它同級 Cache 中相同的 Cache Line 更新成最新數據；Write-Invalidate 就是在寫入數據時，將所有其它同級 Cache 中相同的 Cache Line 標記爲不合法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"從寫操作傳遞信息的方式出發"},{"type":"text","text":"，可以將 Cache Coherence Protocols 劃分爲兩類："},{"type":"text","marks":[{"type":"strong"}],"text":"Snooping"},{"type":"text","text":" 和 "},{"type":"text","marks":[{"type":"strong"}],"text":"Directory"},{"type":"text","text":"。Snooping 將寫數據的信息通過共享總線 (Shared Bus) 廣播給其它同級 Cache，同時保證寫操作的順序一致；Directory 在內存中爲每個 Cache Line 標記額外的元信息，每個 Cache Line 的讀寫控制分而自治，將寫數據的信息通過點對點的方式傳遞。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"任何一種 Cache Coherence Protocol 基本都可以從這兩個維度被歸類爲以下四類："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/045071251cc08241044edb0fe8b0532d.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Write-Update Snooping Example"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cache 中的每條 Cache Line，除了記錄數據本身，額外使用 1 bit 標記 $V$ 是否有效，以及若干 bits 用於存儲 Cache Block 的唯一標識 $T$。多個核內部的 Cache 通過一條共享總線與 Memory 相連，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/78/782460332ca1bd911054bf9096bab955.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"讀取數據時，如果 Cache Line $T$ 在 Cache 中的標記位 $V$ 爲 0，即觸發 Cache Miss，Cache 會向 Memory 發起讀請求；同時其它核的 Cache 會在總線上監聽信息，但它們並不關心讀請求，因此這個過程沒有其它事情發生；如果目標 Block 在 Cache 中的標記位 $V$ 爲 1，則直接返回。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"寫入數據時，Cache 會將寫請求通過總線發送到 Memory 中，並將 Memory Block 中 $T$ 對應 Cache Line 中的數據更新；同時，其它核的 Cache 會在總線上監聽信息，如果發現內部也存有標識符爲 $T$ 的 Memory Block，則將其對應的 Cache Line 更新。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果多個核同時發送針對 Cache Line $T$ 的寫請求，這時只有一個核可以獲得總線的使用權，當整個 Write-Update 完整過程執行完畢後，其它核才能繼續爭奪總線的使用權。這也保證了 Cache Coherence 定義中的第三條。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Optimization #1: Memory Writes"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在原始的 Write-Update Snooping Example 中，我們採用 Write-Through 的方式，每當某個 Cache Line 寫入數據時，都同時寫穿到 Memory 中。本身 Memory 距離較遠，讀寫數據時間長，就容易成爲瓶頸，因此如果能夠儘量使用類似 Write-Back 的策略，將數據保留在 Cache 中，用髒位 (dirty bit) 標記，等到其需要被替換時，再寫入 Memory 中，就能優化該協議的整體性能。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d2cc0de6f61460fe922abcc6e634d9f8.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"讀取數據時，如果 Cache Line $T$ 在 Cache 中的標記爲 $V$ 爲 0，即觸發 Cache Miss，Cache 會向 Memory 發起讀請求；如果其它核的 Cache 已經擁有 $T$ 對應的數據，則會"},{"type":"text","marks":[{"type":"strong"}],"text":"截獲該請求"},{"type":"text","text":"，直接將自己的數據傳輸給請求方，減少讀穿。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"寫入數據時，Cache 會首先將自身 $T$ 對應的數據更新，並且將髒位置爲 1；然後將寫數據的信息傳入共享總線，這時其它核的 Cache 會同時監聽到該消息。如果另一個核的 Cache 內部有相同的 Cache Line $T$，若它的髒位爲 1，則會將 $T$ 更新成爲剛剛監聽到的值，同時將髒位置爲 0；若它的髒位爲 0，則會直接修改數據。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果 Cache 已滿，被迫清出，則通過緩存置換算法選出 Cache Line $T’$。若 $T’$ 的髒位爲 1，則先將數據寫出到緩存。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總而言之：以上修改"},{"type":"text","marks":[{"type":"strong"}],"text":"減少了讀穿和寫穿的頻率"},{"type":"text","text":"，從而提高整體性能。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Optimization #2: Bus Writes"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"儘管增加 Opmization #1能減少讀寫 Memory 的資源消耗，但每次寫數據時，依然要將信息發送到共享總線。大多數情況下，某 Cache Line $T$ 對應的數據只有單個核會訪問，因此如果能夠提前識別其它核的 Cache 是否擁有該數據，避免向總線寫入數據，就可以進一步提高整體性能。正因爲此，我們可以嘗試再加入一個共享標記位 (Shared Bit)，用於標記目標 Cache Line 是否同時存在於其它核的 Cache 中，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d99f4eac9e0ff961e6c32fc665aae5fd.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"讀取數據時，如果發現其它 Cache 已經擁有 $T$ 對應的數據，則二者都將共享標記位置爲 1。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"寫入數據時，如果共享標記位爲 1，則將寫信息發送到共享總線；如果共享標記位爲 0，則直接修改本地 Cache Line 的值即可，並將髒位標記爲 1，無需廣播。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Write-Invalidate Snooping Example"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用 Write-Update Snooping Example + Dirty Bit + Shared Bit 的結構，我們來看 Write-Invalidate Snooping 的工作模式。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"讀取數據時，與 Write-Update Snooping 類似，$V$ 爲 0 時觸發 Cache Miss；$V$ 爲 1 時直接讀取本地緩存。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"寫入數據時，若 Cache Line $T$ 的共享標記位 $S$ 爲 0，則只寫入本地緩存；若共享標記位 $S$ 爲 1，則寫入本地緩存的同時將寫入信息發送到共享總線，其它擁有 $T$ 的 Cache 將有效位 $V$ 置爲 0 即可。由於 Write-Invalidate 不需要更新其它 Cache 中的數據，因此發送到總線中的信息只需包含 Cache Line 的標識符 $T$ 即可。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與 Write-Update Snooping 不同，Write-Invalidatie Snooping 每次寫入數據後，Cache 中 Cache Line $T$ 的共享標記位 $S$ 總是爲 0，只有一個 Cache 中其對應的有效位 $V$ 爲 1，即全局只有一個 Cache 擁有有效數據。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Update V.S. Invalidate Coherence"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Update 與 Invalidate 究竟二者誰更優異？這需要實際運行模式的檢驗，考慮以下 3 種常見場景："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e5/e5425a5d16058972ed183ba8cacc2cde.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"儘管二者看起來各有千秋，"},{"type":"text","marks":[{"type":"strong"}],"text":"在實踐中普遍被採用的還是 Invalidate Coherence"},{"type":"text","text":"。原因在於：在多核 CPU 的運行時中，一個最頻繁的操作就是將一個 Thread 從一個核移動到另一個核上運行。分析一下這種場景："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f3/f3dae1ea654f21317ecfca62b5876da3.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"MSI Coherence"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Write-Invalide Snooping Example 中，我們在每個 Cache Line 上使用了 3 個標記位：有效位 $T$、髒位 $D$ 和共享位 $S$，一共可以表示 8 個狀態。每個 Cache Line 真的需要 8 個狀態嗎？我們發現實際上每個 Cache Line 只需 3 個狀態就足夠實現 Write-Invalide Snooping Protocol："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MODIFIED：修改且獨佔"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SHARED：共享"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"INVALID：無效"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其狀態機如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/89/896b512ac82d31a2093de778707dde89.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"表示 3 個狀態只需要 2 bits，這種更簡單的 Write-Invalid Snooping Protocol 被稱爲 MSI。儘管 MSI 能達到目的，但它在多個場景下仍存在效率問題，因此也有相應的改進版本 MOSI、MOESI 被提出，這裏不再贅述。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Directory Protocols"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於 Snooping 依賴基於共享總線的廣播和監聽，當 CPU 核數大於 8 個以後，共享總線就需要處理更多信號，解決更多衝突，成爲瓶頸。因此"},{"type":"text","marks":[{"type":"strong"}],"text":"拋棄廣播網絡、擁抱點對點網絡通信是獲得擴展性的前提"},{"type":"text","text":"。失去廣播網絡後，如何保證對同一個 Block 的寫入順序在各 CPU 核中保持一致，又重新成爲難題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Directory Protocols 正是爲解決上述問題而被提出。要序列化對同一個 Block 的數據寫入順序，就必須將這些寫入操作集中到一個節點上，但這並未要求對不同 Block 的寫操作集中到一個節點上。於是我們可以"},{"type":"text","marks":[{"type":"strong"}],"text":"將不同 Block 的控制權分散到不同分片中"},{"type":"text","text":"，這裏的分片就是所謂的 Directory，每個 Directory 中包含若干個 Block 的控制信息。每個 Block 在 Directory 中記錄的信息包含兩個部分："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Dirty Bit：是否被修改且未寫回 Memory"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Sharing Vector：哪些 Cache 擁有該 Block Data"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設 CPU 中有 4 個核，每個核擁有私有 Cache，可以爲每個 Block 記錄 5 bits 信息："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/04a4ba205dc9852f343ab638ed11c726.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時整個架構如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f2/f22ed1271afeb3974d7245775e3e5225.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種分片的思想也是解決分佈式系統橫向擴展性的利器，值得深思。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Cache Policy in Cache/DB Architecture"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"‌在 Web APP 開發中，通過引入緩存中間件 (redis/memcache) 來減少數據庫壓力是十分常見的做法，這時服務架構通常如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/75/7501c5b60d7c6340e552a12940d4192a.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何從 Cache 和 DB 讀取、寫入數據就是 Cache/DB Architecture 下的 Cache Policy。與單核 CPU 中的 Cache Policy 不同，由於 Web APP 通常會部署在多個實例上，實踐中幾乎總是有多個進程在並行地增刪改查數據。這時 Web APP 中不同進程寫 Cache、寫 DB 的順序可以用 “一切皆有可能” 來概括。如果要保證二者之間數據的絕對一致，則必須要有分佈式事務的支持，但無論是實現難度，還是分佈式事務下的寫性能下降，都不是開發者所期望的。因此在 Cache/DB Architecture 中，我們對 Cache Policy 的要求可以概括爲："}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終一致性：在寫入 DB 之後，經過足夠長的時間後總能訪問到最近寫入的數據"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Data Inconsistency"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過簡單分析，我們可以找到很多出現數據不一致的場景。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"場景 1：假設寫入數據時，先寫 DB 後寫 Cache：如果寫 DB 成功，寫 Cache 失敗，那麼 Cache 中就會繼續保存着過時的數據。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"場景 2：假設寫入數據時，先寫 DB 後寫 Cache：如果有兩個進程 A、B 同時執行寫數據操作，有可能出現 A 寫 DB、B 寫 DB、B 寫 Cache、A 寫 Cache 的執行順序，那麼 Cache 中就會繼續保存着過時的數據"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"…"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Cache Policies"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Policy 1：Cache Expiry"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要實現 Cache/DB 中數據的最終一致，最簡單的方式莫過於通過在 Cache 中爲緩存數據設置過期時間，在經過這段時間後，會自動再次從數據庫中重新加載數據，這樣就能達到最終一致性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個方案的缺點也很明顯，假如過期時間設置爲 30 分鐘，那麼 Web APP 就需要容忍 30 分鐘的數據不一致，這對很多服務來說幾乎是無法接受的。當然，開發者可以把過期時間設短一些，但設得越短，讀擊穿到 DB 的頻率也就越高，就和 Cache/DB Architecture 的初衷背道而馳。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Policy 2: Cache Aside"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cache Aside 的讀寫邏輯如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/81/81fccab2af6c83004c89977be54175d9.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種方法適用於大多數場景，它通常也是實踐中的標準做法。當然，這種做法也並非完美："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設有兩個進程 A、B：A 寫入 DB，B 讀取數據，A 刪除 Cache 中對應的數據，這時 B 讀到了過時數據"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假設有兩個進程 A、B：B 從 DB 讀取數據到內存，但未寫入 Cache，A 寫入 DB 並刪除 Cache 中對應的數據，B 將內存中的數據寫入 Cache，過時數據會一直存在於 Cache 中直到過期"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A 寫入 DB 後被殺死，過時數據會一直存在於 Cache 中直到過期"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"…"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述做法也可以被稱爲 Write-Invalidate，即寫入 DB 之後將 Cache 中對應的數據置爲失效狀態。"},{"type":"text","marks":[{"type":"strong"}],"text":"爲什麼不使用類似 Write-Update 的做法"},{"type":"text","text":"？這樣還能夠節省一次 DB 與 Cache 之間的網絡 I/O。寫入 DB 後直接寫入 Cache 的做法存在一個致命的場景：A、B 進程同時寫入數據，其執行順序如下："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"A 寫入 DB"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"B 寫入 DB"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"B 寫入 Cache"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"A 寫入 Cache"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好傢伙，這下好了…"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Policy 3: Read Through"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Read Through 的讀寫邏輯如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5e/5ea88ae361b5ee6eacb3d0de917eac3f.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時候服務架構如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1e/1e302bc726b4a6821022f5da873b72e3.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Read Through 的核心問題在於 Cache 需要支持邏輯嵌入，然而一般這種做法會導致運維、部署都不方便。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Policy 4: Write Through"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Write Through 與 Read Through 類似，就是在寫入時由 Cache 層負責寫入 DB 中。這種方案的問題主要包括："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cache 需要支持邏輯嵌入，導致運維、部署不方便"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通常持久性 (Durability) 不在 Cache 的設計目標中，因此在寫入 DB 之前，數據有可能發生丟失"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Poilicy 5: Double Delete"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Double Delete 的讀寫邏輯如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac0d47df614e367fb22e35301bf4c92b.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實它可以被理解成是 Cache Aside 的改進版，通過一段時間後的二次刪除，避免因爲並行問題導致 Cache 中的過時數據覆蓋新寫入數據的情況。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Policy 6：Write Behind"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Write Behind 的讀寫邏輯如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/88/88b1695a51810c6e357e03ccde6a05ec.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時候服務的架構如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ff/ff3a89e7eca15a58bb69c3ac3caa77e0.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種做法可以極大地提高讀寫吞吐量，但缺點也比較明顯："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cache 需要支持邏輯嵌入，導致運維、部署不方便"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用的 MQ 必須是 FIFO 隊列，否則將導致數據寫入 DB 的順序錯誤"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Write Behind 還有一種變體，就是將寫入的順序調換："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/72/724f4f3f9841d048ebd4812331c39679.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時候服務的架構如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/aeffb88ea30f53f836fc901755ad541a.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相較於原版 Write Behind，由於 DB 在複製的過程中已經實現了類似的 MQ，因此只需要開發解析複製日誌的 DB 中間件，僞裝成 Slave 節點，即可實現相應流程。整個架構中無需引入額外的 MQ，減少部署、運維成本。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Connections"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節，我們從連接數的角度觀察一下 Cache/DB Architecture 中不同 Cache Policies 的架構。假設各上游服務與下游服務建立的連接池爲固定大小 N。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮服務會被部署多個副本，在 Cache Expiry、Cache Aside 以及 Double Delete 中，架構中各節點間的連接狀態如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d2b21bdc003435b70a49a47e1218d507.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個服務實例都需要與 DB、Cache 建立 N 個連接，由於其它服務也需要訪問相同的 DB、Cache 集羣，這時候就會出現極高的連接數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Read-Through、Wright-Through 以及 Write-Behind 中，架構中各節點的連接狀態如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d38c2c7e8aede4304916aeeed85d777d.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個服務實例都需要與 Cache 建立 N 個鏈接，Cache 與 MQ、MQ 與 DB 之間都只需要建立 N 個鏈接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Write-Behind 的變體中，解析複製日誌的中間件只需要與數據庫建立 1 個連接即可，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/62/624ea464552e9366566816e3b46cb28a.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Summary"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本小節列舉了多種 Cache Policies，通常最常用的並不是設計最複雜的，具體場景需要具體分析，也許最簡單的做法就能滿足需求。Less code, less bugs : )。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"References‌"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://classroom.udacity.com/courses/ud007","title":null},"content":[{"type":"text","text":"Georgia Tech - HPCA: Lesson 15 & 24"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.sciencedirect.com/topics/engineering/cache-coherence","title":null},"content":[{"type":"text","text":"SienceDirect: Cache Coherence"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Directory-based_cache_coherence","title":null},"content":[{"type":"text","text":"Wikipedia: Directory-based cache coherence"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Directory-based_coherence#Directory_Node","title":null},"content":[{"type":"text","text":"Wikipedia: Directory-based coherence"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Consistency_model","title":null},"content":[{"type":"text","text":"Wikipedia: Consistency Model"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://medium.com/@TechExpertise/cache-coherence-problem-and-approaches-a18cdd48ee0e","title":null},"content":[{"type":"text","text":"Medium: Cache Coherence Problem and Approaches"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://www.cs.utah.edu/~rajeev/cs7820/pres/7968-07.pdf","title":null},"content":[{"type":"text","text":"cs.utah.edu: Directory-Based Cache Coherence"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_506_Spring_2012/8a_cj","title":null},"content":[{"type":"text","text":"CSC/ECE 506 Sprint 2012/8a cj"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://akkadia.org/drepper/cpumemory.pdf","title":null},"content":[{"type":"text","text":"What Every Programmer Should Know About Memory"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/lectures/12_directorycoherence.pdf","title":null},"content":[{"type":"text","text":"CMU: Directory-Based Coherence I"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/lectures/13_directorycoherence2.pdf","title":null},"content":[{"type":"text","text":"CMU: Directory-Based Coherence II"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture06-directory.pdf","title":null},"content":[{"type":"text","text":"CS4 /MSc Parallel Architectures"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://yunpengn.github.io/blog/2019/05/04/consistent-redis-sql/","title":null},"content":[{"type":"text","text":"Consistency between Redis Cache and SQL Database"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://docs.oracle.com/cd/E13924_01/coh.340/e13819/readthrough.htm","title":null},"content":[{"type":"text","text":"Read-Through, Write-Through, Write-Behind Caching and Refresh-Ahead"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.nginx.com/blog/nginx-high-performance-caching/","title":null},"content":[{"type":"text","text":"High-Performance Caching with NGINX and NGINX Plus"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://simongui.github.io/2016/12/02/improving-cache-consistency.html","title":null},"content":[{"type":"text","text":"Improving cache consistentcy"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf","title":null},"content":[{"type":"text","text":"Scaling Memcache at Facebook"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}