實現多級緩存架構設計方案

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC，即“透明多級緩存（Transparent Multilevel Cache）”，是有贊 PaaS 團隊給公司內應用提供的整體緩存解決方案。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 在通用“分佈式緩存解決方案（如 CodisProxy + Redis，如有贊自研分佈式緩存系統 zanKV）”基礎上，增加了以下功能：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用層熱點探測","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用層本地緩存","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用層緩存命中統計","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以幫助應用層解決緩存使用過程中出現的熱點訪問問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"爲什麼要做 TMC","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用有贊服務的電商商家數量和類型很多，商家會不定期做一些“商品秒殺”、“商品推廣”活動，導致“營銷活動”、“商品詳情”、“交易下單”等鏈路應用出現 緩存熱點訪問 的情況：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"活動時間、活動類型、活動商品之類的信息不可預期，導致 緩存熱點訪問 情況不可提前預知；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存熱點訪問 出現期間，應用層少數 熱點訪問 key 產生大量緩存訪問請求：衝擊分佈式緩存系統，大量佔據內網帶寬，最終影響應用層系統穩定性；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了應對以上問題，需要一個能夠 自動發現熱點 並 將熱點緩存訪問請求前置在應用層本地緩存的解決方案，這就是 TMC 產生的原因。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"多級緩存解決方案的痛點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上述描述，我們總結了下列 多級緩存解決方案 需要解決的需求痛點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱點探測：如何快速且準確的發現 熱點訪問 key ？","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據一致性：前置在應用層的本地緩存，如何保障與分佈式緩存系統的數據一致性？","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"效果驗證：如何讓應用層查看本地緩存命中率、熱點 key 等數據，驗證多級緩存效果？","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"透明接入：整體解決方案如何減少對應用系統的入侵，做到快速平滑接入？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 聚焦上述痛點，設計並實現了整體解決方案。以支持“熱點探測”和“本地緩存”，減少熱點訪問時對下游分佈式緩存服務的衝擊，避免影響應用服務的性能及穩定性。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"TMC 整體架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d20293ed4715c4d2b628ff9d03083d1f.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 整體架構如上圖，共分爲三層：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲層：提供基礎的 kv 數據存儲能力，針對不同的業務場景選用不同的存儲服務（codis/zankv/aerospike）；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代理層：爲應用層提供統一的緩存使用入口及通信協議，承擔分佈式數據水平切分後的路由功能轉發工作；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用層：提供統一客戶端給應用服務使用，內置“熱點探測”、“本地緩存”等功能，對業務透明；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本篇聚焦在應用層客戶端的“熱點探測”、“本地緩存”功能。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"TMC 本地緩存","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何透明","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 是如何減少對業務應用系統的入侵，做到透明接入的？對於公司 Java 應用服務，在緩存客戶端使用方式上分爲兩類：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"spring.data.redis","attrs":{}}],"attrs":{}},{"type":"text","text":"包，使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RedisTemplate","attrs":{}}],"attrs":{}},{"type":"text","text":"編寫業務代碼；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"youzan.framework.redis","attrs":{}}],"attrs":{}},{"type":"text","text":"包，使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RedisClient","attrs":{}}],"attrs":{}},{"type":"text","text":"編寫業務代碼；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不論使用以上那種方式，最終通過 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"JedisPool","attrs":{}}],"attrs":{}},{"type":"text","text":"創建的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Jedis","attrs":{}}],"attrs":{}},{"type":"text","text":"對象與緩存服務端代理層做請求交互。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ed/ed23084828f94008fba7db0a9a5f2e40.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 對原生 jedis 包的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"JedisPool","attrs":{}}],"attrs":{}},{"type":"text","text":"和 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Jedis","attrs":{}}],"attrs":{}},{"type":"text","text":"類做了改造，在 JedisPool 初始化過程中集成 TMC“熱點發現”+“本地緩存”功能 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Hermes-SDK","attrs":{}}],"attrs":{}},{"type":"text","text":"包的初始化邏輯，使 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Jedis","attrs":{}}],"attrs":{}},{"type":"text","text":"客戶端與緩存服務端代理層交互時先與 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Hermes-SDK","attrs":{}}],"attrs":{}},{"type":"text","text":"交互，從而完成 “熱點探測”+“本地緩存”功能的透明接入。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 Java 應用服務，只需使用特定版本的 jedis-jar 包，無需修改代碼，即可接入 TMC 使用“熱點發現”+“本地緩存”功能，做到了對應用系統的最小入侵。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"整體結構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4ad6dfe4e796eb0abf72ef57adfebe05.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模塊劃分","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 本地緩存整體結構分爲如下模塊：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Jedis-Client ：Java 應用與緩存服務端交互的直接入口，接口定義與原生 Jedis-Client 無異；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes-SDK ：自研“熱點發現+本地緩存”功能的 SDK 封裝，Jedis-Client 通過與它交互來集成相應能力；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 ：接收 Hermes-SDK 上報的緩存訪問數據，進行熱點探測，將熱點 key 推送給 Hermes-SDK 做本地緩存；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存集羣 ：由代理層和存儲層組成，爲應用客戶端提供統一的分佈式緩存服務入口；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎組件 ：etcd 集羣、Apollo 配置中心，爲 TMC 提供“集羣推送”和“統一配置”能力；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"基本流程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）key 值獲取","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Java 應用調用 Jedis-Client 接口獲取 key 的緩存值時，Jedis-Client 會詢問 Hermes-SDK 該 key 當前是否是 熱點key ；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 熱點key ，直接從 Hermes-SDK 的 熱點模塊 獲取熱點 key 在本地緩存的 value 值，不去訪問 緩存集羣 ，從而將訪問請求前置在應用層；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於非 熱點key ，Hermes-SDK 會通過 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Callable","attrs":{}}],"attrs":{}},{"type":"text","text":"回調 Jedis-Client 的原生接口，從 緩存集羣 拿到 value 值；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 Jedis-Client 的每次 key 值訪問請求，Hermes-SDK 都會通過其 通信模塊 將key 訪問事件 異步上報給 Hermes 服務端集羣 ，以便其根據上報數據進行“熱點探測”；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）key 值過期","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Java 應用調用 Jedis-Client 的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"set()","attrs":{}}],"attrs":{}},{"type":"text","text":" ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"del()","attrs":{}}],"attrs":{}},{"type":"text","text":" ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"expire()","attrs":{}}],"attrs":{}},{"type":"text","text":"接口時會導致對應 key 值失效，Jedis-Client 會同步調用 Hermes-SDK 的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"invalid()","attrs":{}}],"attrs":{}},{"type":"text","text":"方法告知其“key 值失效”事件；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 熱點 key ，Hermes-SDK 的 熱點模塊 會先將 key 在本地緩存的 value 值失效，以達到本地數據強一致 。同時 通信模塊 會異步將“key 值失效”事件通過 etcd 集羣 推送給 Java 應用集羣中其他 Hermes-SDK 節點；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其他 Hermes-SDK 節點的 通信模塊 收到 “key 值失效”事件後，會調用 熱點模塊 將 key 在本地緩存的 value 值失效，以達到集羣數據最終一致 ；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）熱點發現","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 不斷收集 Hermes-SDK 上報的 key 訪問事件 ，對不同業務應用集羣的緩存訪問數據進行週期性（3s 一次）分析計算，以探測業務應用集羣中的熱點 key 列表；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於探測到的熱點 key 列表，Hermes 服務端集羣 將其通過 etcd 集羣 推送給不同業務應用集羣的 Hermes-SDK 通信模塊，通知其對熱點 key 列表進行本地緩存；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）配置讀取","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes-SDK 在啓動及運行過程中，會從 Apollo 配置中心 讀取其關心的配置信息（如：啓動關閉配置、黑白名單配置、etcd 地址...）；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 在啓動及運行過程中，會從 Apollo 配置中心 讀取其關心的配置信息（如：業務應用列表、熱點閾值配置、etcd 地址...）；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"穩定性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 本地緩存穩定性表現在以下方面：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據上報異步化：Hermes-SDK 使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"rsyslog技術","attrs":{}}],"attrs":{}},{"type":"text","text":"對“key 訪問事件”進行異步化上報，不會阻塞業務；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通信模塊線程隔離：Hermes-SDK 的 通信模塊 使用獨立線程池+有界隊列，保證事件上報&監聽的 I/O 操作與業務執行線程隔離，即使出現非預期性異常也不會影響基本業務功能；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存管控：Hermes-SDK 的 熱點模塊 對本地緩存大小上限進行了管控，使其佔用內存不超過 64MB（LRU），杜絕 JVM 堆內存溢出的可能；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一致性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 本地緩存一致性表現在以下方面：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes-SDK 的 熱點模塊 僅緩存 熱點 key 數據，絕大多數非熱點 key數據由 緩存集羣 存儲；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱點 key 變更導致 value 失效時，Hermes-SDK 同步失效本地緩存，保證 本地強一致 ；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱點 key 變更導致 value 失效時，Hermes-SDK 通過 etcd 集羣 廣播事件，異步失效業務應用集羣中其他節點的本地緩存，保證 集羣最終一致 ；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"熱點發現","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"整體流程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4f/4fcd2cfdc811298c0f510800f24641e5.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 熱點發現流程分爲四步：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據收集 ：收集 Hermes-SDK 上報的 key 訪問事件；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱度滑窗 ：對 App 的每個 Key，維護一個時間輪，記錄基於當前時刻滑窗的訪問熱度；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱度匯聚 ：對 App 的所有 Key，以 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"","attrs":{}}],"attrs":{}},{"type":"text","text":"的形式進行 熱度排序彙總；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熱點探測 ：對 App，從 熱 Key 排序彙總 結果中選出 TopN 的熱點 Key ，推送給 Hermes-SDK ；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據收集","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes-SDK 通過本地 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"rsyslog","attrs":{}}],"attrs":{}},{"type":"text","text":"將 key 訪問事件 以協議格式放入 kafka ，Hermes 服務端集羣 的每個節點消費 kafka 消息，實時獲取 key 訪問事件 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"訪問事件協議格式如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"appName：集羣節點所屬業務應用","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"uniqueKey：業務應用 key 訪問事件 的 key","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"sendTime：業務應用 key 訪問事件 的發生時間","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"weight：業務應用 key 訪問事件 的訪問權值","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 節點將收集到的 key 訪問事件 存儲在本地內存中，內存數據結構爲 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map>","attrs":{}}],"attrs":{}},{"type":"text","text":"，對應業務含義映射爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map>","attrs":{}}],"attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"熱度滑窗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/aa/aa04f088123ada22f72ada43180dc489.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"時間滑窗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 節點，對每個 App 的每個 key，維護了一個 時間輪 ：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間輪中共 10 個 時間片 ，每個時間片記錄當前 key 對應 3 秒時間週期的總訪問次數；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間輪 10 個時間片的記錄累加即表示當前 key 從當前時間向前 30 秒時間窗口內的總訪問次數；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"映射任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 節點，對每個 App 每 3 秒 生成一個 映射任務 ，交由節點內 “緩存映射線程池” 執行。映射任務 內容如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對當前 App，從 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map>","attrs":{}}],"attrs":{}},{"type":"text","text":"中取出 appName 對應的 Map ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map>","attrs":{}}],"attrs":{}},{"type":"text","text":"；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"遍歷 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map>","attrs":{}}],"attrs":{}},{"type":"text","text":"中的 key，對每個 key 取出其熱度存入其 時間輪 對應的時間片中；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"熱度匯聚","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/01/01f01290fade246692fb7d2a192d7ab7.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"完成第二步“熱度滑窗”後，映射任務 繼續對當前 App 進行“熱度匯聚”工作：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"遍歷 App 的 key，將每個 key 的 時間輪 熱度進行彙總（即 30 秒時間窗口內總熱度）得到探測時刻 滑窗總熱度 ；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"< key , 滑窗總熱度 >","attrs":{}}],"attrs":{}},{"type":"text","text":" 以排序集合的方式存入 Redis 存儲服務 中，即 熱度匯聚結果；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"熱點探測","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在前幾步，每 3 秒 一次的 映射任務 執行，對每個 App 都會產生一份當前時刻的 熱度匯聚結果 ；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 中的“熱點探測”節點，對每個 App，只需週期性從其最近一份 熱度匯聚結果 中取出達到熱度閾值的 TopN 的 key 列表，即可得到本次探測的 熱點 key 列表 ；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 熱點發現整體流程如下圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/23/236f397a944ca62e1b376dcdb96cb82d.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"特性總結","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"實時性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes-SDK 基於rsyslog + kafka 實時上報 key 訪問事件 。映射任務 3 秒一個週期完成“熱度滑窗” + “熱度匯聚”工作，當有 熱點訪問場景 出現時最長 3 秒即可探測出對應 熱點 key 。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"準確性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"key 的熱度匯聚結果 由“基於時間輪實現的滑動窗口”匯聚得到，相對準確地反應當前及最近正在發生訪問分佈。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"擴展性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 服務端集羣 節點無狀態，節點數可基於 kafka 的 partition 數量橫向擴展。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“熱度滑窗” + “熱度匯聚” 過程基於 App 數量，在單節點內多線程擴展。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"實戰效果","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"快手商家某次商品營銷活動","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有贊商家通過快手直播平臺爲某商品搞活動，造成該商品短時間內被集中訪問產生訪問熱點，活動期間 TMC 記錄的實際熱點訪問效果數據如下：","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"某核心應用的緩存請求&命中率曲線圖**","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2a/2aa9da3c0581ab5177dd958c639efe8c.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖藍線爲應用集羣調用get()方法訪問緩存次數","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖綠線爲獲取緩存操作命中TMC本地緩存的次數","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/36/3692ac3f68f93529340754260dcf5b58.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲本地緩存命中率曲線圖","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看出活動期間緩存請求量及本地緩存命中量均有明顯增長，本地緩存命中率達到近 80%（即應用集羣中 80% 的緩存查詢請求被 TMC 本地緩存攔截）。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"熱點緩存對應用訪問的加速效果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/87/87305fe3c29561d84f6ffc423e8b6cd7.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲應用接口 QPS 曲線","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/44/44376992335b8d93e113b6a8eae05046.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲應用接口 RT 曲線","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看出活動期間應用接口的請求量有明顯增長，由於 TMC 本地緩存的效果應用接口的 RT 反而出現下降。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"雙十一期間部分應用 TMC 效果展示","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"商品域核心應用效果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c9/c9356e28c89136030fdd8feb457fab97.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"活動域核心應用效果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/aa/aadb268c2ba3ab48543d595614114d2d.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d8/d821b12276f939923268e3f6b048696d.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"功能展望","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 目前已爲商品中心、物流中心、庫存中心、營銷活動、用戶中心、網關&消息等多個核心應用模塊提供服務，後續應用也在陸續接入中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TMC 在提供“熱點探測” + “本地緩存”的核心能力同時，也爲應用服務提供了靈活的配置選擇，應用服務可以結合實際業務情況在“熱點閾值”、“熱點 key 探測數量”、“熱點黑白名單”維度進行自由配置以達到更好的使用效果。","attrs":{}}]}]}