自研消息隊列架構設計文檔

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文是遊戲業務線自研消息隊列中間件詳細架構設計文檔，用於指導自研消息隊列後續的開發、測試和運維","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"詳細架構設計文檔。自研的消息隊列採用自研集羣和MySQL存儲。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"詞彙表","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、業務背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務發展很快，系統也越來越多，系統間協作的效率很低，業務上拆分的子系統越來越多，目前系統間的調用都是同步調用，由此帶來幾個明顯的系統問題：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能問題：各個子系統交互頻繁，目前都是系統間通過接口調用的，性能極其低下。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"耦合問題：當新增一個子系統時，例如如果要增加“廣告子系統”，那麼廣告子系統需要開發新的接口給微博發佈子系統調用。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"效率問題：每個子系統提供的接口參數和實現都有一些細微的差別，導致每次都需要重新設計接口和聯調接口，開發團隊和測試團隊花費了許多重複工作量。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上背景，這些問題背後的根本原因在於架構上各業務子系統強耦合，我們需要引入消息隊列進行系統解耦，將目前的同步調用改爲異步通知。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"二、約束和限制","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中間件團隊規模不大，大約6人左右。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前整個業務系統是單機房部署，沒有雙機房。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"三、總體架構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/25/25f942594d04be6130ec2d2a2f31407e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1 架構分析","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1.1 高性能","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統一天內平均每秒寫入消息數爲115條，每秒讀取的消息數是1150條，再考慮到系統的讀寫並不是完全平均的，設計的目標應該以峯值來計算。峯值一般取平均值的3倍，那麼消息隊列系統的TPS是 345，QPS是3450，這個量級的數據意味這並不 需要高性能。雖然根據當前業務規模計算的性能要求並不高，但業務會增長，因此係統設計需要考慮一定的性能餘量。由於現在的基數較低，因此係統的設計目標按照峯值的4倍來計算，因此最終的性能要求是：TPS爲1380，QPS爲13800。TPS爲1380並不高，但是QPS爲13800易筋經比較高了，因此高性能讀取是複雜度之一，但這個高性能要求相比Kafka等系統來說也不是很高。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1.2 高可用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於微博子系統來說，如果消息丟了，導致沒有審覈，然後觸犯了國家法律法規，則是非常嚴重的事情；對於等級子系統來說，如果用戶達到相應等級後，系統沒有給他獎品和專屬服務，則 VIP 用戶會很不滿意，導致用戶流失從而損失收入，雖然也比較關鍵，但沒有審覈子系統丟消息那麼嚴重。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合來看，消息隊列需要高可用性，包括消息寫入、消息存儲、消息讀取都需要保證高可用性。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1.3 高可擴展性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列的功能很明確，基本無需擴展，因此可擴展性不是這個消息隊列的複雜度關鍵。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合分析下來，消息隊列的複雜性主要體現在幾個方面：高性能消息讀取、高可用消息 寫入、高可用消息存儲、高可用消息讀取。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2 總體架構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/25/25f942594d04be6130ec2d2a2f31407e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）採用數據分散集羣的架構，集羣中的服務器進行分組，每個分組存儲一部分消息數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）每個分組包含一臺主 MySQL 和一臺備 MySQL，分組內主備數據複製，分組間數據不同步。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（3）正常情況下，分組內的主服務器對外提供消息寫入和消息讀取服務，備服務器不對外提供服務；主服務 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（4）器宕機的情況下，備服務器對外提供消息讀取的服務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（5）客戶端採取輪詢的策略寫入和讀取消息。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"四、詳細設計","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.1 核心功能","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息發送流程","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務系統發佈消息時，首先寫入日誌表，日誌表寫入成功就代表消息寫入成功；後臺線程再從日誌表中讀取消息寫入記錄，將消息內容寫入消息表中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息消費流程","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列系統提供SDK供各業務系統調用，SDK從配置中讀取所有消息隊列系統的服務器信息，輪流向所有服務器發起消息讀取請求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列服務器需要記錄每個消費者的消費狀態，即當前消費者已經讀取到了哪些消息，當收到消息讀取請求時，返回下一條未被讀取的消息給消費者。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.2 關鍵設計","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息發送可靠性","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務服務器中嵌入消息隊列系統提供的 SDK，SDK 支持輪詢發送消息，當某個分組的主服務器無法發送消息時，SDK 挑選下一個分組主服務器重發消息，依次嘗試所有主服務器直到發送成功；如果全部主服務器都無法發送，SDK 可以緩存消息，也可以直接丟棄消息，具體策略可以在啓動 SDK 的時候通過配置指定。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果 SDK 緩存了一些消息未發送，此時恰好業務服務器又重啓，則所有緩存的消息將永久丟失，這種情況 SDK 不做處理，業務方需要針對某些非常關鍵的消息自己實現永久存儲的功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息存儲可靠性","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息存儲在 MySQL 中，每個分組有一主一備兩臺 MySQL 服務器，MySQL 服務器之間複製消息以保證消息存儲高可用。如果主備間出現複製延遲，恰好此時 MySQL 主服務器宕機導致數據無法恢復，則部分消息會永久丟失，這種情況不做針對性設計，DBA 需要對主備間的複製延遲進行監控，當複製延遲超過 30 秒的時候需要及時告警並進行處理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息如何存儲","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個消息隊列對應一個 MySQL 表，消息隊列名就是表名，包含的字段：消息ID（遞增生成）、消息內容、消息發佈時間、消息發佈者。日誌表表名爲MQ_LOG ，包含的字段：日誌ID、發佈者信息、發佈時間、隊列名稱、消息內容。日誌表需要及時清除已經寫入消息表的日誌數據，消息表最多保存30天的消息數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲消息的整體流程：業務系統發佈消息時，首先寫入日誌表，日誌表寫入成功就代表消息寫入成功；後臺線程再從日誌表中讀取消息寫入記錄，將消息內容寫入消息表中。業務系統讀取消息時，從消息表中讀取。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.3 設計規範","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列服務器使用 Spring Boot + Netty 開發","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL 使用 Innodb 存儲引擎","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"五、質量設計","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.1 安全設計","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫連接字符串採用加密方式統一保存。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止SQL注入（開發用代碼實現，如參數化sql等技術）。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有敏感業務操作均記錄日誌","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.2 可靠性設計","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用Zookeeper 來做主備決測，主備服務器都連接到zookeeper 建立自己的節點，主服務器的路徑規則爲“MQ/server/分區編號/master”，備機爲“MQ/server/分區編號/slave”，節點類型爲EPHEMERAL。備機監聽主機的節點 信息，當發現主服務器節點斷連後，備服務器修改自己的狀態，對外提供消息讀取服務。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"六、演進規劃","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列一期基本功能快速上線，但要基本完善後才能正式推出給其他人用。","attrs":{}}]}]}