鍵 | 說明 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ws:user:clients:${uid} | 存儲用戶和 WebSocket 連接的關係,採用有序集合方式存儲 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ws:guid:clients:${guid} | 存儲文件和 WebSocket 連接的關係,採用有序結合方式存儲 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ws:client:${socket.id} | 存儲當前 WebSocket 連接下的全部用戶和文件關係數據,採用 Redis Hash 方式進行存儲,對應 key 爲 user 和 guid"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由客戶端觸發或組件服務觸發的消息推送,通過 Redis 存儲的數據結構,在 WS-API 服務查詢到返回消息體的目標客戶端的 Socket ID,再有 WS-Gateway 服務進行集羣消費,如果 Socket ID 不在當前節點,則需要進行節點與會話關係的查詢,找到客端戶 Socket ID 實際對應的 WS-Gateway 節點,通常有以下兩種方案:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
|
石墨文檔Websocket百萬長連接技術實踐
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Web 服務端推送技術經過了長輪詢、短輪詢的發展,最終到 HTML5 標準帶來的 WebSocket 規範逐步成爲了目前業內主流技術方案。它使得消息推送、消息通知等功能的實現變得異常簡單,那麼百萬級別連接下的 Websocket 網關該如何實踐呢?本文整理自石墨文檔資深工程師杜旻翔在重構石墨websocket網關的技術實踐。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1 引言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在石墨文檔的部分業務中,例如文檔分享、評論、幻燈片演示和文檔表格跟隨等場景,涉及到多客戶端數據同步和服務端批量數據推送的需求,一般的 HTTP 協議無法滿足服務端主動 Push 數據的場景,因此選擇採用 WebSocket 方案進行業務開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着石墨文檔業務發展,目前日連接峯值已達百萬量級,日益增長的用戶連接數和不符合目前量級的架構設計導致了內存和 CPU 使用量急劇增長,因此我們考慮對網關進行重構。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2 網關 1.0"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 1.0 是使用 Node.js 基於 Socket.IO 進行修改開發的版本,很好的滿足了當時用戶量級下的業務場景需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1 架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 1.0 版本架構設計圖:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/29\/29ecb547e8148d8b00938726ffa28b81.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 1.0 客戶端連接流程:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"用戶通過 NGINX 連接網關,該操作被業務服務感知;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"業務服務感知到用戶連接後,會進行相關用戶數據查詢,再將消息 Pub 到 Redis;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"網關服務通過 Redis Sub 收到消息;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"查詢網關集羣中的用戶會話數據,向客戶端進行消息推送。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.2 痛點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然 1.0 版本的網關在線上運行良好,但是不能很好的支持後續業務的擴展,並且有以下幾個問題需要解決:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源消耗:Nginx 僅使用證書,大部分請求被透傳,產生了一定的資源浪費,同時之前的 Node 網關性能不好,消耗大量的 CPU、內存。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"維護與觀測:未接入石墨的監控體系,無法和現有監控告警聯通,維護上存在一定的困難;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務耦合問題:業務服務與網關功能被集成到了同一個服務中,無法針對業務部分性能損耗進行鍼對性水平擴容,爲了解決性能問題,以及後續的模塊擴展能力,都需要進行服務解耦。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3 網關 2.0"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 2.0 需要解決很多問題:石墨文檔內部有很多組件:文檔、表格、幻燈片和表單等等。在 1.0 版本中組件對網關的業務調用可以通過:Redis、Kafka 和 HTTP 接口,來源不可查,管控困難。此外,從性能優化的角度考慮也需要對原有服務進行解耦合,將 1.0 版本網關拆分爲網關功能部分和業務處理部分,網關功能部分爲 WS-Gateway:集成用戶鑑權、TLS 證書驗證和 WebSocket 連接管理等;業務處理部分爲 WS-API:組件服務直接與該服務進行 gRPC 通信。可針對具體的模塊進行鍼對性擴容;服務重構加上 Nginx 移除,整體硬件消耗顯著降低;服務整合到石墨監控體系。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1 整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 2.0 版本架構設計圖:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4d\/4d9f02fdb5d3f82329a2208e3e823400.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關 2.0 客戶端連接流程:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端與 WS-Gateway 服務通過握手流程建立 WebSocket 連接;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"連接建立成功後,WS-Gateway 服務將會話進行節點存儲,將連接信息映射關係緩存到 Redis 中,並通過 Kafka 向 WS-API 推送客戶端上線消息;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"WS-API 通過 Kafka 接收客戶端上線消息及客戶端上行消息;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"WS-API 服務預處理及組裝消息,包括從 Redis 獲取消息推送的必要數據,並進行完成消息推送的過濾邏輯,然後 Pub 消息到 Kafka;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"WS-Gateway 通過 Sub Kafka 來獲取服務端需要返回的消息,逐個推送消息至客戶端。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2 握手流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡狀態良好的情況下,完成如下圖所示步驟 1 到步驟 6 之後,直接進入 WebSocket 流程;網絡環境較差的情況下,WebSocket 的通信模式會退化成 HTTP 方式,客戶端通過 POST 方式推送消息到服務端,再通過 GET 長輪詢的方式從讀取服務端返回數據。客戶端初次請求服務端連接建立的握手流程:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b7\/b7f2855975aa21eb91030197925db441.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Client 發送 GET 請求嘗試建立連接;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Server 返回相關連接數據,sid 爲本次連接產生的唯一 Socket ID,後續交互作爲憑證;"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"{\"sid\":\"xxx\",\"upgrades\":[\"websocket\"],\"pingInterval\":xxx,\"pingTimeout\":xxx}"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Client 攜帶步驟 2 中的 sid 參數再次請求;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Server 返回 40,表示請求接收成功;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Client 發送 POST 請求確認後期降級通路情況;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"Server 返回 ok,此時第一階段握手流程完成;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"嘗試發起 WebSocket 連接,首先進行 2probe 和 3probe 的請求響應,確認通信通道暢通後,即可進行正常的 WebSocket 通信。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.3 TLS 內存消耗優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端與服務端連接建立採用的 wss 協議,在 1.0 版本中 TLS 證書掛載在 Nginx 上,HTTPS 握手過程由 Nginx 完成,爲了降低 Nginx 的機器成本,在 2.0 版本中我們將證書掛載到服務上,通過分析服務內存,如下圖所示,TLS 握手過程中消耗的內存佔了總內存消耗的大概 30% 左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a4\/a425ea15769e24313301af1757daf747.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個部分的內存消耗無法避免,我們有兩個選擇:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用七層負載均衡,在七層負載上進行 TLS 證書掛載,將 TLS 握手過程移交給性能更好的工具完成;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"優化 Go 對 TLS 握手過程性能,在與業內大佬曹春暉(曹大)的交流中瞭解到,他最近在 Go 官方庫提交的 PR "},{"type":"link","attrs":{"href":"https:\/\/github.com\/golang\/go\/issues\/43563","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/golang\/go\/issues\/43563"}]},{"type":"text","text":" ,以及相關的性能測試數據 "},{"type":"link","attrs":{"href":"https:\/\/github.com\/golang\/go\/pull\/48229","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/golang\/go\/pull\/48229"}]},{"type":"text","text":" 。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.4 Socket ID 設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對每次連接必須產生一個唯一碼,如果出現重複會導致串號,消息混亂推送的問題。選擇 SnowFlake 算法作爲唯一碼生成算法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"物理機場景中,對副本所在物理機進行固定編號,即可保證每個副本上的服務產生的 Socket ID 是唯一值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"K8S 場景中,這種方案不可行,於是採用註冊下發的方式返回編號,WS-Gateway 所有副本啓動後向數據庫寫入服務的啓動信息,獲取副本編號,以此作爲參數作爲 SnowFlake 算法的副本編號進行 Socket ID 生產,服務重啓會繼承之前已有的副本編號,有新版本下發時會根據自增 ID 下發新的副本編號。於此同時,Ws-Gateway 副本會向數據庫寫入心跳信息,以此作爲網關服務本身的健康檢查依據。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.5 集羣會話管理方案:事件廣播"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端完成握手流程後,會話數據在當前網關節點內存存儲,部分可序列化數據存儲到 Redis,存儲結構說明如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.