Uber實時推送平臺實踐:gRPC 推動基礎設施的發展

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"本文最初發表於Uber官方博客,InfoQ經對方授權對全文進行了如下翻譯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber的業務遍佈全球,每天需要處理全球數百萬人次的出行,實時性對Uber而言非常重要。在一次行程中,多個參與者可以修改和查看正在進行中的行程狀態,這需要實時更新。無論是取車時間、到達時間、路線還是在打開應用時附近的司機數量,所有參與者和應用都必須保持實時信息同步。本文介紹了Uber如何通過輪詢保持信息實時更新以及基於gRPC 雙向流協議構建應用。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"輪詢更新"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Uber的應用場景下,司機側需要每隔幾秒鐘對服務器進行輪詢,以查看是否有新的訂單。乘客側可以每隔幾秒鐘輪詢一次服務器,以檢查是否分配了司機。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些應用程序的輪詢頻率取決於所輪詢的數據變化率。在Uber這樣的大型應用中,變化率的取值範圍非常大,從幾秒鐘到幾小時不等。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"移動應用輪詢的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在某些時候,發送到後端 API 網關的請求中 80% 都是輪詢調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主動輪詢可以保持應用響應,但是會佔用大量服務器資源。輪詢頻率上的任何錯誤都會導致後端負載和性能下降。當需要更多實時動態數據時,這種方法是行不通的,因爲它將在後端增加大量負載。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輪詢會導致電池消耗加快、應用遲鈍以及網絡級擁堵。這一點在 2G\/3G 網絡或整個城市網絡不穩定的地方尤爲明顯,應用在每次輪詢時都會多次嘗試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着功能數量的增加,開發者試圖讓現有的輪詢 API 過載,或創建一個新的 API。在高峯時期,該應用會輪詢幾十個 API。每一個 API 都有多個功能過載。這些輪詢 API 最終只會變成一組負載分片的 API,供應用輪詢其功能。保持 API 級別的一致性和邏輯分離,仍然是一個日益增長的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用的冷啓動是輪詢策略中最具挑戰的場景。每當用戶打開應用,所有功能都需要從後端提取出最新的狀態來渲染用戶界面。這樣就會產生多個競爭的併發 API 調用,應用只有從服務器獲取關鍵組件後才能渲染。由於所有 API 都包含一些關鍵信息的片段,沒有優先級,因此應用的加載時間會持續增加。惡劣的網絡條件會使冷啓動問題更加惡化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯然,我們需要對市場上各個參與者的狀態同步方式進行徹底改革。在創建推送消息平臺的過程中,我們允許服務器根據需要嚮應用發送數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在採用該體系結構時,我們發現效率有了顯著提高,同時也解決了很多問題和挑戰。接下來的部分,我們將介紹整個該平臺是如何演變的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"取消輪詢,引入 RAMEN"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然使用推送消息是取消輪詢的必然選擇,但是如何進行架構設計卻有許多考慮。主要的設計原則有以下四點。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"從輪詢遷移到推送更容易"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現有的很多輪詢端點都可以爲企業提供動力。在不重寫的情況下,新系統必須利用現有輪詢 API 中的負載來構建業務邏輯。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"易於開發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與開發輪詢 API 相比,開發人員不應採取完全不同的方式來推送數據。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"可靠性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在網絡上,所有消息都應該可靠地發送,如果發送失敗,則應重試"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"線路效率"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Uber的快速發展,對用戶來說傳輸數據的成本是一個挑戰,特別是那些每天與平臺連接數小時的司機。這個協議必須在服務器和移動應用之間傳輸最少的數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將新系統命名爲 RAMEN("},{"type":"text","marks":[{"type":"strong"}],"text":"R"},{"type":"text","text":"ealtime"},{"type":"text","marks":[{"type":"strong"}],"text":"A"},{"type":"text","text":"synchronous"},{"type":"text","marks":[{"type":"strong"}],"text":"M"},{"type":"text","text":"Essaging"},{"type":"text","marks":[{"type":"strong"}],"text":"N"},{"type":"text","text":"etwork,意即實時異步消息網絡)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f1\/f1e2d1a07e6f839d5894b86c530d7be8.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 1:整個系統的高級架構"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"確定生成消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時信息隨時都在數百名乘客、司機、餐廳和行程中發生變化。信息的生命週期從確定“何時”爲用戶生成信息負載開始。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Fireball 是一個微服務,負責解決“何時推送消息”的問題。決策的很大一部分被捕獲爲配置。它監聽系統中發生的各種類型事件,並確定是否需要推送所涉及的用戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子,當司機“接受”訂單時,司機和行程實體的狀態會發生變化。這種變化會觸發 Fireball 服務。Fireball 根據配置來確定應該將哪種推送消息發送給相關的參與者。許多情況下,一次觸發需要多個用戶的多個消息負載。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"觸發器可以是任何類型的重要事件,並且應該爲其生成推送負載。例如,像請求乘車這樣的用戶操作、應用程序的打開、固定時間間隔的計時器滴答聲、消息總線上的後端業務事件,或者地理上的出入口事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些觸發器全部被過濾,然後轉換成對各種 API 網關端點的調用。爲了產生合適的本地化響應負載,API 網關需要諸如設備區域、設備操作系統和應用程序版本等用戶上下文信息。在調用 API 網關時, Fireball 獲取設備上的下文 RAMEN 服務器,並將其添加到頭文件。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"生成消息負載"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來自Uber應用的所有服務器調用都由 API 網關提供("},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/gatewayuberapi\/","title":"","type":null},"content":[{"type":"text","text":"在此"}]},{"type":"text","text":"閱讀有關我們網關演變的更多信息),推送的負載也以同樣的方式生成。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當 Fireball 確定誰以及何時推送消息時,API 網關負責確定“推送什麼”。網關調用各種域服務來產生正確的推負載。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在產生負載的方式上,網關中的所有 API 都是類似的。但是,這個 API 可以分爲 Pull API 和 Push API。"},{"type":"text","marks":[{"type":"strong"}],"text":"Pull API"},{"type":"text","text":"指的是在移動設備上爲任何 HTTP 操作調用的端點。所謂"},{"type":"text","marks":[{"type":"strong"}],"text":"Push API"},{"type":"text","text":",就是從 Fireball 調用的端點,還有一個附加的 Push 中間件,它可以截取來自 Pull API 的響應並將其轉發給 Push 消息傳輸系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這兩者之間設置 API 網關有許多好處:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pull API 和 Push API 共享了大多數端點業務邏輯。一個給定的負載可以從 Pull API 無縫地切換到 Push API。舉例來說,不管應用是通過 Pull API 調用來拉取用戶對象,還是通過 Push API 調用來發送用戶對象,都使用相同的邏輯。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關負責處理許多交叉問題,如推送消息的速率限制、路由和模式驗證。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與網關一起, Fireball 生成推送消息,並在適當的時候發送給用戶。“推送消息系統”負責向移動設備發送此消息。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"推送消息負載元數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲進行優化,每個推送消息都有不同的配置。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"優先級"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲對於不同的用例,會產生數百種不同的負載,所以首先要對發送給應用的東西進行優先級排序。我們接下來將會看到Uber所採用的協議限制了在一個連接上發送多個併發負載。接收設備的帶寬也受到限制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲獲取相對優先級的感受,在理解其影響的基礎上,將消息大致分成三個不同的優先級:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高:重要的核心用戶體驗消息"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"中:增量用戶體驗功能消息"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"低:高數據負載大小或低頻非關鍵消息"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種優先級配置隨後將用於管理平臺的各種行爲。舉例來說,當建立連接時,消息以優先級的遞減順序被放入套接字中。如果 RPC 發生故障,通過服務器端重試,高優先級的消息會變得更可靠,並支持跨區域複製。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"存活時間"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推送消息是爲了改善實時體驗。因此,每條消息都有一個定義的存活時間值,範圍從幾秒到 30 分鐘不等。消息傳輸系統會將消息持久化,並重新嘗試傳輸消息,直到存活時間值到期。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"重複數據刪除"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該配置確定了在通過不同的觸發器或重試多次產生相同的消息類型時,推送消息是否應該被重複數據刪除。對大多數用例而言,發送特定類型的最新推送消息就足夠了,這使我們能夠降低總體數據傳輸率。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消息傳輸"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推送消息系統的最後一個組成部分是實際的負載傳輸服務。它的功能是在消息負載到達之後,保持與全球數以百萬計的移動應用程序的有效連接,並快速地將它們同時發送。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"世界各地的移動網絡提供了不同程度的可靠性,所以傳輸系統需要對故障作出可靠響應。本系統提供“至少一次”的傳輸保障。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RAMEN 傳輸協議"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要提供可靠的傳輸渠道,我們必須利用基於 TCP 的持久連接,以便從應用向我們的數據中心提供傳輸服務。在 2015 年的應用協議中,我們的選擇是利用 HTTP\/1.1 與長輪詢、Web Sockets 或最終的服務器發送事件(Server-Sent events,SSE)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據安全性、對移動 SDK 的支持、二進制大小的影響等因素,我們最終選擇了 SSE,這對於Uber已經支持的 HTTP+ JSON API 協議棧是非常簡單可行的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而,SSE 是一個單向協議,即數據只能從服務器發送到應用。爲提供至少一次上述保證,需要在應用協議之上建立傳輸協議來進行確認和重試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 SSE 定義了一種非常優雅和簡單的協議方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d2\/d28437935cab91a676cafb84c35698dc.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 2:SSE協議的服務器 - 客戶端交互"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端在任何新的會話開始時,在第一次 HTTP 請求"},{"type":"codeinline","content":[{"type":"text","text":"\/ramen\/receive?seq=0"}]},{"type":"text","text":"時開始接收消息,序列號爲 0。服務器用 HTTP 200 和“Content-Type: text\/event-stream”響應,以維持 SSE 連接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來,服務器按照優先級降序發送所有待處理的消息,並關聯遞增的序列號。因爲底層的傳輸協議是 TCP 連接,所以如果沒有發送 seq#3 的消息,則該連接應該已斷開、超時或失敗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端應該在下次使用它看到的最大序列號(本例中 seq=2)重新連接。這樣可以告訴服務器即使 3 號被寫進套接字中,它也不會被髮送。這樣,服務器就會重新發送相同的消息或者以 seq=3 開始的任何新優先級更高的消息。該協議建立了所需的可恢復流式連接,服務器做大部分的存儲工作,在客戶端實現起來很簡單。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解連接是否處於活動狀態,服務器每 4 秒發送一條單字節大小的心跳消息。若客戶機在 7 秒內未看到心跳或消息,則認爲連接已中斷並重新連接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上述協議中,每當客戶端使用更高的序列號重新連接時,它作爲一個確認機制,允許服務器刷新舊消息。當網絡運行良好時,用戶可以保持長達幾分鐘的連接,從而使服務器繼續積累舊消息。爲減輕此問題,應用會每 30 秒調用"},{"type":"codeinline","content":[{"type":"text","text":"\/ramen\/ack?seq=N"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協議的簡單性允許用許多不同的語言和平臺快速編寫客戶端。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"設備上下文存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次建立連接時,RAMEN 服務器都附加存儲設備上下文。這個上下文公開給 Fireball,這樣用戶就可以訪問設備上下文,該id根據用戶及其設備參數產生,具有唯一的哈希值。這樣,即使用戶同時使用多個設備或應用,且設置不同,也可以隔離推送消息。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"消息存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RAMEN 服務器將所有的消息保存在內存中,或者備份在數據庫中。如果連接不穩定,服務器可以繼續重試發送,直到 TTL 到期。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"實施細節"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首代 RAMEN 服務器是用 Node.js 編寫的,使用了“Ringpop”這個優步內部一致的哈希 \/ 分片框架。"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/ringpop-open-source-nodejs-library\/","title":"","type":null},"content":[{"type":"text","text":"Ringpop"}]},{"type":"text","text":"是一種去中心化的分片系統。所有的連接都用一個用戶 UUID 進行分片,並使用 Redis 作爲持久化數據存儲。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"在全球範圍部署 RAMEN"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨後的一年半時間裏,該推送平臺在全公司得到了廣泛應用。該系統最多保持 60 萬個併發流連接,在高峯期每秒向三種不同類型的應用程序推送超過 7 萬條 QPS 推送消息。該系統很快就成爲服務器客戶端 API 基礎設施中最重要的部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着流量和持久連接數量的增加,我們的技術選擇也需要進行擴展。基於 Ringpop 的分佈式分片是一個非常簡單的架構,但是不能隨着環中節點的增加而擴展。Ringpop 庫使用 gossip 協議對成員進行評估。gossip 協議的收斂時間隨環的大小增加而增加。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外,Node.js worker 是單線程的,它會增加事件循環的滯後程度,從而使成員信息的收斂變得更慢。這會導致拓撲信息不一致,以及消息丟失、超時和出錯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2017 年初,我們決定重啓 RAMEN 協議的服務器實現,以便繼續擴展。我們在本次迭代中使用了下列技術:Netty、Apache Zookeeper、Apache Helix、Redis 和 Apache Cassandra。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Netty"},{"type":"text","text":":Netty 是一個廣泛使用的高性能庫,用於構建網絡服務器和客戶端。Netty 的 bytebuf 支持零拷貝緩衝區,這使得系統非常高效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Apache ZooKeeper"},{"type":"text","text":":擁有一致的網絡連接哈希,可以在中間不需要任何存儲層的情況下直接傳輸數據。但是,我們沒有選擇分散式的拓撲管理,而是選擇了 ZooKeeper 的集中式共享。ZooKeeper 是一種非常強大的分佈式同步和配置管理系統,可以快速檢測連接節點的故障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Apache Helix"},{"type":"text","text":":Helix 是一個強大的集羣管理框架,在 ZooKeeper 上運行,支持定義自定義拓撲和重新平衡算法。它還很好地將拓撲邏輯從核心業務邏輯中抽象出來。它使用 ZooKeeper 來監控連接的工作器,並傳播分片狀態信息的變化。它還允許我們編寫一個自定義的“Leader-Follower”拓撲結構,以及自定義的漸進式再平衡算法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Redis 與 Apache Cassandra:由於我們正準備進行多區域雲架構,所以需要適當地複製和存儲消息。Cassandra 是一種持久的、跨區域複製的存儲。在 Cassandra 之上使用 Redis 作爲容量緩存,以避免由部署或故障轉移事件中常見的碎片系統引起的驚羣問題(thundering herd problems)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"譯註"},{"type":"text","text":":驚羣問題(thundering herd problems),是計算機科學中,當許多進程等待一個事件,事件發生後這些進程被喚醒,但只有一個進程能獲得 CPU 執行權,其他進程又得被阻塞,這造成了嚴重的系統上下文切換代價。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3f\/3fd9135056e9989d3ed6f57a5dd472cd.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 3:新 RAMEN 後端服務器的架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"StreamGate"},{"type":"text","text":":該服務實現了 Netty 上的 RAMEN 協議,並且擁有所有的邏輯,包括連接處理、消息和存儲。這個服務還實現了一個 Apache Helix 參與者,它與 ZooKeeper 建立連接,並維護心跳。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"StreamgateFE"},{"type":"text","text":"("},{"type":"text","marks":[{"type":"strong"}],"text":"Streamgate Front End"},{"type":"text","text":"):該服務作爲一個 Apache Helix Spectator,監聽 ZooKeeper 的拓撲變化。它實現了一個反向代理。每個來自客戶端 Fireball、網關或移動應用的請求都是使用拓撲信息分片並路由到正確的 Streamgate 工作器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Helix Controllers"},{"type":"text","text":":顧名思義,這是一個由五個節點組成的獨立服務,只負責運行 Apache Helix Controller 進程,是進行拓撲管理的“大腦”。無論何時啓動或停止任何 Streamgate 節點,它都會檢測到更改並重新分配分片分區。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近幾年來,我們一直在使用這種架構,並能夠在服務器端實現 99.99% 的基礎架構可靠性。這種推送基礎設施的應用越來越多,支持 iOS、 Android 和 Web 平臺上的十幾種不同類型的應用。在 150 萬以上併發連接的情況下,我們已經在運行這個系統,每秒推送超過 25 萬條消息。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"gRPC 推動基礎設施的未來"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種服務器端基礎設施一直保持穩定。在網絡條件日益改善、應用範圍日益擴大的新城市裏,我們將致力於不斷提高向移動設備發送消息的長尾可靠性。爲了填補空白,我們已經嘗試了新的協議和開發方法。經過檢驗,以下幾點是導致可靠性下降的主要原因。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"丟失確認"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上文定義的 RAMEN 協議是爲減少數據傳輸而優化的,因此只有每 30 秒或在客戶端重新連接時纔會報告確認。這樣會導致確認被延遲,在某些情況下,確認消息傳輸失敗。這樣很難區分消息的真實丟失與確認請求的失敗。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"連接穩定性差"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"保持服務器和客戶端之間的健康連接是至關重要的。在處理錯誤、超時、後退或應用程序生命週期事件(打開或關閉)、網絡狀態變化、主機名和數據中心故障切換方面,不同平臺的客戶端實現之間存在許多細微的差別。這樣會在不同版本中產生性能差異。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"傳輸限制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲協議是在 SSE 上實現的,所以數據傳輸是單向的。有些新的應用經驗需要我們實現雙向信息傳輸。在沒有實時測量往返時間的情況下,無法確定網絡狀況、傳輸速度和減少線路阻塞。在沒有 base64 這樣的文本編碼的情況下, SSE 也是一種基於文本的協議,這限制了我們傳輸二進制負載的能力,導致負載變大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2019 年底,我們開始開發下一代的 RAMEN 協議,以克服上述缺點。經過大量的考慮,我們選擇了在 gRPC 上構建它。 gRPC 是一種被廣泛採用的 RPC 協議棧,具有跨多種語言的客戶機和服務器的標準化實現。對於許多不同的 RPC 方法,它提供了一流的支持,並且可以與 QUIC 傳輸層協議進行互操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新的基於 GRPC 的 RAMEN 協議擴展了以前的基於 SSE 的協議,但其中存在一些重要區別:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前將立即發送反向流的確認。這樣就提高了確認的可靠性,但數據傳輸量卻略微增加。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時確認使我們能夠實時測量 RTT,瞭解網絡狀態。可將消息的真實損失與網絡損失區分開來。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其提供了協議之上的抽象層來支持流複用等功能。同時也使我們可以嘗試採用網絡優先級和流控制算法來提高數據使用和通信延遲的效率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個協議抽象了消息負載,以支持不同類型的序列化。將來我們可以研究其他的序列化,但 gRPC 保留在傳輸層。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強大的不同語言客戶端實現還允許我們快速支持不同類型的應用和設備。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這項工作目前正處於測試發佈階段,目前對其未來看好。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Uber的出行體驗中, 推送平臺是一個不可或缺的部分。現在這個平臺已經提供了上百種不同的功能,以下是該平臺在獲得巨大成功的幾個主要原因。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"關注點分離"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着業務需求的變化,消息觸發、創建和交付系統之間的明確職責劃分使我們能夠將焦點轉移到平臺的不同部分。在 Apache Helix 中,將交付組件與拓撲邏輯、流媒體的核心業務邏輯高度分離,這使得 gRPC 在完全相同的架構上得到支持,但是使用不同的線協議。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"行業標準技術"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以行業標準技術爲基礎,使實施更穩健且長遠,成本更低。上面的系統維護開銷已經很小了。作爲一個平臺,我們可以在團隊規模上提供極高的效率。以我們的經驗來看,Helix 和 Zookeeper 非常穩定。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"更簡單的設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這個協議,我們可以在不同的網絡環境下擴展到數百萬用戶、數百種功能和數十種應用,這個協議的簡單性使得它容易擴展,發展迅速。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uday Kiran Medisetty,Uber高級工程師;Nilesh Mahajan,Uber工程師;Anirudh Raja,Uber二級軟件工程師;Madan Thangavelu,Uber高級工程經理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/eng.uber.com\/real-time-push-platform\/"}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章