乾貨 | 秒級上下線，攜程服務註冊中心架構演進

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"攜程的微服務框架產品從2013年發展至今，已經歷了7年多的打造。其中所使用的服務註冊中心也從最開始人工數據維護架構演進到了現在全自動、百萬容量級的架構。本文將逐一回顧攜程服務註冊中心所經歷的三輪迭代過程，並重點介紹最新的第三版架構的設計與實現。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、服務註冊中心是什麼？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/70\/70bc6684d1d7089bfab25d8d16714115.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖2-1 微服務架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"微服務架構中所要解決的最核心的技術問題有兩點，一個是服務發現，另一個是負載均衡。而服務註冊中心就是用來解決服務發現問題的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖2-1 所示，在微服務架構中，服務提供方(ServiceProvider)，需要手動或自動地將服務地址註冊到服務註冊中心（Registry）。註冊的信息包括但不限於ServiceID 和URL。服務消費方（ServiceConsumer）在首次調用服務前，需要先從服務註冊中心查詢對應服務的註冊信息，然後依據返回的服務地址信息來發起調用。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、攜程服務註冊中心演變史"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 人工數據維護階段"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在攜程微服務架構推廣初期，爲了快速搭建微服務體系，服務的調用過程繼續使用傳統的域名方式來進行。在服務治理層面，服務提供方首先需要將服務的一個完整URL提交到註冊中心。服務消費方在運行時會定期從註冊中心同步最新的URL，併發起服務調用。而這個URL與應用服務器的關聯關係則由運維人員人工在負載均衡設備上配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種模式下的服務註冊中心的優點是結構簡單、容易實現且運維工作量小，有利於微服務架構快速推廣。而缺點則主要集中在以下幾個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置複雜：負載均衡和服務發現的數據依賴人工維護，影響開發效率和體驗。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單點問題：服務調用強依賴於負載均衡設備，該設備的可用性會直接影響到微服務體系的可用性。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能問題：服務調用需要經過一層負載均衡設備，存在額外的網絡開銷，會直接影響到性能。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 基於etcd的服務註冊中心"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在攜程微服務體系擴展到Java平臺時，我們希望能夠解決前面由於使用外部負載均衡設備所帶來的各種缺陷，所以計劃將負載均衡設備的功能集成到微服務的SDK中，同時由註冊中心下發的服務註冊信息從之前的固定使用域名的URL，改爲服務集羣各臺服務器IP所對應的URL。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"改進後的工作流程是這樣的：服務提供方啓動後，SDK會把包含本機 IP 的服務實例地址上報給註冊中心；而服務消費方啓動後，SDK會定期從註冊中心獲取最新的服務地址列表，並使用內置的負載均衡算法選出一個地址來發起請求。同時，爲了保證服務註冊數據的有效性，其中設置有“存活時間”（TTL，Time to live）。所以需要服務註冊中心支持清理過期的註冊數據。在設計新的架構時，綜合以上這些考慮，我們選擇了etcd來存儲服務註冊數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a3\/a397d88aede091bfa03d0b0bad27de36.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖3-1 基於 etcd 的服務註冊中心架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 etcd 的服務註冊中心整體架構，如圖 3-1所示，包含三個角色。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供應用接入服務註冊中心的基本 API。應用通過嵌入到應用程序內的 SDK，實現服務的自注冊和自發現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Session"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責處理Client提交的服務註冊和發現請求。Client的請求經Session協議轉換後，直接轉發給 etcd。etcd的響應數據經Session的服務治理邏輯處理後，再返回給Client。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"etcd"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責存儲服務註冊數據。集羣內各節點間使用Raft協議來進行數據同步。在沒有網絡分區的情況下，節點上數據可以做到完全一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"etcd 滿足CAP（Consistency：數據一致性、Availability：服務可用性、Partition-tolerance：分區容錯性）中的 CP，即優先考慮分佈式緩存的數據一致性。從其設計的出發點來看，etcd 不適合對讀寫性能要求特別高的場景，而是適合量小且需要高可靠和一致性數據存儲服務，比如配置數據、K8s中的集羣元數據等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在經過一段時間的線上部署和運維後，我們發現etcd中存在潛在的可用性和性能問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先說下可用性問題。假設etcd集羣存在 A、B、C、D和E 五個節點，A 是當前集羣的 Leader 節點。如果此時發生網絡分區故障，其中A、B 在一個分區，而C、D和E在另一個分區。Leader A向所有的Follower發送心跳，但無法獲取到大多數節點響應（計算公式爲(N+2)\/2，即在擁有五個節點的集羣中需要至少獲得三個節點的響應）。心跳超時後，集羣進入選舉階段。但受到網絡分區的影響，A和B都無法獲得大多數節點投票。所以由於缺少Leader，A和B 所在的分區會處於不可用的狀態，無法寫入數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再說下性能問題。etcd所有的寫操作都由 Leader 節點負責執行。而自注冊服務實例的健康檢測，是依賴註冊中心數據中的過期機制實現的。所以各個服務實例需要不斷的發送心跳，來保持數據的活躍和有效。但這樣就會產生大量的寫操作，對Leader節點的性能和網絡帶寬都是一個極大的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在服務發現的場景下，服務註冊中心的可用性比數據一致性更加重要。數據不一致可以通過客戶端容錯（比如熔斷或踢出不可用服務器等），來把影響降到最低，甚至可以忽略不計。而可用性的下降將直接會導致服務的註冊和發現異常，甚至會引發大規模的生產故障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合以上問題，並考慮到 etcd 無法很好的接入攜程當時的運維和監控體系，我們走上了自研服務註冊中心的道路。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 攜程自研的服務註冊中心"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在設計這套自研的服務註冊中心時，我們參考了當時業界使用比較廣泛的由 Netflix 開源的 Eureka。新版註冊中心同樣沒有使用外部存儲，而是將服務的註冊數據保存在內存中。節點間採用對等的架構設計。所有節點都可以接受客戶端的讀寫請求。節點間會進行數據同步，實現數據最終一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在基本的服務註冊和發現功能外，爲了提升效率，我們還在其中增加了服務變更通知推送功能。這樣客戶端可以以最快的速度獲取到更新的服務註冊信息，目前已經實現了服務實例的秒級上下線。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將這套全新的服務註冊中心的開發代號起名爲 Artemis。爲了簡單，後文中均以該開發代號來進行指代。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、Artemis 架構說明"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1 總體架構"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7e\/7ea77460690a4e2db49054629703eb90.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 4-1 Artemis 架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Artemis 的整體架構與基於 etcd的服務註冊中心類似。如圖4-1 所示，一共包含四個角色："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供應用接入註冊中心基本API。應用通過引用Artemis 對外提供SDK，以編程方式實現服務註冊和發現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Session"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責接受 Client 的服務註冊和發現請求。Session作爲中間層將服務提供方的註冊請求複製分發給Data，並從Data上查詢服務註冊數據或推送數據變化給服務消費方。Session節點自身是無狀態的，集羣規模可隨着Client的規模增長而擴容，支持Artemis 服務能力的水平擴展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Data"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責存儲服務註冊數據，數據按ServiceId進行一致性哈希分片存儲，通過多副本備份保證數據的高可用。Data集羣規模可隨着註冊數據量增長而持續擴容，從而支持 Artemis 數據存儲容量的水平擴展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MetaServer"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責從K8s同步Artemis集羣服務器地址列表。在Artemis集羣發生變化時，MetaServer會實時通知到Session。Session在程序啓動或者收到Artemis 集羣變化通知時，將主動從MetaServer拉取最新的Artemis地址列表並緩存到本地。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2 如何支持海量數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統在處理海量數據時，首先是考慮如何拆分數據，其次是在數據拆後的如何保障系統的可用性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Artemis使用一致性哈希環來拆分數據。一致性哈希環的基本使用方式是通過一個哈希函數來計算數據或節點的哈希值，令該哈希函數的數據值域爲一個環，即哈希函數輸出的最大值是最小值的前序，節點依據其哈希函數計算結果分佈在環上，每個節點負責處理從自己開始逆時針至下一個節點全部哈希值域上的數據。Artemis使用服務註冊數據的ServiceId來計算哈希值，這樣可以保證同一個服務的註冊數據可以被存儲在相同的節點上，減少網絡調用的操作量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例：假設一致性哈希函數值域是[0, 8)，系統中有三個節點 A、B、C，分別處於一致性哈希環的2、5、6位置。由此可知，節點A的負責範圍爲 [7,8）和 [0,3)，節點B的負責範圍爲[3, 6)，節點C的負責範圍爲[6, 7)，如圖 4-2所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8e\/8ef8860bf66739f6507c90a691555697.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖4-2一致性哈希環"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用一致性哈希環拆分數據的優點在於可以任意動態擴容或縮容節點。對集羣進行擴容或縮容的操作，僅會影響與被操作節點相鄰的節點上的數據分佈。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但最基本的一致性哈希環用法存在一個很明顯的缺陷，那就是環上的節點分佈不均勻。由此所帶來另一個較爲嚴重的問題是，當一個節點出現異常時，該節點的壓力會全部轉移到相鄰的一個節點；而當一個新節點加入時，只能爲一個相鄰節點分攤壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一種常見的改進算法是引入虛擬節點（virtual node）的概念。系統在初始化時，每個真實節點都會對應的創建多個虛擬節點。虛擬節點的個數一般遠大於集羣中服務器的個數。依據虛擬節點的哈希值，系統將它們分佈到環上。在操作數據時，首先需要通過數據的哈希值在環上找到對應的虛擬節點，然後從元數據中查找到對應的真實節點，再進行數據讀寫操作。使用虛擬節點有多個好處。首先，一旦系統中某個節點出現不可用，其對應的所有虛擬節點也會同時變爲不可用，從而它的服務壓力會被均衡的分配到多個相鄰的真實節點上。同理，一旦系統中加入一個新節點，也將在環上引入多個虛擬節點，從而使得新節點可以均衡的分擔多個真實節點的壓力。從全局看，這種實現方式更加容易實現集羣擴容時的負載均衡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Artemis使用一致性哈希環加虛擬節點的方法，實現了海量數據的分片存儲和集羣擴縮容時的負載均衡。而在數據拆分後的集羣可用性方面，Artemis則是通過數據副本策略來保障的。每一條服務註冊數據同時被存儲在多個節點上，其中一個是主副本節點，其餘的是從副本節點。假如我們需要在集羣中選擇N個節點來存儲同一條數據，那麼在根據哈希函數計算出數據中ServiceID的哈希值後，系統會從哈希值落到環上的位置開始順時針依次選擇連續N個不同的真實節點來存儲這一份數據的各個副本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5e\/5e64cf3b020ec8935bbb3dab4987a64e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖4-3 一致性哈希環（6個虛擬節點，2個註冊數據副本）"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3 服務實例秒級上下線"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在設計服務註冊中心時，一個重要的考量指標就是服務實例的上下線延遲。這個延遲是指從服務註冊中心確定服務實例發生了上下線變更起，到服務消費方收到更新後的註冊數據的時間間隔。延遲越高，則服務流量切換所需時間就越長，涉及到流量切換的場景（故障轉移、服務發佈）給用戶帶來的體驗就越差。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了降低服務實例的上下線延遲，Artemis基於WebSocket實現了服務實例的上下線通知功能。通知可以秒級送達到服務消費方。這一功能的具體實現過程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務消費方在初始化過程中，會先經Session域名查詢Session的IP地址列表並緩存到本地，然後再從列表中選擇一臺Session服務器與之建立 WebSocket長連接，併發送服務訂閱請求。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Session在收到服務訂閱請求後，先會將服務訂閱信息和 WebSocket 連接的映射關係存儲到本地。後續當Session收到Data推送的服務變更消息時，它會先從上述映射關係中查詢該服務對應的變更訂閱方（即對應的WebSocket 連接列表），然後將消息通過這些連接推送出去。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在收到服務變更消息後，服務消費方會根據消息的內容更新本地緩存中的服務地址列表。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"4.3.1 服務實例上線過程"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d1\/d1c83b8e209e15089ae08e6091321661.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖4-4 服務實例上線過程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖 4-4所示，這是一次服務實例正常上線過程，其中包含了服務註冊數據在 Artemis內部的流轉過程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務提供方發送註冊數據給Session。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Session收到服務實例的註冊數據，依據ServiceID在環上查找到相應的Data節點列表，再將數據寫到對應的數個Data節點上。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Data在收到數據後，先將數據寫入本地緩存，然後推送服務實例上線消息給所有的Session節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Session在收到服務實例上線消息後，將消息推送給對應的服務消費方。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務消費方在收到服務實例上線消息後，將消息中所包含的服務地址加入到本地緩存中的服務地址列表，後續客戶端 SDK中的負載均衡模塊將分配部分流量給新上線的服務實例。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"4.3.2 服務實例下線過程"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ac\/acf4bc13e1f6a1d4be1b996b9fb6aab8.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖4-5服務實例正常下線過程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務實例下線分爲正常下線和異常下線兩種情況。正常下線並不會在服務消費方引起調用異常，而異常下線則可能會導致服務消費方出現短時間的調用異常。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務實例正常下線，一般是通過監聽應用程序關閉事件（如 JVM的 Shutdown Hook），主動觸發服務實例註銷操作，將服務實例從 Artemis 中刪除。服務下線大致過程與服務上線過程類似，這裏就不再贅述了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/09\/092dd43a393a2a0a15b7d29c144e25c5.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖4-6 服務異常下線過程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務實例異常下線，是指服務因意外情況（如宕機、網絡中斷或斷電等）而不可用，但沒有將註冊數據從Artemis中刪除。這些異常的註冊數據，依賴Artemis的健康檢測機制進行處理。類似於Eureka和etcd等系統中的數據過期機制，Artemis中的服務實例註冊數據以Lease（租約）的形式存在，需要服務提供方不斷髮送心跳來續約。同時 Artemis內部會運行一個異步線程來自動踢出到期的 Lease。異常下線的服務實例由於不會再繼續上報心跳，它的註冊數據在一段時間後（TTL）將自動被Artemis清理掉。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Artemis也支持用戶在客戶端自定義健康檢測邏輯，當應用程序不健康時，應用程序可以主動更新服務提供方的狀態或停止上報心跳。那麼服務提供方狀態又是如何被服務消費方感知到的呢？當服務提供方註冊數據修改後，服務註冊數據會生成一個新的版本號（單調遞增），並在下一次上報心跳時，發送給Artemis。Artemis在收到服務提供方的心跳後，會先檢查心跳中服務註冊數據的版本號。如果該版本號大於本地Lease中的服務註冊數據版本號，Artemis就會更新Lease中的服務註冊數據，並生成一條服務變化消息，逐級經Data、Session 推送給服務消費方。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、小結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文介紹了攜程的微服務註冊中心架構演進迭代過程，並重點介紹了當前版本服務註冊中心（Artemis）的架構，以及海量數據存儲和服務秒級上下線機制的實現。在攜程微服務架構演進的過程中，服務發現的主要方式從手工維護，逐步過渡到自注冊和自發現，解決了註冊地址的維護複雜性問題。負載均衡的主要實現方式也從外部設備的負載均衡，逐步過渡到在應用程序內嵌代理（SDK）的軟件負載均衡，解決了單點問題和性能問題，但同時也帶來了多語言、多版本SDK維護的複雜性問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在，攜程正在構建全新的 ServiceMesh 平臺，計劃以K8s替換Artemis來作爲服務的註冊中心，並通過 sidecar模式將服務發現、負載均衡以及一些切面功能（例如熔斷、限流、監控等）從SDK中剝離出來，使得這些功能可以獨立於用戶應用外進行更新升級。ServiceMesh 與雲原生技術的推廣，將極大的提升服務的治理效率，爲攜程的微服務開發者和使用者帶來更上一層樓的使用體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"參考文檔"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、"},{"type":"link","attrs":{"href":"https:\/\/github.com\/netflix\/eureka","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/netflix\/eureka"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、"},{"type":"link","attrs":{"href":"https:\/\/github.com\/sofastack\/sofa-registry","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/sofastack\/sofa-registry"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Alex，攜程資深軟件工程師，關注微服務架構及分佈式緩存技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：攜程技術（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/zyjsNs_cefbkVMfg5i2nJw","title":"xxx","type":null},"content":[{"type":"text","text":"乾貨 | 秒級上下線，攜程服務註冊中心架構演進"}]}]}]}