gRPC服務註冊發現及負載均衡的實現方案與源碼解析

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天聊一下gRPC的服務發現和負載均衡原理相關的話題，不同於"},{"type":"codeinline","content":[{"type":"text","text":"Nginx"}]},{"type":"text","text":"、"},{"type":"codeinline","content":[{"type":"text","text":"Lvs"}]},{"type":"text","text":"或者"},{"type":"codeinline","content":[{"type":"text","text":"F5"}]},{"type":"text","text":"這些服務端的負載均衡策略，gRPC採用的是客戶端實現的負載均衡。什麼意思呢，對於使用服務端負載均衡的系統，客戶端會首先訪問負載均衡的域名/IP，再由負載均衡按照策略分發請求到後端具體某個服務節點上。而對於客戶端的負載均衡則是，客戶端從可用的後端服務節點列表中根據自己的負載均衡策略選擇一個節點直連後端服務器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"軟件包的"},{"type":"codeinline","content":[{"type":"text","text":"naming"}]},{"type":"text","text":"組件裏提供了一個命名解析器（naming resolver）結合"},{"type":"codeinline","content":[{"type":"text","text":"gRPC"}]},{"type":"text","text":"本身自帶的"},{"type":"codeinline","content":[{"type":"text","text":"RoundRobin"}]},{"type":"text","text":" 輪詢調度負載均衡器，讓使用者能方便地搭建起一套服務註冊/發現和負載均衡體系。如果輪詢調度滿足不了調度需求或者不想使用"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"作爲服務的註冊中心和命名解析器的話，可以通過寫代碼實現"},{"type":"codeinline","content":[{"type":"text","text":"gRPC"}]},{"type":"text","text":"定義的"},{"type":"codeinline","content":[{"type":"text","text":"Resolver"}]},{"type":"text","text":"和"},{"type":"codeinline","content":[{"type":"text","text":"Balancer"}]},{"type":"text","text":"接口來滿足系統的自定義需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文引用的源碼對應的版本爲：gRPC v1.2.x、 Etcd v3.3"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你對gRPC和Etcd還不瞭解，可以先看看我很早之前寫的"},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&albumid=1358237826197962753&_biz=MzUzNTY5MzU2MA==#wechat_redirect","title":""},"content":[{"type":"text","text":"gRPC入門"}]},{"type":"text","text":"和"},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/mp/appmsgalbum?action=getalbum&albumid=1574539663539781634&_biz=MzUzNTY5MzU2MA==#wechat_redirect","title":""},"content":[{"type":"text","text":"Etcd入門 "}]},{"type":"text","text":"系列的文章。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"gRPC服務註冊發現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先來簡單的說明一下用"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"實現服務註冊和發現的原理。服務註冊和發現這個流程可以用下面這個示意圖簡單描述出來："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8c/8cd31f8b1ed5d04561522e4ec4e59f27.jpeg","alt":null,"title":"gRPC使用Etcd實現服務發現","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖的服務A包含了兩個節點，服務在節點上啓動後，會以包含服務名加節點IP的唯一標識作爲Key（比如/service/a/114.128.45.117），服務節點IP和端口信息作爲值存儲到"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"上。這些Key都是帶租約的Key，需要我們的服務自己去定期續租，一旦服務節點本身宕掉，比如node2上的服務宕掉，無法完成續租後，那麼它對應的Key：/service/a/114.128.45.117 就會過期，客戶端也就無法再從Etcd上獲取到這個服務節點的信息了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與此同時客戶端也會利用"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"的"},{"type":"codeinline","content":[{"type":"text","text":"Watch"}]},{"type":"text","text":"功能監聽以"},{"type":"codeinline","content":[{"type":"text","text":"/servive/a"}]},{"type":"text","text":"爲前綴的所有Key的變化，如果有新增或者刪除節點Key的事件發生"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"都會通過"},{"type":"codeinline","content":[{"type":"text","text":"WatchChan"}]},{"type":"text","text":"發送給客戶端，"},{"type":"codeinline","content":[{"type":"text","text":"WatchChan"}]},{"type":"text","text":"在編程語言上的實現就是"},{"type":"codeinline","content":[{"type":"text","text":"Go"}]},{"type":"text","text":"的"},{"type":"codeinline","content":[{"type":"text","text":"Channel"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"服務註冊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"的服務註冊，官方提供的軟件包裏並沒有提供統一的註冊函數供調用。那麼我們在新增服務節點後怎麼把節點的信息存儲到"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"上並通知給命名解析器呢？在Etcd源碼包的naming/grpc.go裏可以發現提供了一個"},{"type":"codeinline","content":[{"type":"text","text":"Update"}]},{"type":"text","text":"方法，這個"},{"type":"codeinline","content":[{"type":"text","text":"Update"}]},{"type":"text","text":"既能執行添加也能執行刪除操作："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"func (gr *GRPCResolver) Update(ctx context.Context, target string, nm naming.Update, opts ...etcd.OpOption) (err error) {\n\tswitch nm.Op {\n\tcase naming.Add:\n\t\tvar v []byte\n\t\tif v, err = json.Marshal(nm); err != nil {\n\t\t\treturn status.Error(codes.InvalidArgument, err.Error())\n\t\t}\n\t\t_, err = gr.Client.KV.Put(ctx, target+\"/\"+nm.Addr, string(v), opts...)\n\tcase naming.Delete:\n\t\t_, err = gr.Client.Delete(ctx, target+\"/\"+nm.Addr, opts...)\n\tdefault:\n\t\treturn status.Error(codes.InvalidArgument, \"naming: bad naming op\")\n\t}\n\treturn err\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務在啓動完成後可以通過"},{"type":"codeinline","content":[{"type":"text","text":"Update"}]},{"type":"text","text":"方法把自己的服務地址和端口"},{"type":"codeinline","content":[{"type":"text","text":"Put"}]},{"type":"text","text":"到自定義的target爲前綴的key裏，針對上面圖示裏的例子，變量target就應該是我們定義的服務名/service/a。一般在具體實踐裏都是自己根據系統的需求封裝"},{"type":"codeinline","content":[{"type":"text","text":"Update"}]},{"type":"text","text":"方法完成服務註冊，以及服務節點Key在Etcd上的定期續租，這塊每個公司的實踐都不一樣，我就不放具體的代碼了，一般續租都是通過"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"租約裏的"},{"type":"codeinline","content":[{"type":"text","text":"KeepAlive"}]},{"type":"text","text":"方法實現的（Lease.KeepAlive）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"服務發現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在註冊完新節點、或者是原來的節點停掉後，客戶端是怎麼知道的呢？這塊就需要命名解析器Resolver來幫助實現了，Resolver的作用可以理解爲從一個字符串映射到一組IP端口等信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"gRPC對Resolver的接口定義如下："}]},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"type Resolver interface {\n\t// Resolve creates a Watcher for target.\n\tResolve(target string) (Watcher, error)\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"命名解析器的Resolve方法會返回一個Watcher，這個Watcher可以監聽命名解析器發來的target（類似上面例子裏說的與服務名相對應的Key）對應的後端服務器地址信息變化，通知Balancer對自己維護的地址進行動態地增刪。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Watcher接口的定義如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"//源碼地址 https://github.com/grpc/grpc-go/blob/v1.2.x/naming/naming.go\ntype Watcher interface {\n\tNext() ([]*Update, error)\n\t// Close closes the Watcher.\n\tClose()\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Etcd爲這兩個接口都提供了實現："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"// 源碼地址：https://github.com/etcd-io/etcd/blob/release-3.3/clientv3/naming/grpc.go\n\n// GRPCResolver 實現了grpc的naming.Resolver接口\ntype GRPCResolver struct {\n\t// Client is an initialized etcd client.\n\tClient *etcd.Client\n}\n\nfunc (gr *GRPCResolver) Resolve(target string) (naming.Watcher, error) {\n\tctx, cancel := context.WithCancel(context.Background())\n\tw := &gRPCWatcher{c: gr.Client, target: target + \"/\", ctx: ctx, cancel: cancel}\n\treturn w, nil\n}\n\n// 實現了grpc的naming.Watcher接口\ntype gRPCWatcher struct {\n\tc *etcd.Client\n\ttarget string\n\tctx context.Context\n\tcancel context.CancelFunc\n\twch etcd.WatchChan\n\terr error\n}\n\nfunc (gw *gRPCWatcher) Next() ([]*naming.Update, error) {\n\tif gw.wch == nil {\n\t\t// first Next() returns all addresses\n\t\treturn gw.firstNext()\n\t}\n\n\t// process new events on target/*\n\twr, ok := 1時，client是不會阻塞的。\n if cnt == 1 && rr.waitCh != nil {\n close(rr.waitCh)\n rr.waitCh = nil\n }\n //返回禁用該地址的方法\n return func(err error) {\n rr.down(addr, err)\n }\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"關閉連接"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關閉連接使用的是Down方法，這個方法就簡單, 直接找到addr置爲不可用就行了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"func (rr *roundRobin) down(addr Address, err error) {\n rr.mu.Lock()\n defer rr.mu.Unlock()\n for _, a := range rr.addrs {\n if addr == a.addr {\n a.connected = false\n break\n }\n }\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"客戶端獲取連接"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端在調用"},{"type":"codeinline","content":[{"type":"text","text":"gRPC"}]},{"type":"text","text":"具體"},{"type":"codeinline","content":[{"type":"text","text":"Method"}]},{"type":"text","text":"的"},{"type":"codeinline","content":[{"type":"text","text":"Invoke"}]},{"type":"text","text":"方法裏，會去"},{"type":"codeinline","content":[{"type":"text","text":"RoundRobin"}]},{"type":"text","text":"的連接池addrs裏獲取連接，如果addrs爲空，或者addrs裏的地址都不可用，"},{"type":"codeinline","content":[{"type":"text","text":"Get()"}]},{"type":"text","text":"方法會返回錯誤。但是如果設置了"},{"type":"codeinline","content":[{"type":"text","text":"failfast = false"}]},{"type":"text","text":"，"},{"type":"codeinline","content":[{"type":"text","text":"Get()"}]},{"type":"text","text":"方法會阻塞在"},{"type":"codeinline","content":[{"type":"text","text":"waitCh"}]},{"type":"text","text":"這個通道上，直至"},{"type":"codeinline","content":[{"type":"text","text":"Up"}]},{"type":"text","text":"方法給到通知，然後輪詢調度可用的地址。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"func (rr *roundRobin) Get(ctx context.Context, opts BalancerGetOptions) (addr Address, put func(), err error) {\n var ch chan struct{}\n rr.mu.Lock()\n if rr.done {\n rr.mu.Unlock()\n err = ErrClientConnClosing\n return\n }\n \n if len(rr.addrs) > 0 {\n // addrs的長度可能變化，如果next值超出了，就置爲0，從頭開始調度。\n if rr.next >= len(rr.addrs) {\n rr.next = 0\n }\n next := rr.next\n //遍歷整個addrs數組，直到選出一個可用的地址\n for {\n a := rr.addrs[next]\n // next值加一，當然是循環的，到len(addrs)後，變爲0\n next = (next + 1) % len(rr.addrs)\n if a.connected {\n addr = a.addr\n rr.next = next\n rr.mu.Unlock()\n return\n }\n if next == rr.next {\n // 遍歷完一圈了，還沒找到，走下面邏輯\n break\n }\n }\n }\n if !opts.BlockingWait { //如果是非阻塞模式，如果沒有可用地址，那麼報錯\n if len(rr.addrs) == 0 {\n rr.mu.Unlock()\n err = status.Errorf(codes.Unavailable, \"there is no address available\")\n return\n }\n // Returns the next addr on rr.addrs for failfast RPCs.\n addr = rr.addrs[rr.next].addr\n rr.next++\n rr.mu.Unlock()\n return\n }\n // Wait on rr.waitCh for non-failfast RPCs.\n // 如果是阻塞模式，那麼需要阻塞在waitCh上，直到Up方法給通知\n if rr.waitCh == nil {\n ch = make(chan struct{})\n rr.waitCh = ch\n } else {\n ch = rr.waitCh\n }\n rr.mu.Unlock()\n for {\n select {\n case 0 {\n if rr.next >= len(rr.addrs) {\n rr.next = 0\n }\n next := rr.next\n for {\n a := rr.addrs[next]\n next = (next + 1) % len(rr.addrs)\n if a.connected {\n addr = a.addr\n rr.next = next\n rr.mu.Unlock()\n return\n }\n if next == rr.next {\n // 遍歷完一圈了，還沒找到，可能剛Up的地址被down掉了，重新等待。\n break\n }\n }\n }\n // The newly added addr got removed by Down() again.\n if rr.waitCh == nil {\n ch = make(chan struct{})\n rr.waitCh = ch\n } else {\n ch = rr.waitCh\n }\n rr.mu.Unlock()\n }\n }\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個"},{"type":"codeinline","content":[{"type":"text","text":"gRPC"}]},{"type":"text","text":"基於"},{"type":"codeinline","content":[{"type":"text","text":"Etcd"}]},{"type":"text","text":"實現服務註冊/發現以及負載均衡的流程和關鍵的源碼實現就梳理完了，其實源碼實現的細節遠比我這裏列舉的要複雜，這篇文章的目的也是希望能記錄下一學習和實踐gRPC的負載均衡和服務解析時的一些關鍵路徑。另外需要注意的是本文裏使用的是gRPC v1.2.x的代碼，在1.3版本後官方包重新調整了目錄和包名，與本文裏列舉的源碼以及Balancer的使用上都會有些出入，不過原理還是大致一樣的，只不過每一版都一直在此基礎上演進。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看到這裏了，如果喜歡我的文章可以幫我點個贊，我會每週通過技術文章分享我的所學所見和第一手實踐經驗，感謝你的支持。微信搜索關注公衆號「網管叨bi叨」第一時間獲取我的文章推送。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}