Kubernetes 網絡通訊模型解析

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kubernetes 基礎架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/94506c23637b0cbe421db622e5377f1e.jpeg","alt":null,"title":"架構圖","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"border"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從架構圖看到，一次外部訪問請求，並不會直接請求到kubernetes中具體的某個集羣或者某一個資源模塊，是需要經過各模塊資源間的調度來完成的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d5/d55ef07e389ec98a94a19fb15840d25f.jpeg","alt":null,"title":"網絡","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"繼續剖析Kubernetes的網絡，如上圖我們可以發現Kubernetes存在3種網絡：節點網絡，Pod網絡，Service網絡，最後再加上Internet與Service之間的網絡總共4種。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是我們可以總結Kubernetes需要解決4種通信模式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    1.容器和容器之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    2.Pod和Pod之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    3.Pod和Service之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    4.Internet和Service之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"容器和容器之間的網絡"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每一個Pod管理着一個或多個container，而對於同一個Pod之間的容器之間共享一個網絡命名空間，它們之間可以直接通過localhost進行訪問通信，這裏以docker作爲容器爲例，容器間正好採用了container模式，額外創建啓動一個容器，而這個容器不會有自己的網卡以及配置IP地址，而是和一個指定的容器共享IP和端口；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9a/9a819eaa8c7556f6492b749a187166ee.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Kubernetes裏面每一個Pod都有這樣的容器，叫做Pause，它有獨立的命名空間，Pod會把新創建的docker通過—net=container的方式加入到Pause的網絡命名空間，但是這裏需要注意的是文件系統和進程列表等還是隔離的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b2/b260ecee4dd1ab7770c2f0cfcf48a464.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Pod和Pod之間通信"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"同一個Node之間的Pod"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對同一個Node下的Pod之間通信採用了Linux的虛擬網以太設備（以太網橋），即通過虛擬網以太設備將兩個Pod的網絡命名空間聯繫起來。大致流程是每一個Node都有一個root命名空間，每一個Pod通信時候都會通過以太網設備鏈接到root nets，然後root-nets再通過以太網設備轉發到目的Pod的命名空間；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cd/cdba333c680d026f88a4fac2dc67a8c1.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"非同一個Node的Pod與Pod之間通信方式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於同一個Node間的Pod的通信我們不難聯想到在不同的Node之間可以再借助一個類似網橋的設備存儲Node之間的Mac地址，如以下架構："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d3684bfd1a5ee54087b362d5b7685e93.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Node1中的Pod1通過以太網設備eth0把數據包發送到關聯到root命名空間的veth0上，然後數據包Node1接受到數據包後，查找當前路由找不到Node2的Pod1的Mac地址，則把數據包轉發中間網絡，當數據包達Node2上後被Node2的eth0設備進行處理，Node2的root nets把數據流到veth0裏，最後被流傳到Node2的Pod1。這一就完成了一次Node之間的Pod通信；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"具體通信實現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是針對如果進行上述的網絡配置，kubernetes本身沒有具體實現，而是通過開發CNI接口供開源社區實現。由於目前CoreOS的Flannel在kubernetes實現CNI，這裏針對Flannel的展開說明："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8d442f2a54d20c3594d077114805a888.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖：Node1中Pod1裏WebServer1--->Node2中Pod2裏的WebServer2"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Node1-Pod1-WebServer1 (10.12.13.5)發現請求不在同一個Ip地址，則通過訪問docker的統一eth0(10.12.13.5)將請求太網橋訪問到Root-net"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   1.Root-net在本地中路由表中找不到於是把請求數據包分裝給了flannel；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    2.Flannel獲取到數據包後通過查詢ETCD得到Web1請求的數據包的Mac地址和目的Ip地址；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    3.Flannel再次封裝數據包傳給Node1，並告知Node1目的Mac地址；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    4.Node1通過Mac地址RARP找到了Node2，並把數據包給Node2；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    5.Node2獲取數據包或解析拆包 得知flannel的地址，這個時候Flannel再拆包，併發給Pod2的root-net；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    6.Pod2的root-net通過以太網設備找到路由地址docker，然後把請求轉發給docker；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    7.最終docker獲取請求後，解析數據包得到請求的webServer2地址是10.22.13.5;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這一就完成了一次非Node之前的網絡通信請求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Pod與Service之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Pod銷燬或者新增等操作，導致Ip地址不是持久不變的。這個時候就需要kube-proxy-service來實現Pod對外訪問接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kube-proxy-service在邏輯上代表了後端多個Pod，在每一個Node中都會運行一個kube-proxy，集羣Service都會通過kube-proxy-service 來訪問到每一個Pod，而kube-proxy-service通過兩種方式負載均衡：IPVS和Iptables；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"IPVS："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於LVS的4層負載均衡也許大家都不陌生，而LVS的IP負載均衡正是通過IPVS內核模塊實現的，而IPVS的大致作用：安裝在Director Server上，同時在Director Server 虛擬出一個IP地址，用戶必須使用這個虛擬的IP地址訪問集羣服務。而在IPV中負載均衡依賴-netfilter框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"IP Tables ："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"區別於IPVS的IP tables是用戶態的應用，基於表來管理規則，在其內可以定義NAT，Filter，Mangle等規則參數。在Kubernetes中，kube-proxy的控制器來負責維護Ip tables ，該控制器的職責是監聽Ip tables的變化，並實時通知到API-service，保證數據通過Service能準確傳達到對應的Pod"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Service和Internet之間的通信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Ingress在絕大部分都用於Kubernetes在7層的接口對外暴露調用，這裏可以使得Ks的Service無需通過IP：Port的方法是提供訪問。至於爲什麼不使用Nginx，因爲在Kubernetes中Service變化十分頻繁，人爲的手動修改變得不太可能，因此ingress被運用到kubernetes中實現service與Internet之間的通信。以下Ingress進行展開簡單的舉例說明；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ingress相當於一個7層的負載均衡器，是對kubernetes反向代理的一個抽象。工作原理也確實類似於Nginx，可以理解成在 Ingress 裏建立一個個映射規則 , ingress Controller 通過監聽 Ingress這個api對象裏的配置規則並轉化成 Nginx 的配置（kubernetes聲明式API和控制循環） , 然後對外部提供服務。基於具體的Ingress工作原理和配置這裏不作更多複述。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9a/9aee466aa0d86015c1f11be937a2657b.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文參參考資料如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://kubernetes.io/zh/docs/home/","title":null},"content":[{"type":"text","text":"https://kubernetes.io/zh/docs/home/"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://kubernetes.io/zh/docs/concepts/services-networking/ingress/","title":null},"content":[{"type":"text","text":"https://kubernetes.io/zh/docs/concepts/services-networking/ingress/"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://en.wikipedia.org/wiki/Linux_Virtual_Server","title":null},"content":[{"type":"text","text":"https://en.wikipedia.org/wiki/Linux_Virtual_Server"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"騰訊雲崗位職位招聘，歡迎大家投遞：[email protected]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}