從字節跳動到火山引擎（二）：私有云 PaaS 場景下的 Kubernetes 集羣部署實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人們常說：沒有最好的架構，只有最合適的架構。對於 Kubernetes 集羣部署來說也是如此。在本文中，火山引擎雲原生研發工程師王敏傑將給大家介紹一種 Kubernetes 集羣部署的大致思路，希望可以給大家帶來一些參考。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kubernetes 集羣簡介","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Kubernetes 集羣組件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kubernetes 集羣的一些關鍵組件包括：API Server、Controller Manager、Scheduler、Kubelet、Kube-Proxy、Kubectl。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/92/9268d11ff77062034f57563814cc02c0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 1：Kubernetes 集羣組件示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考上圖左半部分，這裏有一個 Kubectl。以常規的 Deployment 創建過程來講：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Kubectl 以 Deployment 的 YAML 文件或命令行操作創建一個 Deployment。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Kubectl 會把請求發給 API Server。API Server 接收到請求之後，經過一定的驗證（驗證信息是從 Kubectl 的 kubeconfig 裏面提交上來的）會把這些信息會存入 etcd。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"etcd 存儲完 Deployment 的信息之後，Controller Manager 裏的 Deployment Controller 會 get 到這個信息，並創建對應的 ReplicaSet。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"Controller Manager 中的 ReplicaSet Controller watch 到這個信息之後，會創建對應的 Pod 資源。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"Scheduler 會對每個集羣的節點進行打分操作以選擇最合適的節點，並把這個節點的信息綁定到 Pod 資源上。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"這時候 node 節點上運行的 Kubelet 通過請求 API Server 會得到創建對應 Pod 的任務，Kubelet 會把 Pod 啓動需要的 volume 等依賴提前掛載起來。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"之後 Docker 或 Containerd 等 runtime 會去拉起對應的容器，這個流程相當於把一個 Deployment 真正創建起來了。","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kube-Proxy 這個組件主要負責當前節點上的網絡路由等配置，有兩種部署模式:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"iptables 模式","attrs":{}},{"type":"text","text":"：使用 iptables 分發的路由規則","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"IPVS 模式","attrs":{}},{"type":"text","text":"：使用內核的 IPVS 路由功能","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"兩種模式從功能上對 Kubernetes 集羣來說是大同小異的，當然我們更推薦使用 IPVS 的模式。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們從圖中可以看到左邊 Master 節點只有一個單節點。這時不管集羣是運行在虛擬機還是在物理機上，都會面臨服務器宕機的風險。爲了避免這種風險，我們可以使用下圖的拓撲結構。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/74/7440efbf070a52bb71925a4fe36b1715.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 2：Kubernetes 集羣 load balancer 拓撲","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這張圖我們從下往上看。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最下面是用三個節點的形式來運行一個集羣的 Master 節點，作爲一個集羣的控制面。我們會在每個 Master 節點上啓動 etcd 服務， ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"etcd 通過相互綁定，實現獨立的 etcd 集羣","attrs":{}},{"type":"text","text":"。在每個 Master 節點上會運行 API Server、Controller Manager、Scheduler 等組件，它們不會像 etcd 自己組成一個集羣，可以看作是各自獨立運行","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 API Server 之上有一個 load balancer，它是把多個 API Server 節點進行串聯的關鍵組件。load balancer 的功能類似於 Nginx，會把 worker node 發送的請求代理到各個 API Server 上，把流量/請求全部分發過去，這樣就可以實現 API Server 的 HA 功能","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個架構中 load balancer 也存在","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"單點","attrs":{}},{"type":"text","text":"的問題。對此有很多常規的處理方式，比如使用 keepalive 創建 VIP。我們採用的是分佈式 LB 集羣部署方式。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"分佈式 LB 集羣部署","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是部署架構方案：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/aa/aa2b21807ced62d4047e3b8460ffcc60.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 3：Kubernetes 集羣分佈式 LB 集羣拓撲","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"左邊有 3 個 Master 節點，都是用 Kubelet 拉起 Static Pod。API Server、Controller Manager、Scheduler、Kube-Proxy 都是用 Static Pod 運行的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比較關鍵的一點是在上圖右邊的 node 上。除了常規部署的 Kubelet 和 Kube-Proxy 這兩個組件之外，還會以 Static Pod 的形式運行一個 Nginx 服務，用於監聽本地 localhost:6443 端口。Nginx 服務使用反向代理的方式，在 upstream 中填寫所有 Master 節點 IP 和 6443 端口。這時 node 上的 Kubelet 服務在請求 API Server 時，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"其實請求的是本地的 6443 端口","attrs":{}},{"type":"text","text":"。再通過 Nginx 把流量/請求轉發到 Master 節點上，即實現了 Node 節點的請求。這樣就可以避免上述 load balancer 單點的問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"單 Kubernetes 集羣部署","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介紹完 Kubernetes 集羣組件，下面一起來看看單 Kubernetes 集羣部署。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲我們是私有云 PaasS 場景，可能會面臨如下問題：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"沒有公網","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不能方便地從公網拉取二進制文件或者鏡像","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自己構建的鏡像不希望在公網出現","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種情況下就需要在部署集羣之前預先部署一些依賴：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/22e2a385cfd28242100aacf68a84b98b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 4：單 Kubernetes 集羣部署方式示意圖","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，如上圖左邊所示，我們啓動了一個 HTTP 服務（可以是 Nginx 或 Apache 等服務）。它的作用可以理解成是一個文件服務器，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"裏面存放的是操作系統軟件源以及 Kubernetes 集羣的二進制文件","attrs":{}},{"type":"text","text":"（Kubelet、Kubectl 等）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，我們會安裝一個鏡像倉庫。集羣使用的","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"鏡像","attrs":{}},{"type":"text","text":"都存放在裏面，後續產品或業務組件更新迭代也會把鏡像推送到這邊來","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面兩個服務起來之後，我們會在這個節點或者服務器上啓動另外一個容器——","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"控制集羣部署腳本","attrs":{}},{"type":"text","text":"。這個容器裏的腳本是 Ansible playbook，會通過 SSH 的方式登錄到集羣的每個節點上進行部署操作","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上這些預置的操作都完成之後，就可以開始進行集羣部署了。我們的集羣部署工具是在 Kubeadm 基礎上進行的部署腳本開發。在集羣部署時，etcd、Kubelet、Containerd 等服務以二進制的方式運行，其他 Kubernetes 組件都以容器的方式運行。有些 Kubeadm 沒有支持配置的參數，我們會在部署腳本中使用 patch 的方式修改，以滿足對集羣的要求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/88/889547f3101f2d4897132d21a198e261.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 5：Kubernetes 集羣部署步驟","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖 5 展示了我們的部署步驟。從上往下看：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"操作系統跳板機配置","attrs":{}},{"type":"text","text":"（ssh-proxy）：這一塊放到後面講","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"操作系統初始化","attrs":{}},{"type":"text","text":"：修改或者更新操作系統的內核參數、依賴的安裝包等","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"部署前檢測和配置","attrs":{}},{"type":"text","text":"：集羣部署很多時候比較耗時，爲了避免部署過程中的一些風險，我們在集羣部署之前會進行一些配置或環境檢測。如果發現有不符合集羣部署的需求就會在這裏退出，在這個階段也會額外修改一些集羣對操作系統依賴的配置","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Runtime 安裝","attrs":{}},{"type":"text","text":"：我們採用的是 Containerd。如果需要使用 Docker 之類的 Runtime，可以在這個步驟中進行替換","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"二進制文件下載以及組件鏡像預下載","attrs":{}},{"type":"text","text":"：爲了加快後續的部署效率，我們會提前並行地在需要的節點下載一些二進制文件或組件鏡像","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"etcd 部署安裝","attrs":{}},{"type":"text","text":"：Kubeadm 支持 etcd 容器化運行。爲了適配更多的部署場景要求，etcd 服務使用二進制方式安裝","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"集羣節點二進制文件安裝","attrs":{}},{"type":"text","text":"：之後我們會在所有節點上安裝 Kubelet，Master 節點上也會安裝 Kubectl 以方便後續維護。Kubelet 安裝完成之後，第一個 Master 節點會用 Kubeadm init 操作來進行安裝，Master2、Master3 則會用 Kubeadm join --control-plane 的方式來進行安裝，這樣就能完成 Master 節點的部署。在 node 節點上我們會用 Kubeadm join 方式把集羣節點加進來","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"節點創建完成後，都還處於 not ready 的狀態，這時需要","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"安裝網絡組件","attrs":{}},{"type":"text","text":"。我們採用的是 Calico + VXLAN 的方式。Calico 安裝完之後集羣的所有節點都會進入 ready 狀態","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集羣部署完成之後，爲了方便之後的維護，以及維護過程中可以瞭解部署時的特殊配置，我們會把集羣信息全部存放到集羣的 ConfigMap 中","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是單 Kubernetes 集羣的部署方式。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"多 Kubernetes 集羣部署模式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面提到過我們是私有云 PaaS 環境，這種環境有如下一些特點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很多情況下，私有云都存在於自建機房。自建機房很可能是純內網的環境（如果有公網肯定是最好的）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具有不斷擴展的業務需求，具有可規劃性：可以不斷地往集羣內添加機器，能夠支持規模的擴大","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"沒有大規模的突發業務流量","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具有多個可以劃分的業務線，或有需要隔離的業務。當有一些業務需要單獨進行隔離時，如果這些業務運行在單集羣中，從網絡等維度上來說可能無法做到真正隔離。但是如果讓不同的業務運行在不同的 Kubernetes 集羣中，就可以做到完全隔離","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用多 Kubernetes 集羣也會給運維帶來一些挑戰，主要包括多集羣的配置管理、多集羣的統一維護入口、多集羣中應用組件的統一管理。這些都是要在集羣部署方案設計時要考慮的內容。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來就給大家介紹我們設計的多集羣部署方案。該方案有如下幾個特點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"所有集羣配置都會存放在控制集羣中","attrs":{}},{"type":"text","text":"。用戶集羣中也會存放一些相應的配置，但是它只存放集羣自身相關的配置。在控制集羣中可以看到所有用戶集羣的具體配置","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"控制集羣與用戶集羣之間在部署時僅由一臺服務器連接","attrs":{}},{"type":"text","text":"。如果機房存在於不同的 IDC，網絡打通會有一些困難；如果用戶集羣規模比較大時，需要開通很多訪問端口權限，工作量會很大。這個設計方案是隻要有一臺服務器能允許我們正常登陸上去，集羣就可以正常的部署","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"用戶集羣在添加刪除節點的過程中不依賴於控制集羣信息","attrs":{}},{"type":"text","text":"，在集羣內部就可以完成相應的操作","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6e/6ebcf742ae8ac3ca1c529fa9ca53e8d3.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 6：多 Kubernetes 集羣部署架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們的部署方案示意圖。圖中相同名稱的組件是使用相同的鏡像進行啓動的，通過不同的啓動參數來滿足不同的需求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從左上開始看。左上角有一個部署節點，它是前面提到的安裝了軟件源、鏡像倉庫服務的節點。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個節點上我們有兩個配置文件：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"env","attrs":{}},{"type":"text","text":" 和 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"inventory","attrs":{}},{"type":"text","text":"。大家如果熟悉 Ansible 可能會比較清楚， env 存放控制集羣配置， inventory 存放控制集羣節點。在這個節點上，我們會用 Containerd 或 Docker 等 Runtime 把容器運行起來，然後把 env 和 inventory 這兩個配置文件掛載到容器中，在裏面調用對應的 Ansible 腳本進行連接。腳本會連接 master1 節點，同時使用 ssh-proxy 的方式連接其餘的服務器節點，這就是圖 5 中的第一個階段。在 Deploy node 上的容器中，通過修改 .ssh/ssh-config 的配置文件，實現的這種跳轉方式。跳轉之後，雖然在 Ansible 的部署日誌中我們看到的是直接連接了集羣所有節點，但實際的 SSH 連接鏈路是通過 Master1 節點進行跳轉。當然如果使用非集羣節點作爲跳板機，只要能夠使用 SSH 登陸也是可以的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Master1 或者控制集羣上，每個節點都有一個 Kubeadm 和對應的 Runtime。通過這些工具，我們會把整個集羣啓動起來。之後會安裝 Cluster Component 和 Cluster Controller 等組件，用於支撐用戶集羣或當前控制集羣節點添加的工作。Cluster Controller 主要負責觸發當前集羣的擴縮容以及用戶集羣創建等任務。如果需要控制集羣新增一個節點，Cluster Controller 會創建一個 Cluster Job 的腳本任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當我們要部署用戶集羣的時候，也是由控制集羣的 Cluster Controller 組件創建 Cluster Job 進行集羣部署。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶集羣和控制集羣複用了一套部署安裝的邏輯。不同的地方在於：用戶集羣的 Cluster Controller 的作用只是進行當前集羣節點的增刪操作，不關心集羣的其他操作。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結展望","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次分享主要向大家介紹了 Kubernetes 集羣組件、單 Kubernetes 集羣部署方式以及多 kubernetes 集羣的部署邏輯。如果大家有使用多 Kubernetes 集羣的需求，可以考慮本文介紹的部署方式，實際的業務其實都是運行在用戶集羣中，控制集羣控制的是整個平臺的集羣，這樣可以帶來如下收益：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一定程度降低運維人員維護集羣的工作量，不會因爲集羣規模增加帶來巨大的工作量。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不會受制於單 Kubernetes 集羣的規模限制，可以支持更大的業務發展。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用多 Kubernetes 集羣模式可以很方便的對不同業務進行隔離。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後說一下個人的一些觀點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"業務組件容器化","attrs":{}},{"type":"text","text":"：如果後面很多業務都要上 Kubernetes，業務組件的容器化將會是必須要做的。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Kubernetes 集羣部署簡單化","attrs":{}},{"type":"text","text":"：我剛開始接觸 Kubernetes 集羣的時候，大多數操作都是手動的，API Server、 Controller Manager、Scheduler 等組件沒有以容器方式運行，很多都是用 systemd 的形式部署，非常耗時。現在社區有很多集羣部署工具，像 Kind、Minikube 對於單集羣或測試集羣的部署非常方便。Kubespray、Kops 等工具是成型的 HA 的 Kubernetes 集羣部署方案。Kubeadm 是一個十分好用的部署工具，它把部署過程拆成了很多小的步驟，只需一條命令就可以完成一個部署操作，非常靈活方便。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Kubernetes 集羣配置個性化","attrs":{}},{"type":"text","text":"：當不同的業務上到 Kubernetes 集羣或私有云以後，會對集羣有一些特殊要求。這種情況下就需要對每個 Kubernetes 集羣都有一些個性化的支撐。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"多 Kubernetes 集羣普遍化","attrs":{}},{"type":"text","text":"：多 Kubernetes 集羣更利於集羣的使用和運維，我認爲未來將更加普遍。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Q1：如何解決混合雲情況下非 K8s 集羣和 K8s 集羣之間的 Dubbo 依賴問題？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：常規私有云場景下，可以採用 Calico 的 BGP 網絡，這個網絡可能會對機房網絡有一定的要求，因爲它需要依賴於一些物理的路由規則的分發。另外一種方式就是使用 IPVLAN 作爲 CNI 的方案，這個方式可能會更適用於 Dubbo 環境，只是對集羣的規模會有一定的限制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Q2：Nginx Static Pod 做 LB， VIP 和公網 IP 怎麼分配？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：我們把它設計成分佈式 LB 的方式，最主要的目的就是讓集羣自身不會依賴於一個固定 LB 的 IP。如果集羣外部要對它有一個請求，可以額外創建一個 LB 的服務來進行請求。這樣即使 LB 掛掉了或者要進行切換，也不會影響集羣穩定性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Q3：CNI 之間的選擇主要考慮到哪些方面？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：性能是一方面，此外實際的使用環境以及通用性也是需要考慮的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Q4：爲什麼使用 VXLAN 模式而不是 BGP？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：如果只是單純的使用，VXLAN 和 BGP 可能不會有很大的差異。使用 BGP 更多的是爲了把路由信息通過網絡設備建立連接，這個過程對物理網絡有一些依賴。我們爲了有更好的兼容性，還是採用了 VXLAN 模式。當然實際部署過程中，我們也是支持通過修改 env 文件的配置，使用 BGP 進行部署。","attrs":{}}]}]}