k8s極簡史：K8s多集羣技術發展的歷史、現狀與未來

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"引子"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲原生技術的普及，越來越多的企業使用Kubernetes來管理應用，並且集羣規模也呈爆發式增長，企業也亟需應對隨集羣規模增長而帶來的各種挑戰。同時，爲了更好地提供高可用、彈性伸縮的應用，企業也對容器混合雲解決方案產生了極大的興趣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"縱觀容器混合雲市場，主要的雲服務提供商紛紛推出了自家的解決方案，例如華爲雲的MCP、Google的Anthos、Vmware的 Tanzu、IBM的 Cloud Pak、Red Hat的ACM for K8s等等。"},{"type":"text","text":"可以說，當前容器混合雲市場紛繁嘈雜、百家爭鳴，儘管各廠商不遺餘力地鼓吹自家解決方案，但有個不爭的事實是在容器混合雲領域暫未出現領軍產品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"混合雲市場的亂象源於兩點，一是各廠商均嗅到了商機，發現了這一廣闊的藍海市場，急於在這場競爭中搶佔先機C位出道；二是開源界暫無統一的事實標準。根據歷史規律，後者是解決這一亂象的關鍵所在，正像Kubernetes終結容器編排領域的紛爭一模一樣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在開源領域，致力於混合雲領域的項目數量與廣闊的市場相比，顯得極不相稱。目前只有Rancher的Fleet、SAP公司力推的Gardener、以及Kubernetes社區的Kubefed。Fleet和Gardener要麼缺乏創新，要麼格局較低，難成大氣，"},{"type":"text","marks":[{"type":"strong"}],"text":"最有可能形成標準的便是被寄予厚望的、也是當前Kubernetes社區唯一的官方項目Kubefed。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"K8s多集羣歷史"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kubernetes社區早在2015年便發佈了集羣聯邦技術白皮書，併成立了“SIG-Federation”（後更名爲SIG-Multicluster）特別興趣小組致力於多集羣領域的研究，該興趣小組由華爲領銜，同時也吸引了包括Google、Redhat在內的一線大廠。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SIG-Federation於2016年正式推出官方項目Federation，並在此基礎上發展出了Kubefed項目，而且技術架構也發生了較大的變化，因此Federation項目常常被稱爲Federation V1，而Kubefed則被稱爲Federation V2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Federation V1架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一代架構中，引入了Federated API Server，用於增加集羣相關API，屏蔽集羣差異，統一請求入口，同時提供一個Cluster Controller用於管理多個集羣狀態、集羣級別對象創建，並且Service Controller用來實現跨集羣服務發現。整體架構如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8e6140b6aeccda41849eabaf25819cd6.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"V1架構兼容K8S原生API，從單集羣到多集羣演進可以變得很自然，但它也有幾個不得不面對的缺陷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• 集羣信息嵌入原生API的Annotation中（如下圖所示），會導致原生API體積膨脹而醜陋；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• 沒有集羣生命週期管理特有API，導致其生命週期管理能力無法擴展；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• 無法提供API對象版本控制，比如Deployment在K8S爲GA，但在Federation中可能仍是Beta；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/43/43146c2a131aba13b228949edfe5c324.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Federation V2架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在第二代架構中，利用CRD來提供獨立的API對象，新的API來封裝K8S原生對象，同時也可以方便的對新增API提供版本管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體架構如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48d5fae83c994d1c9a13db51e5ce3771.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨架構升級，Federation項目也更名爲Kubefed。在新的架構中，Kubefed提供兩種配置類型："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• Type configuration（類型配置）： 定義Kubefed接管的K8S的資源類型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• Cluster configuration（集羣配置）： 定義Kubefed接管的K8S集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在類型配置中有三個關鍵的概念，用於控制資源向拖管集羣分發策略："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• Templates（模版）：定義一個原生的K8S資源類型；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• Placement（安置）：定義資源將分發的集羣；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• Overrides（修正）：針對集羣自由修正資源；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個典型的Secret配置如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"apiVersion: types.kubefed.io/v1beta1\nkind: FederatedSecret\nmetadata:\n name: test-secret\n namespace: test-namespace\nspec:\n template:\n data:\n A: YWxhIG1hIGtvdGE=\n type: Opaque\n placement:\n clusters:\n - name: cluster2\n - name: cluster1\n overrides:\n - clusterName: cluster2\n clusterOverrides:\n - path: /data\n value:\n A: null"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述配置中，通過template指定原生資源屬性，通過placement指定資源將分發到cluster1 和 cluster2集羣，最後overrides指示了分發到cluster2集羣時，消除Secret的data信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"K8s多集羣現狀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"KubeFed的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kubernetes社區當前已將Federation (v1)項目關閉，着重發展Kubefed，但該項目尚停留在beta階段，社區開發幾乎停滯，作爲社區官方項目在該領域中的領導地位也在逐漸減弱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Kubefed項目最大的問題是使用了非Kubernetes原生API來管理用戶應用部署"},{"type":"text","text":"，用戶必須先改造既有的工作流程纔可遷移到Kubefed提供的API，這不僅擡高了使用門檻，而且Kubefed爲每種資源類型均提供了CRD API，種類繁多的API也增加了用戶的學習成本。某位社區致力於多集羣管理的架構師坦言：“Kubefed項目強制用戶使用非原生API，這個錯誤的決定很大程度上導致了它的發展不如預期。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，多集羣管理場景中，應用的多集羣分發與監控應該是最基本的訴求，而"},{"type":"text","marks":[{"type":"strong"}],"text":"Kubefed只完成了應用分發，對於應用的運行狀態缺乏監管。"},{"type":"text","text":"用戶使用Kubefed分發應用只能看到應用是否分發成功，對於應用運行狀態，用戶仍需要遍歷集羣分別獲取。對用戶使用造成了極大的不便。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"K8s多集羣管理標準化工作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前Kubernetes社區針對Kubefed相關問題已經進行了多次討論，目前"},{"type":"text","marks":[{"type":"strong"}],"text":"多集羣管理相關標準制定工作主要圍繞在跨集羣服務發現和工作負載配置管理"},{"type":"text","text":"，這兩塊也是實現多集羣應用管理最基礎的功能部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a.多集羣Service API"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在多集羣應用背景下，用戶已經習慣於將應用分發到多個集羣，但對於Service應用而言，集羣是個硬性障礙，運行於集羣中的工作負載無法高效地訪問其他集羣中暴露的服務。多集羣Service API旨在提供解決這個問題的標準，它主要包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）定義一組API支持跨集羣的Service服務發現和消費；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）集羣中應用跨集羣訪問Service行爲與本集羣一致；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該Service API提供ServiceExport對象表示單個集羣中需要暴露到多集羣的Service："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"// ServiceExport declares that the associated service should be exported to\n// other clusters.\ntype ServiceExport struct {\n metav1.TypeMeta `json:\",inline\"`\n // +optional\n metav1.ObjectMeta `json:\"metadata,omitempty\"`\n // +optional\n Status ServiceExportStatus `json:\"status,omitempty\"`\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個需要暴露給其他集羣的Service均對應一個ServiceExport對象。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，Service API還提供了ServiceImport對象，表示跨集羣的Service定義："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"// ServiceImport describes a service imported from clusters in a supercluster.\ntype ServiceImport struct {\n metav1.TypeMeta `json:\",inline\"`\n // +optional\n metav1.ObjectMeta `json:\"metadata,omitempty\"`\n // +optional\n Spec ServiceImportSpec `json:\"spec,omitempty\"`\n // +optional\n Status ServiceImportStatus `json:\"status,omitempty\"`\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該Service API 提案已被社區接納，該提案只定義了跨集羣Service的聲明方式，並沒有對其實現細節進行約束，可以想見，將來會有多種具體的解決方案被提出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b.多集羣工作負載模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於聯邦應用如何在多集羣中分發，SIG-Multicluster也在進行嘗試一種與現有Kubefed不同的處理思路。Kubefed當前從一系列FederatedXXX配置中剝離出Kubernetes原生應用分發到多集羣，而新的嘗試是提供一個通用的ManifestWork對象封裝所有的應用，如下API設計："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"// ManifestWork represents a manifests workload that hub wants to deploy on the managed cluster.\n\n// A manifest workload is defined as a set of kubernetes resources.\n\n// ManifestWork must be created in the cluster namespace on the hub, so that agent on the\n\n// corresponding managed cluster can access this resource and deploy on the managed\n\n// cluster.\n\ntype ManifestWork struct {\n\n metav1.TypeMeta `json:\",inline\"`\n\n metav1.ObjectMeta `json:\"metadata,omitempty\"`\n\n\n // Spec represents a desired configuration of work to be deployed on the managed cluster.\n\n Spec ManifestWorkSpec `json:\"spec\"`\n\n\n // Status represents the current status of work\n\n // +optional\n\n Status ManifestWorkStatus `json:\"status,omitempty\"`\n\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與Kubefed爲每種應用類型均提供一個FederatedXXX 類型相比，這種新型的工作負載API則顯得更加簡單和通用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未來展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"K8s多集羣技術是容器混合雲/多雲解決方案的核心技術領域，涉及到資源、應用、數據、流量多個層面，以及統一配置、註冊、可視化、自動彈性等多個功能領域。"},{"type":"text","text":"目前開源業界包括K8s社區的KubeFed項目、以及現有市面上的各種產品與解決方案都沒有能夠覆蓋完整的多集羣技術領域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"華爲雲MCP容器多雲平臺在K8s多集羣技術領域屬於較早也是實現較爲全面的產品，而同時華爲雲作爲KubeFed社區項目的發起者與領導者，將在未來致力於完善現有KubeFed的功能集，並且實現K8s多集羣技術的標準化。"},{"type":"text","text":"下圖描述了K8s多集羣技術的全景，目前華爲雲已經在KubeFed自身以及周邊關聯的多個技術領域開展了相關工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3a/3a9645d7dc8c8ec521b1e365d70071d1.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs?utm_source=infoq&utm_medium=blog-article&utm_campaign=blog-article","title":""},"content":[{"type":"text","text":"點擊關注，第一時間瞭解華爲雲新鮮技術~"}],"marks":[{"type":"strong"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}