KubeSphere 多集羣管理大招：使用 QKE 管理多個 ACK 集羣

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各位社區的小夥伴，大家好。我是紅亞科技的 CTO 盧興民，今天給大家分享一下我們公司的團隊在使用 KubeSphere 進行多集羣管理，還有應用發佈上的實踐工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"基於 KubeSphere 部署青椒課堂 —— 業務架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/06\/06a369ac014585dce35d541af6194601.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先介紹一下我們應用的架構。我們獨立出來的這一部分服務，主要是用來爲青椒課堂的學生和老師提供虛擬化的操作環境，是一個虛擬化資源調度服務。它包含了 API server、proxy、jobrunner，還有 image registry mirror。它本身有調度容器和虛擬機的能力，而調度虛擬機是直接跟 IAAS層（比如阿里雲）去做的對接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最初我們的副本只在北京有一個集羣，所以也就沒有必要考慮用一個更方便的形式去部署多個集羣。當時就是一個可視化的 PAAS 平臺，人工操作即可。但是隨着我們用戶的數量變多，服務遍佈全國，我們就必須做如圖所示的這種多可用區的架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"青椒課堂 region 服務部署在 KubeSphere 上，應用特點如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我們把服務部署在多個地域，比如北京、廣東、上海"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"但是各個副本之間不需要進行通訊，只是簡單的一個多副本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"直接受主業務的調度，就是提供一個 API server 被主業務去調度。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"當業務需要變更的時候，比如增刪組件，或者組件配置的變更，我們需要在多個副本里面同時進行。這個時候就我們用可視化的 PAAS 平臺就會有一個問題，我們需要人工在各個地域進行手工調整。當有幾個的時候我們還能接受，但當有十幾個的時候，就很難接受這樣的手工調整的過程。並且人工操作會有很大概率出現問題，例如組件變更在不同的可用區內沒有做到同步調整，導致業務出現故障。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"希望達成的目標"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/49\/49d8dc422e4f19dec2006516d66c18c2.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們做多集羣的管理，想達成的目標如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我們希望有一個開箱即用的 KubeSphere 服務。在 KubeSphere 3.0 發佈之後，我進行了很多測試，KubeSphere 提供了很好的安裝方式，包括 KubeKey 這些形式，但我還是認爲過於麻煩了。因爲我們只需要一個可用的生產級別的控制檯，從而不需要投入太多的人工去部署。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我們不希望遷移原有的業務。因爲我們原有的服務有一些依賴阿里雲，所以我們希望Member集羣都維持在原有的阿里雲的集羣不變。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"操作簡單一些。在納管 member 集羣的過程中，不需要打通 VPC 等之類的操作，降低部署複雜度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我們選擇了 KubeSphere，滿足我們所有的需求。而且在上線 QKE 3.0 之後，我們幾乎可以通過點幾下鼠標，在分鐘級的時間內創建出來生產級別的 KubeSphere 集羣、Host 集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用現狀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/dc\/dcf648a547b58d08e9f237e61a6141f9.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個是我們目前的使用狀況。目前我們在 KubeSphere 上接入了三套阿里雲 ACK 集羣，進行統一納管。目前還有一些機器沒有遷移過來，我們在逐步的進行遷移工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"應用打包與規範定義"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e0\/e067b61f8eb85ee09cb62d19ae4acbc7.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們原有的應用是在一個可視化 PAAS 平臺，所以我們沒有使用 kubernetes 原生的這種包管理工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼我們在做應用打包的時候爲什麼會選擇 helm 呢？主要是基於以下幾點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"helm 確實是一個應用非常廣泛的 kubernetes 包管理工具。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我們對 KubeSphere 3.0 中的多集羣應用非常感興趣，應用市場也是支持 helm 的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"我們需要將應用的變更，應用定義的過程進行版本化。用 helm 這種打包的方式就可以很好實現。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"應用部署"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/03\/035ecb9e3782d76a68d2848c9ba70173.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在應用部署上我們嘗試了幾個方案，最終選擇了 Spinnaker 作爲我們的應用發佈的方案，主要是基於以下幾點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"製品綁定的功能非常有用，我們可以在不修改 helm 包內部配置的情況下，跟製品庫形成聯動，直接去對每一個組件的 image 進行選擇和切換，這在發佈的過程中非常方便。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我們可以將多個集羣的部署定義在一個流程內，增加前後依賴關係、人工確認等流程，這使我們的配置不再繁瑣。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Spinnaker 支持將集羣差異化的配置放到同一個流程裏。當然 KubeSphere 多集羣應用也能實現這樣的功能。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/eb\/eb14b15d1b7b351035543cdb7b93537c.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，這是我們做好了製品綁定之後達到的效果。通過修改發佈流程的啓動參數，每一個組件在部署的時候都可以去選擇它所對需要部署的鏡像版本，以及用到的 helm 版本和所要發佈的集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d3\/d3c9a7f99035404c8031916b547496e7.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我們整個應用定義的過程，首先是確定應用部署所使用的 helm 包，選取部署過程中需要使用的鏡像。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右上方的第二張圖的 Bake 階段可以用來做不同集羣之間的配置、差異化的管理。比如說我們在生產環境加載 values-prod.yaml 配置文件，但是每一個集羣之間會有一定的差異化配置，都可以在 Bake 階段使用 values 覆蓋去進行配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Bake 階段相當於是 helm 包生成一組部署到 kubernetes 集羣裏面的 yaml 文件，然後在 deploy 階段進行製品綁定之後，就可以將 image 字段自動進行一個替換，部署你想要的鏡像版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"頭圖：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者：盧興民"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：https:\/\/mp.weixin.qq.com\/s\/CoufSH5LTQbob4scTtL05A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：KubeSphere 多集羣管理大招：使用 QKE 管理多個 ACK 集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來源：KubeSphere雲原生 - 微信公衆號 [ID：gh_f5606464ef2f]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"轉載：著作權歸作者所有。商業轉載請聯繫作者獲得授權，非商業轉載請註明出處。"}]}]}