KubeSphere 多集群管理大招：使用 QKE 管理多个 ACK 集群

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各位社区的小伙伴，大家好。我是红亚科技的 CTO 卢兴民，今天给大家分享一下我们公司的团队在使用 KubeSphere 进行多集群管理，还有应用发布上的实践工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"基于 KubeSphere 部署青椒课堂 —— 业务架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/06\/06a369ac014585dce35d541af6194601.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先介绍一下我们应用的架构。我们独立出来的这一部分服务，主要是用来为青椒课堂的学生和老师提供虚拟化的操作环境，是一个虚拟化资源调度服务。它包含了 API server、proxy、jobrunner，还有 image registry mirror。它本身有调度容器和虚拟机的能力，而调度虚拟机是直接跟 IAAS层（比如阿里云）去做的对接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最初我们的副本只在北京有一个集群，所以也就没有必要考虑用一个更方便的形式去部署多个集群。当时就是一个可视化的 PAAS 平台，人工操作即可。但是随着我们用户的数量变多，服务遍布全国，我们就必须做如图所示的这种多可用区的架构。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"青椒课堂 region 服务部署在 KubeSphere 上，应用特点如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我们把服务部署在多个地域，比如北京、广东、上海"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"但是各个副本之间不需要进行通讯，只是简单的一个多副本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"直接受主业务的调度，就是提供一个 API server 被主业务去调度。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"当业务需要变更的时候，比如增删组件，或者组件配置的变更，我们需要在多个副本里面同时进行。这个时候就我们用可视化的 PAAS 平台就会有一个问题，我们需要人工在各个地域进行手工调整。当有几个的时候我们还能接受，但当有十几个的时候，就很难接受这样的手工调整的过程。并且人工操作会有很大概率出现问题，例如组件变更在不同的可用区内没有做到同步调整，导致业务出现故障。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"希望达成的目标"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/49\/49d8dc422e4f19dec2006516d66c18c2.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们做多集群的管理，想达成的目标如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"我们希望有一个开箱即用的 KubeSphere 服务。在 KubeSphere 3.0 发布之后，我进行了很多测试，KubeSphere 提供了很好的安装方式，包括 KubeKey 这些形式，但我还是认为过于麻烦了。因为我们只需要一个可用的生产级别的控制台，从而不需要投入太多的人工去部署。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我们不希望迁移原有的业务。因为我们原有的服务有一些依赖阿里云，所以我们希望Member集群都维持在原有的阿里云的集群不变。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"操作简单一些。在纳管 member 集群的过程中，不需要打通 VPC 等之类的操作，降低部署复杂度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我们选择了 KubeSphere，满足我们所有的需求。而且在上线 QKE 3.0 之后，我们几乎可以通过点几下鼠标，在分钟级的时间内创建出来生产级别的 KubeSphere 集群、Host 集群。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用现状"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/dc\/dcf648a547b58d08e9f237e61a6141f9.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个是我们目前的使用状况。目前我们在 KubeSphere 上接入了三套阿里云 ACK 集群，进行统一纳管。目前还有一些机器没有迁移过来，我们在逐步的进行迁移工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"应用打包与规范定义"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e0\/e067b61f8eb85ee09cb62d19ae4acbc7.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们原有的应用是在一个可视化 PAAS 平台，所以我们没有使用 kubernetes 原生的这种包管理工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么我们在做应用打包的时候为什么会选择 helm 呢？主要是基于以下几点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"helm 确实是一个应用非常广泛的 kubernetes 包管理工具。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我们对 KubeSphere 3.0 中的多集群应用非常感兴趣，应用市场也是支持 helm 的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"我们需要将应用的变更，应用定义的过程进行版本化。用 helm 这种打包的方式就可以很好实现。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"应用部署"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/03\/035ecb9e3782d76a68d2848c9ba70173.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在应用部署上我们尝试了几个方案，最终选择了 Spinnaker 作为我们的应用发布的方案，主要是基于以下几点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"制品绑定的功能非常有用，我们可以在不修改 helm 包内部配置的情况下，跟制品库形成联动，直接去对每一个组件的 image 进行选择和切换，这在发布的过程中非常方便。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"我们可以将多个集群的部署定义在一个流程内，增加前后依赖关系、人工确认等流程，这使我们的配置不再繁琐。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Spinnaker 支持将集群差异化的配置放到同一个流程里。当然 KubeSphere 多集群应用也能实现这样的功能。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/eb\/eb14b15d1b7b351035543cdb7b93537c.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，这是我们做好了制品绑定之后达到的效果。通过修改发布流程的启动参数，每一个组件在部署的时候都可以去选择它所对需要部署的镜像版本，以及用到的 helm 版本和所要发布的集群。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d3\/d3c9a7f99035404c8031916b547496e7.webp","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上图是我们整个应用定义的过程，首先是确定应用部署所使用的 helm 包，选取部署过程中需要使用的镜像。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右上方的第二张图的 Bake 阶段可以用来做不同集群之间的配置、差异化的管理。比如说我们在生产环境加载 values-prod.yaml 配置文件，但是每一个集群之间会有一定的差异化配置，都可以在 Bake 阶段使用 values 覆盖去进行配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Bake 阶段相当于是 helm 包生成一组部署到 kubernetes 集群里面的 yaml 文件，然后在 deploy 阶段进行制品绑定之后，就可以将 image 字段自动进行一个替换，部署你想要的镜像版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"头图：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者：卢兴民"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：https:\/\/mp.weixin.qq.com\/s\/CoufSH5LTQbob4scTtL05A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文：KubeSphere 多集群管理大招：使用 QKE 管理多个 ACK 集群"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"来源：KubeSphere云原生 - 微信公众号 [ID：gh_f5606464ef2f]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"转载：著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。"}]}]}