Kubernetes学习笔记之Calico CNI Plugin源码解析(一)

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Overview"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前在"},{"type":"link","attrs":{"href":"http:\/\/mp.weixin.qq.com\/s?__biz=MzU4ODgyMDI0Mg==&mid=2247488126&idx=1&sn=f3e3aa373924f938d9c85b7cfa7e9bd2&chksm=fdd7a803caa02115f3a3fc7cabad4e44f53ad33b8aa53107b5d98140bb0d9affb2cd68d3ff5f&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"Kubernetes学习笔记之kube-proxy service实现原理"}]},{"type":"text","text":"学习到calico会在worker节点上为pod创建路由route和虚拟网卡virtual interface，并为pod分配pod ip，以及为worker节点分配pod cidr网段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们生产k8s网络插件使用calico cni，在安装时会安装两个插件：calico和calico-ipam，官网安装文档 "},{"type":"text","marks":[{"type":"strong"}],"text":"Install the plugin"},{"type":"text","text":"(https:\/\/docs.projectcalico.org\/getting-started\/kubernetes\/hardway\/install-cni-plugin#install-the-plugin)也说到了这一点，而这两个插件代码在 "},{"type":"text","marks":[{"type":"strong"}],"text":"calico.go"},{"type":"text","text":"(https:\/\/github.com\/projectcalico\/cni-plugin\/blob\/release-v3.17\/cmd\/calico\/calico.go) ，代码会编译出两个二进制文件：calico和calico-ipam。calico插件主要用来创建route和virtual interface，而calico-ipam插件主要用来分配pod ip和为worker节点分配pod cidr。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"重要问题是，calico是如何做到的？"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Sandbox container"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kubelet进程在开始启动时，会调用容器运行时"},{"type":"text","marks":[{"type":"strong"}],"text":"SyncPod"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kubelet.go#L1692) 来创建pod内相关容器，"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要做了几件事情 "},{"type":"text","marks":[{"type":"strong"}],"text":"L657-L856"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kuberuntime\/kuberuntime_manager.go#L657-L856)："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建sandbox container，这里会调用cni插件创建network等步骤，同时考虑了边界条件，创建失败会kill sandbox container等等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建ephemeral containers、init containers和普通的containers。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里只关注创建sandbox container过程，只有这一步会创建pod network，这个sandbox container创建好后，其余container都会和其共享同一个network namespace，所以一个pod内各个容器看到的网络协议栈是同一个，ip地址都是相同的，通过port来区分各个容器。具体创建过程，会调用容器运行时服务创建容器，这里会先准备好pod的相关配置数据，创建network namespace时也需要这些配置数据 "},{"type":"text","marks":[{"type":"strong"}],"text":"L36-L138"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kuberuntime\/kuberuntime_sandbox.go#L36-L138) ："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (m *kubeGenericRuntimeManager) createPodSandbox(pod *v1.Pod, attempt uint32) (string, string, error) {\n \/\/ 生成pod相关配置数据\n podSandboxConfig, err := m.generatePodSandboxConfig(pod, attempt)\n \/\/ ...\n \/\/ 这里会在宿主机上创建pod logs目录，在\/var\/log\/pods\/{namespace}_{pod_name}_{uid}目录下\n err = m.osInterface.MkdirAll(podSandboxConfig.LogDirectory, 0755)\n \/\/ ...\n \/\/ 调用容器运行时创建sandbox container，我们生产k8s这里是docker创建\n podSandBoxID, err := m.runtimeService.RunPodSandbox(podSandboxConfig, runtimeHandler)\n \/\/ ...\n return podSandBoxID, \"\", nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"k8s使用cri(container runtime interface)来抽象出标准接口，目前docker还不支持cri接口，所以kubelet做了个适配模块dockershim，代码在pkg\/kubelet\/dockershim。上面代码中的runtimeService对象就是dockerService对象，所以可以看下dockerService.RunPodSandbox()代码实现 "},{"type":"text","marks":[{"type":"strong"}],"text":"L76-L197"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/dockershim\/docker_sandbox.go#L76-L197)："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n\/\/ 创建sandbox container，以及为该container创建network\nfunc (ds *dockerService) RunPodSandbox(ctx context.Context, r *runtimeapi.RunPodSandboxRequest) (*runtimeapi.RunPodSandboxResponse, error) {\n config := r.GetConfig()\n \/\/ Step 1: Pull the image for the sandbox.\n \/\/ 1. 拉取镜像\n image := defaultSandboxImage\n podSandboxImage := ds.podSandboxImage\n if len(podSandboxImage) != 0 {\n image = podSandboxImage\n }\n if err := ensureSandboxImageExists(ds.client, image); err != nil {\n return nil, err\n }\n \n \/\/ Step 2: Create the sandbox container.\n \/\/ 2. 创建sandbox container\n createResp, err := ds.client.CreateContainer(*createConfig)\n \/\/ ...\n resp := &runtimeapi.RunPodSandboxResponse{PodSandboxId: createResp.ID}\n ds.setNetworkReady(createResp.ID, false)\n \n \/\/ Step 3: Create Sandbox Checkpoint.\n \/\/ 3. 创建checkpoint\n if err = ds.checkpointManager.CreateCheckpoint(createResp.ID, constructPodSandboxCheckpoint(config)); err != nil {\n return nil, err\n }\n \n \/\/ Step 4: Start the sandbox container.\n \/\/ Assume kubelet's garbage collector would remove the sandbox later, if\n \/\/ startContainer failed.\n \/\/ 4. 启动容器\n err = ds.client.StartContainer(createResp.ID)\n\n \/\/ ...\n \/\/ Step 5: Setup networking for the sandbox.\n \/\/ All pod networking is setup by a CNI plugin discovered at startup time.\n \/\/ This plugin assigns the pod ip, sets up routes inside the sandbox,\n \/\/ creates interfaces etc. In theory, its jurisdiction ends with pod\n \/\/ sandbox networking, but it might insert iptables rules or open ports\n \/\/ on the host as well, to satisfy parts of the pod spec that aren't\n \/\/ recognized by the CNI standard yet.\n \/\/ 5. 这一步为sandbox container创建网络，主要是调用calico cni插件创建路由和虚拟网卡，以及为pod分配pod ip，为该宿主机划分pod网段\n cID := kubecontainer.BuildContainerID(runtimeName, createResp.ID)\n networkOptions := make(map[string]string)\n if dnsConfig := config.GetDnsConfig(); dnsConfig != nil {\n \/\/ Build DNS options.\n dnsOption, err := json.Marshal(dnsConfig)\n if err != nil {\n return nil, fmt.Errorf(\"failed to marshal dns config for pod %q: %v\", config.Metadata.Name, err)\n }\n networkOptions[\"dns\"] = string(dnsOption)\n }\n\n \/\/ 这一步调用网络插件来setup sandbox pod\n \/\/ 由于我们网络插件都是cni(container network interface)，所以代码在 pkg\/kubelet\/dockershim\/network\/cni\/cni.go\n err = ds.network.SetUpPod(config.GetMetadata().Namespace, config.GetMetadata().Name, cID, config.Annotations, networkOptions)\n \/\/ ...\n return resp, nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于我们网络插件都是cni(container network interface)，代码 ds.network.SetUpPod继续追下去发现实际调用的是 cniNetworkPlugin.SetUpPod()，代码在 "},{"type":"text","marks":[{"type":"strong"}],"text":"pkg\/kubelet\/dockershim\/network\/cni\/cni.go"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/dockershim\/network\/cni\/cni.go#L300-L321) ："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (plugin *cniNetworkPlugin) SetUpPod(namespace string, name string, id kubecontainer.ContainerID, annotations, options map[string]string) error {\n \/\/ ...\n netnsPath, err := plugin.host.GetNetNS(id.ID)\n \/\/ ...\n \/\/ Windows doesn't have loNetwork. It comes only with Linux\n if plugin.loNetwork != nil {\n \/\/ 添加loopback\n if _, err = plugin.addToNetwork(cniTimeoutCtx, plugin.loNetwork, name, namespace, id, netnsPath, annotations, options); err != nil {\n return err\n }\n }\n \/\/ 调用网络插件创建网络相关资源\n _, err = plugin.addToNetwork(cniTimeoutCtx, plugin.getDefaultNetwork(), name, namespace, id, netnsPath, annotations, options)\n return err\n}\n\nfunc (plugin *cniNetworkPlugin) addToNetwork(ctx context.Context, network *cniNetwork, podName string, podNamespace string, podSandboxID kubecontainer.ContainerID, podNetnsPath string, annotations, options map[string]string) (cnitypes.Result, error) {\n \/\/ 这一步准备网络插件所需相关参数，这些参数最后会被calico插件使用\n rt, err := plugin.buildCNIRuntimeConf(podName, podNamespace, podSandboxID, podNetnsPath, annotations, options)\n \/\/ ...\n \/\/ 这里会调用调用cni标准库里的AddNetworkList函数，最后会调用calico二进制命令\n res, err := cniNet.AddNetworkList(ctx, netConf, rt)\n \/\/ ...\n return res, nil\n}\n\n\/\/ 这些参数主要包括container id，pod等相关参数\nfunc (plugin *cniNetworkPlugin) buildCNIRuntimeConf(podName string, podNs string, podSandboxID kubecontainer.ContainerID, podNetnsPath string, annotations, options map[string]string) (*libcni.RuntimeConf, error) {\n rt := &libcni.RuntimeConf{\n ContainerID: podSandboxID.ID,\n NetNS: podNetnsPath,\n IfName: network.DefaultInterfaceName,\n CacheDir: plugin.cacheDir,\n Args: [][2]string{\n {\"IgnoreUnknown\", \"1\"},\n {\"K8S_POD_NAMESPACE\", podNs},\n {\"K8S_POD_NAME\", podName},\n {\"K8S_POD_INFRA_CONTAINER_ID\", podSandboxID.ID},\n },\n }\n \/\/ port mappings相关参数\n \/\/ ... \n \/\/ dns 相关参数\n \/\/ ...\n return rt, nil\n} "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"addToNetwork()函数会调用cni标准库里的 "},{"type":"text","marks":[{"type":"strong"}],"text":"AddNetworkList"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/libcni\/api.go#L400-L440)函数。CNI是容器网络标准接口Container Network Interface，这个代码仓库提供了CNI标准接口的相关实现，所有K8s网络插件都必须实现该CNI代码仓库中的接口，K8s网络插件如何实现规范可见 SPEC.md(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/SPEC.md) ，我们也可实现遵循该标准规范实现一个简单的网络插件。所以kubelet、cni和calico的三者关系就是：kubelet调用cni标准规范代码包，cni调用calico插件二进制文件。cni代码包中的AddNetworkList相关代码如下 "},{"type":"text","marks":[{"type":"strong"}],"text":"AddNetworkList"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/libcni\/api.go#L400-L440)："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (c *CNIConfig) addNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) (types.Result, error) {\n c.ensureExec()\n pluginPath, err := c.exec.FindInPath(net.Network.Type, c.Path)\n \/\/ ...\n \/\/ pluginPath就是calico二进制文件路径，这里其实就是调用 calico ADD命令，并传递相关参数，参数也是上文描述的已经准备好了的\n \/\/ 参数传递也是写入了环境变量，calico二进制文件可以从环境变量里取值\n return invoke.ExecPluginWithResult(ctx, pluginPath, newConf.Bytes, c.args(\"ADD\", rt), c.exec)\n}\n\n\/\/ AddNetworkList executes a sequence of plugins with the ADD command\nfunc (c *CNIConfig) AddNetworkList(ctx context.Context, list *NetworkConfigList, rt *RuntimeConf) (types.Result, error) {\n \/\/ ...\n for _, net := range list.Plugins {\n result, err = c.addNetwork(ctx, list.Name, list.CNIVersion, net, result, rt)\n \/\/ ...\n }\n \/\/ ...\n return result, nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上pluginPath就是calico二进制文件路径，这里calico二进制文件路径参数是在启动kubelet时通过参数 --cni-bin-dir 传进来的，可见官网"},{"type":"text","marks":[{"type":"strong"}],"text":"kubelet command-line-tools-reference"},{"type":"text","text":"(https:\/\/kubernetes.io\/docs\/reference\/command-line-tools-reference\/kubelet\/) ，并且启动参数 --cni-conf-dir 包含cni配置文件路径，该路径包含cni配置文件内容类似如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n{\n \"name\": \"k8s-pod-network\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"calico\",\n \"log_level\": \"debug\",\n \"log_file_path\": \"\/var\/log\/calico\/cni\/cni.log\",\n \"datastore_type\": \"kubernetes\",\n \"nodename\": \"minikube\",\n \"mtu\": 1440,\n \"ipam\": {\n \"type\": \"calico-ipam\"\n },\n \"policy\": {\n \"type\": \"k8s\"\n },\n \"kubernetes\": {\n \"kubeconfig\": \"\/etc\/cni\/net.d\/calico-kubeconfig\"\n }\n },\n {\n \"type\": \"portmap\",\n \"snat\": true,\n \"capabilities\": {\"portMappings\": true}\n },\n {\n \"type\": \"bandwidth\",\n \"capabilities\": {\"bandwidth\": true}\n }\n ]\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"cni相关代码是个标准骨架，核心还是需要调用第三方网络插件来实现为sandbox创建网络资源。cni也提供了一些示例plugins，代码仓库见 "},{"type":"text","marks":[{"type":"strong"}],"text":"containernetworking\/plugins"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/plugins)，并配有文档说明见 plugins docs(https:\/\/www.cni.dev\/plugins\/) ，比如可以参考学习官网提供的 "},{"type":"text","marks":[{"type":"strong"}],"text":"static IP address management plugin"},{"type":"text","text":"(https:\/\/www.cni.dev\/plugins\/ipam\/static\/) 。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"总之，kubelet在创建sandbox container时候，会先调用cni插件命令，如 calico ADD 命令并通过环境变量传递相关命令参数，来给sandbox container创建network相关资源对象，比如calico会创建route和virtual interface，以及为pod分配ip地址，和从集群网段cluster cidr中为当前worker节点分配pod cidr网段，并且会把这些数据写入到calico datastore数据库里。所以，关键问题，还是得看calico插件代码是如何做的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"参考链接"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/docs.projectcalico.org\/networking\/use-specific-ip"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/mp.weixin.qq.com\/s\/lyfeZh6VWWjXuLY8fl3ciw"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/www.yuque.com\/baxiaoshi\/tyado3\/lvfa0b"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/github.com\/containernetworking\/cni"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/github.com\/projectcalico\/cni-plugin"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：360技术（ID：qihoo_tech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/iSTwPy41RmrzbYFi10BJGQ","title":"xxx","type":null},"content":[{"type":"text","text":"Kubernetes学习笔记之Calico CNI Plugin源码解析(一)"}]}]}]}