Kubernetes學習筆記之Calico CNI Plugin源碼解析(一)

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Overview"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前在"},{"type":"link","attrs":{"href":"http:\/\/mp.weixin.qq.com\/s?__biz=MzU4ODgyMDI0Mg==&mid=2247488126&idx=1&sn=f3e3aa373924f938d9c85b7cfa7e9bd2&chksm=fdd7a803caa02115f3a3fc7cabad4e44f53ad33b8aa53107b5d98140bb0d9affb2cd68d3ff5f&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"Kubernetes學習筆記之kube-proxy service實現原理"}]},{"type":"text","text":"學習到calico會在worker節點上爲pod創建路由route和虛擬網卡virtual interface,併爲pod分配pod ip,以及爲worker節點分配pod cidr網段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們生產k8s網絡插件使用calico cni,在安裝時會安裝兩個插件:calico和calico-ipam,官網安裝文檔 "},{"type":"text","marks":[{"type":"strong"}],"text":"Install the plugin"},{"type":"text","text":"(https:\/\/docs.projectcalico.org\/getting-started\/kubernetes\/hardway\/install-cni-plugin#install-the-plugin)也說到了這一點,而這兩個插件代碼在 "},{"type":"text","marks":[{"type":"strong"}],"text":"calico.go"},{"type":"text","text":"(https:\/\/github.com\/projectcalico\/cni-plugin\/blob\/release-v3.17\/cmd\/calico\/calico.go) ,代碼會編譯出兩個二進制文件:calico和calico-ipam。calico插件主要用來創建route和virtual interface,而calico-ipam插件主要用來分配pod ip和爲worker節點分配pod cidr。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"重要問題是,calico是如何做到的?"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Sandbox container"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kubelet進程在開始啓動時,會調用容器運行時"},{"type":"text","marks":[{"type":"strong"}],"text":"SyncPod"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kubelet.go#L1692) 來創建pod內相關容器,"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要做了幾件事情 "},{"type":"text","marks":[{"type":"strong"}],"text":"L657-L856"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kuberuntime\/kuberuntime_manager.go#L657-L856):"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建sandbox container,這裏會調用cni插件創建network等步驟,同時考慮了邊界條件,創建失敗會kill sandbox container等等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建ephemeral containers、init containers和普通的containers。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏只關注創建sandbox container過程,只有這一步會創建pod network,這個sandbox container創建好後,其餘container都會和其共享同一個network namespace,所以一個pod內各個容器看到的網絡協議棧是同一個,ip地址都是相同的,通過port來區分各個容器。具體創建過程,會調用容器運行時服務創建容器,這裏會先準備好pod的相關配置數據,創建network namespace時也需要這些配置數據 "},{"type":"text","marks":[{"type":"strong"}],"text":"L36-L138"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/kuberuntime\/kuberuntime_sandbox.go#L36-L138) :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (m *kubeGenericRuntimeManager) createPodSandbox(pod *v1.Pod, attempt uint32) (string, string, error) {\n \/\/ 生成pod相關配置數據\n podSandboxConfig, err := m.generatePodSandboxConfig(pod, attempt)\n \/\/ ...\n \/\/ 這裏會在宿主機上創建pod logs目錄,在\/var\/log\/pods\/{namespace}_{pod_name}_{uid}目錄下\n err = m.osInterface.MkdirAll(podSandboxConfig.LogDirectory, 0755)\n \/\/ ...\n \/\/ 調用容器運行時創建sandbox container,我們生產k8s這裏是docker創建\n podSandBoxID, err := m.runtimeService.RunPodSandbox(podSandboxConfig, runtimeHandler)\n \/\/ ...\n return podSandBoxID, \"\", nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"k8s使用cri(container runtime interface)來抽象出標準接口,目前docker還不支持cri接口,所以kubelet做了個適配模塊dockershim,代碼在pkg\/kubelet\/dockershim。上面代碼中的runtimeService對象就是dockerService對象,所以可以看下dockerService.RunPodSandbox()代碼實現 "},{"type":"text","marks":[{"type":"strong"}],"text":"L76-L197"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/dockershim\/docker_sandbox.go#L76-L197):"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n\/\/ 創建sandbox container,以及爲該container創建network\nfunc (ds *dockerService) RunPodSandbox(ctx context.Context, r *runtimeapi.RunPodSandboxRequest) (*runtimeapi.RunPodSandboxResponse, error) {\n config := r.GetConfig()\n \/\/ Step 1: Pull the image for the sandbox.\n \/\/ 1. 拉取鏡像\n image := defaultSandboxImage\n podSandboxImage := ds.podSandboxImage\n if len(podSandboxImage) != 0 {\n image = podSandboxImage\n }\n if err := ensureSandboxImageExists(ds.client, image); err != nil {\n return nil, err\n }\n \n \/\/ Step 2: Create the sandbox container.\n \/\/ 2. 創建sandbox container\n createResp, err := ds.client.CreateContainer(*createConfig)\n \/\/ ...\n resp := &runtimeapi.RunPodSandboxResponse{PodSandboxId: createResp.ID}\n ds.setNetworkReady(createResp.ID, false)\n \n \/\/ Step 3: Create Sandbox Checkpoint.\n \/\/ 3. 創建checkpoint\n if err = ds.checkpointManager.CreateCheckpoint(createResp.ID, constructPodSandboxCheckpoint(config)); err != nil {\n return nil, err\n }\n \n \/\/ Step 4: Start the sandbox container.\n \/\/ Assume kubelet's garbage collector would remove the sandbox later, if\n \/\/ startContainer failed.\n \/\/ 4. 啓動容器\n err = ds.client.StartContainer(createResp.ID)\n\n \/\/ ...\n \/\/ Step 5: Setup networking for the sandbox.\n \/\/ All pod networking is setup by a CNI plugin discovered at startup time.\n \/\/ This plugin assigns the pod ip, sets up routes inside the sandbox,\n \/\/ creates interfaces etc. In theory, its jurisdiction ends with pod\n \/\/ sandbox networking, but it might insert iptables rules or open ports\n \/\/ on the host as well, to satisfy parts of the pod spec that aren't\n \/\/ recognized by the CNI standard yet.\n \/\/ 5. 這一步爲sandbox container創建網絡,主要是調用calico cni插件創建路由和虛擬網卡,以及爲pod分配pod ip,爲該宿主機劃分pod網段\n cID := kubecontainer.BuildContainerID(runtimeName, createResp.ID)\n networkOptions := make(map[string]string)\n if dnsConfig := config.GetDnsConfig(); dnsConfig != nil {\n \/\/ Build DNS options.\n dnsOption, err := json.Marshal(dnsConfig)\n if err != nil {\n return nil, fmt.Errorf(\"failed to marshal dns config for pod %q: %v\", config.Metadata.Name, err)\n }\n networkOptions[\"dns\"] = string(dnsOption)\n }\n\n \/\/ 這一步調用網絡插件來setup sandbox pod\n \/\/ 由於我們網絡插件都是cni(container network interface),所以代碼在 pkg\/kubelet\/dockershim\/network\/cni\/cni.go\n err = ds.network.SetUpPod(config.GetMetadata().Namespace, config.GetMetadata().Name, cID, config.Annotations, networkOptions)\n \/\/ ...\n return resp, nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於我們網絡插件都是cni(container network interface),代碼 ds.network.SetUpPod繼續追下去發現實際調用的是 cniNetworkPlugin.SetUpPod(),代碼在 "},{"type":"text","marks":[{"type":"strong"}],"text":"pkg\/kubelet\/dockershim\/network\/cni\/cni.go"},{"type":"text","text":"(https:\/\/github.com\/kubernetes\/kubernetes\/blob\/release-1.17\/pkg\/kubelet\/dockershim\/network\/cni\/cni.go#L300-L321) :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (plugin *cniNetworkPlugin) SetUpPod(namespace string, name string, id kubecontainer.ContainerID, annotations, options map[string]string) error {\n \/\/ ...\n netnsPath, err := plugin.host.GetNetNS(id.ID)\n \/\/ ...\n \/\/ Windows doesn't have loNetwork. It comes only with Linux\n if plugin.loNetwork != nil {\n \/\/ 添加loopback\n if _, err = plugin.addToNetwork(cniTimeoutCtx, plugin.loNetwork, name, namespace, id, netnsPath, annotations, options); err != nil {\n return err\n }\n }\n \/\/ 調用網絡插件創建網絡相關資源\n _, err = plugin.addToNetwork(cniTimeoutCtx, plugin.getDefaultNetwork(), name, namespace, id, netnsPath, annotations, options)\n return err\n}\n\nfunc (plugin *cniNetworkPlugin) addToNetwork(ctx context.Context, network *cniNetwork, podName string, podNamespace string, podSandboxID kubecontainer.ContainerID, podNetnsPath string, annotations, options map[string]string) (cnitypes.Result, error) {\n \/\/ 這一步準備網絡插件所需相關參數,這些參數最後會被calico插件使用\n rt, err := plugin.buildCNIRuntimeConf(podName, podNamespace, podSandboxID, podNetnsPath, annotations, options)\n \/\/ ...\n \/\/ 這裏會調用調用cni標準庫裏的AddNetworkList函數,最後會調用calico二進制命令\n res, err := cniNet.AddNetworkList(ctx, netConf, rt)\n \/\/ ...\n return res, nil\n}\n\n\/\/ 這些參數主要包括container id,pod等相關參數\nfunc (plugin *cniNetworkPlugin) buildCNIRuntimeConf(podName string, podNs string, podSandboxID kubecontainer.ContainerID, podNetnsPath string, annotations, options map[string]string) (*libcni.RuntimeConf, error) {\n rt := &libcni.RuntimeConf{\n ContainerID: podSandboxID.ID,\n NetNS: podNetnsPath,\n IfName: network.DefaultInterfaceName,\n CacheDir: plugin.cacheDir,\n Args: [][2]string{\n {\"IgnoreUnknown\", \"1\"},\n {\"K8S_POD_NAMESPACE\", podNs},\n {\"K8S_POD_NAME\", podName},\n {\"K8S_POD_INFRA_CONTAINER_ID\", podSandboxID.ID},\n },\n }\n \/\/ port mappings相關參數\n \/\/ ... \n \/\/ dns 相關參數\n \/\/ ...\n return rt, nil\n} "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"addToNetwork()函數會調用cni標準庫裏的 "},{"type":"text","marks":[{"type":"strong"}],"text":"AddNetworkList"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/libcni\/api.go#L400-L440)函數。CNI是容器網絡標準接口Container Network Interface,這個代碼倉庫提供了CNI標準接口的相關實現,所有K8s網絡插件都必須實現該CNI代碼倉庫中的接口,K8s網絡插件如何實現規範可見 SPEC.md(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/SPEC.md) ,我們也可實現遵循該標準規範實現一個簡單的網絡插件。所以kubelet、cni和calico的三者關係就是:kubelet調用cni標準規範代碼包,cni調用calico插件二進制文件。cni代碼包中的AddNetworkList相關代碼如下 "},{"type":"text","marks":[{"type":"strong"}],"text":"AddNetworkList"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/cni\/blob\/master\/libcni\/api.go#L400-L440):"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nfunc (c *CNIConfig) addNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) (types.Result, error) {\n c.ensureExec()\n pluginPath, err := c.exec.FindInPath(net.Network.Type, c.Path)\n \/\/ ...\n \/\/ pluginPath就是calico二進制文件路徑,這裏其實就是調用 calico ADD命令,並傳遞相關參數,參數也是上文描述的已經準備好了的\n \/\/ 參數傳遞也是寫入了環境變量,calico二進制文件可以從環境變量裏取值\n return invoke.ExecPluginWithResult(ctx, pluginPath, newConf.Bytes, c.args(\"ADD\", rt), c.exec)\n}\n\n\/\/ AddNetworkList executes a sequence of plugins with the ADD command\nfunc (c *CNIConfig) AddNetworkList(ctx context.Context, list *NetworkConfigList, rt *RuntimeConf) (types.Result, error) {\n \/\/ ...\n for _, net := range list.Plugins {\n result, err = c.addNetwork(ctx, list.Name, list.CNIVersion, net, result, rt)\n \/\/ ...\n }\n \/\/ ...\n return result, nil\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上pluginPath就是calico二進制文件路徑,這裏calico二進制文件路徑參數是在啓動kubelet時通過參數 --cni-bin-dir 傳進來的,可見官網"},{"type":"text","marks":[{"type":"strong"}],"text":"kubelet command-line-tools-reference"},{"type":"text","text":"(https:\/\/kubernetes.io\/docs\/reference\/command-line-tools-reference\/kubelet\/) ,並且啓動參數 --cni-conf-dir 包含cni配置文件路徑,該路徑包含cni配置文件內容類似如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n{\n \"name\": \"k8s-pod-network\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"calico\",\n \"log_level\": \"debug\",\n \"log_file_path\": \"\/var\/log\/calico\/cni\/cni.log\",\n \"datastore_type\": \"kubernetes\",\n \"nodename\": \"minikube\",\n \"mtu\": 1440,\n \"ipam\": {\n \"type\": \"calico-ipam\"\n },\n \"policy\": {\n \"type\": \"k8s\"\n },\n \"kubernetes\": {\n \"kubeconfig\": \"\/etc\/cni\/net.d\/calico-kubeconfig\"\n }\n },\n {\n \"type\": \"portmap\",\n \"snat\": true,\n \"capabilities\": {\"portMappings\": true}\n },\n {\n \"type\": \"bandwidth\",\n \"capabilities\": {\"bandwidth\": true}\n }\n ]\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"cni相關代碼是個標準骨架,核心還是需要調用第三方網絡插件來實現爲sandbox創建網絡資源。cni也提供了一些示例plugins,代碼倉庫見 "},{"type":"text","marks":[{"type":"strong"}],"text":"containernetworking\/plugins"},{"type":"text","text":"(https:\/\/github.com\/containernetworking\/plugins),並配有文檔說明見 plugins docs(https:\/\/www.cni.dev\/plugins\/) ,比如可以參考學習官網提供的 "},{"type":"text","marks":[{"type":"strong"}],"text":"static IP address management plugin"},{"type":"text","text":"(https:\/\/www.cni.dev\/plugins\/ipam\/static\/) 。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總之,kubelet在創建sandbox container時候,會先調用cni插件命令,如 calico ADD 命令並通過環境變量傳遞相關命令參數,來給sandbox container創建network相關資源對象,比如calico會創建route和virtual interface,以及爲pod分配ip地址,和從集羣網段cluster cidr中爲當前worker節點分配pod cidr網段,並且會把這些數據寫入到calico datastore數據庫裏。所以,關鍵問題,還是得看calico插件代碼是如何做的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"參考鏈接"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/docs.projectcalico.org\/networking\/use-specific-ip"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/mp.weixin.qq.com\/s\/lyfeZh6VWWjXuLY8fl3ciw"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/www.yuque.com\/baxiaoshi\/tyado3\/lvfa0b"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/github.com\/containernetworking\/cni"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/github.com\/projectcalico\/cni-plugin"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自:360技術(ID:qihoo_tech)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接:"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/iSTwPy41RmrzbYFi10BJGQ","title":"xxx","type":null},"content":[{"type":"text","text":"Kubernetes學習筆記之Calico CNI Plugin源碼解析(一)"}]}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章