本文分享自華爲雲社區《K8s 鏡像緩存管理 kube-fledged 認知》，作者： 山河已無恙。
我們知道 k8s 上的容器調度需要在調度的節點行拉取當前容器的鏡像，在一些特殊場景中，
-
需要
快速啓動和/或擴展的應用程序。例如，由於數據量激增，執行實時數據處理的應用程序需要快速擴展。 -
鏡像比較龐大，涉及多個版本，節點存儲有限，需要動態清理不需要的鏡像
-
無服務器函數通常需要在幾分之一秒內立即對傳入事件和啓動容器做出反應。 -
在邊緣設備上運行的
IoT 應用程序，需要容忍邊緣設備和鏡像鏡像倉庫之間的間歇性網絡連接。 -
如果需要從
專用倉庫中拉取鏡像，並且無法授予每個人從此鏡像倉庫拉取鏡像的訪問權限，則可以在羣集的節點上提供鏡像。 -
如果集羣管理員或操作員需要對應用程序進行升級，並希望事先驗證是否可以成功拉取新鏡像。
kube-fledged 是一個 kubernetes operator，用於直接在 Kubernetes 集羣的 worker 節點上創建和管理容器鏡像緩存。它允許用戶定義鏡像列表以及這些鏡像應緩存到哪些工作節點上（即拉取）。因此，應用程序 Pod 幾乎可以立即啓動，因爲不需要從鏡像倉庫中提取鏡像。
kube-fledged 提供了 CRUD API 來管理鏡像緩存的生命週期，並支持多個可配置的參數，可以根據自己的需要自定義功能。
Kubernetes 具有內置的鏡像垃圾回收機制。節點中的 kubelet 會定期檢查磁盤使用率是否達到特定閾值（可通過標誌進行配置）。一旦達到這個閾值，kubelet 會自動刪除節點中所有未使用的鏡像。
需要在建議的解決方案中實現自動和定期刷新機制。如果鏡像緩存中的鏡像被 kubelet 的 gc 刪除，下一個刷新週期會將已刪除的鏡像拉入鏡像緩存中。這可確保鏡像緩存是最新的。
設計流程
https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png
部署 kube-fledged
Helm 方式部署
──[[email protected]]-[~/ansible] └─$mkdir kube-fledged ┌──[[email protected]]-[~/ansible] └─$cd kube-fledged ┌──[[email protected]]-[~/ansible/kube-fledged] └─$export KUBEFLEDGED_NAMESPACE=kube-fledged ┌──[[email protected]]-[~/ansible/kube-fledged] └─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE} namespace/kube-fledged created ┌──[[email protected]]-[~/ansible/kube-fledged] └─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/ "kubefledged-charts" has been added to your repositories ┌──[[email protected]]-[~/ansible/kube-fledged] └─$helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "kubefledged-charts" chart repository ...Successfully got an update from the "kubescape" chart repository ...Successfully got an update from the "rancher-stable" chart repository ...Successfully got an update from the "skm" chart repository ...Successfully got an update from the "openkruise" chart repository ...Successfully got an update from the "awx-operator" chart repository ...Successfully got an update from the "botkube" chart repository Update Complete. ⎈Happy Helming!⎈
┌──[[email protected]]-[~/ansible/kube-fledged] └─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait
實際部署中發現，由於網絡問題，chart 無法下載，所以通過 make deploy-using-yaml 使用 yaml 方式部署
Yaml 文件部署
┌──[[email protected]]-[~/ansible/kube-fledged] └─$git clone https://github.com/senthilrch/kube-fledged.git 正克隆到 'kube-fledged'... remote: Enumerating objects: 10613, done. remote: Counting objects: 100% (1501/1501), done. remote: Compressing objects: 100% (629/629), done. remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112 接收對象中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done. 處理 delta 中: 100% (4431/4431), done. ┌──[[email protected]]-[~/ansible/kube-fledged] └─$ls kube-fledged ┌──[[email protected]]-[~/ansible/kube-fledged] └─$cd kube-fledged/ ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$make deploy-using-yaml kubectl apply -f deploy/kubefledged-namespace.yaml
第一次部署，發現鏡像拉不下來
┌──[[email protected]]-[~] └─$kubectl get all -n kube-fledged NAME READY STATUS RESTARTS AGE pod/kube-fledged-controller-df69f6565-drrqg 0/1 CrashLoopBackOff 35 (5h59m ago) 21h pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2 0/1 Init:CrashLoopBackOff 35 (5h58m ago) 21h pod/kubefledged-controller-55f848cc67-7f4rl 1/1 Running 0 21h pod/kubefledged-webhook-server-597dbf4ff5-l8fbh 0/1 Init:CrashLoopBackOff 34 (6h ago) 21h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-fledged-webhook-server ClusterIP 10.100.194.199 <none> 3443/TCP 21h service/kubefledged-webhook-server ClusterIP 10.101.191.206 <none> 3443/TCP 21h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-fledged-controller 0/1 1 0 21h deployment.apps/kube-fledged-webhook-server 0/1 1 0 21h deployment.apps/kubefledged-controller 0/1 1 0 21h deployment.apps/kubefledged-webhook-server 0/1 1 0 21h NAME DESIRED CURRENT READY AGE replicaset.apps/kube-fledged-controller-df69f6565 1 1 0 21h replicaset.apps/kube-fledged-webhook-server-7bcd589bc4 1 1 0 21h replicaset.apps/kubefledged-controller-55f848cc67 1 1 0 21h replicaset.apps/kubefledged-webhook-server-597dbf4ff5 1 1 0 21h ┌──[[email protected]]-[~] └─$
這裏我們找一下要拉取的鏡像
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat *.yaml | grep image: - image: senthilrch/kubefledged-controller:v0.10.0 - image: senthilrch/kubefledged-webhook-server:v0.10.0 - image: senthilrch/kubefledged-webhook-server:v0.10.0
單獨拉取一些，當前使用 ansible 在所有工作節點批量操作
┌──[[email protected]]-[~/ansible] └─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml
其他相關的鏡像都拉取一下
操作完成之後容器狀態全部正常
┌──[[email protected]]-[~/ansible] └─$kubectl -n kube-fledged get all NAME READY STATUS RESTARTS AGE pod/kube-fledged-controller-df69f6565-wdb4g 1/1 Running 0 13h pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp 1/1 Running 0 13h pod/kubefledged-controller-55f848cc67-klxlm 1/1 Running 0 13h pod/kubefledged-webhook-server-597dbf4ff5-ktbsh 1/1 Running 0 13h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-fledged-webhook-server ClusterIP 10.100.194.199 <none> 3443/TCP 36h service/kubefledged-webhook-server ClusterIP 10.101.191.206 <none> 3443/TCP 36h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-fledged-controller 1/1 1 1 36h deployment.apps/kube-fledged-webhook-server 1/1 1 1 36h deployment.apps/kubefledged-controller 1/1 1 1 36h deployment.apps/kubefledged-webhook-server 1/1 1 1 36h NAME DESIRED CURRENT READY AGE replicaset.apps/kube-fledged-controller-df69f6565 1 1 1 36h replicaset.apps/kube-fledged-webhook-server-7bcd589bc4 1 1 1 36h replicaset.apps/kubefledged-controller-55f848cc67 1 1 1 36h replicaset.apps/kubefledged-webhook-server-597dbf4ff5 1 1 1 36h
驗證是否安裝成功
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$kubectl get pods -n kube-fledged -l app=kubefledged NAME READY STATUS RESTARTS AGE kubefledged-controller-55f848cc67-klxlm 1/1 Running 0 16h kubefledged-webhook-server-597dbf4ff5-ktbsh 1/1 Running 0 16h ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$kubectl get imagecaches -n kube-fledged No resources found in kube-fledged namespace.
使用 kubefledged
創建鏡像緩存對象
根據 Demo 文件，創建鏡像緩存對象
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$cd deploy/ ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster - images: - ghcr.io/jitesoft/nginx:1.23.1 # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - us.gcr.io/k8s-artifacts-prod/cassandra:v7 - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0 nodeSelector: tier: backend # Specifies a list of image pull secrets to pull images from private repositories into the cache imagePullSecrets: - name: myregistrykey
官方的 Demo 中對應的 鏡像拉取不下來，所以換一下
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7 Error response from daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
爲了測試選擇器標籤的使用,我們找一個節點的標籤單獨做鏡像緩存
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get nodes --show-labels
同時我們直接從公有倉庫拉取鏡像，所以不需要 imagePullSecrets 對象
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$vim kubefledged-imagecache.yaml
修改後的 yaml 文件
-
添加了一個所有節點的 liruilong/my-busybox:latest 鏡像緩存
-
添加了一個
kubernetes.io/hostname: vms105.liruilongs.github.io對應標籤選擇器的liruilong/hikvision-sdk-config-ftp:latest鏡像緩存
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster - images: - liruilong/my-busybox:latest # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - liruilong/hikvision-sdk-config-ftp:latest nodeSelector: kubernetes.io/hostname: vms105.liruilongs.github.io # Specifies a list of image pull secrets to pull images from private repositories into the cache #imagePullSecrets: #- name: myregistrykey ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
直接創建報錯了
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl create -f kubefledged-imagecache.yaml Error from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io") ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get imagecaches -n kube-fledged No resources found in kube-fledged namespace. ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
解決辦法，刪除對應的對象，重新創建
我在當前項目的一個 issues 下面找到了解決辦法 https://github.com/senthilrch/kube-fledged/issues/76
看起來這是因爲 Webhook CA 是硬編碼的，但是當 webhook 服務器啓動時，會生成一個新的 CA 捆綁包並更新 webhook 配置。當發生另一個部署時，將重新應用原始 CA 捆綁包，並且 Webhook 請求開始失敗，直到再次重新啓動 Webhook 組件以修補捆綁包init-server
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$make remove-kubefledged-and-operator # Remove kubefledged kubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml error: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2" ensure CRDs are installed first
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged] └─$make deploy-using-yaml kubectl apply -f deploy/kubefledged-namespace.yaml namespace/kube-fledged created kubectl apply -f deploy/kubefledged-crd.yaml customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged .................... kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch Waiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available... deployment "kubefledged-webhook-server" successfully rolled out kubectl get pods -n kube-fledged NAME READY STATUS RESTARTS AGE kubefledged-controller-55f848cc67-76c4v 1/1 Running 0 112s kubefledged-webhook-server-597dbf4ff5-56h6z 1/1 Running 0 66s
重新創建緩存對象，創建成功
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl create -f kubefledged-imagecache.yaml imagecache.kubefledged.io/imagecache1 created ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get imagecaches -n kube-fledged NAME AGE imagecache1 10s ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
查看當前被納管的鏡像緩存
┌──[[email protected]]-[~/ansible/kube-fledged] └─$kubectl get imagecaches imagecache1 -n kube-fledged -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 83, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20169836", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/my-busybox:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:06:47Z", "message": "All requested images pulled succesfully to respective nodes", "reason": "ImageCacheRefresh", "startTime": "2024-03-02T01:05:33Z", "status": "Succeeded" } } ┌──[[email protected]]-[~/ansible/kube-fledged] └─$
通過 ansible 來驗證
┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.102 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.101 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.103 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.105 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.100 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.106 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB ┌──[[email protected]]-[~/ansible] └─$
┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | CHANGED | rc=0 >> liruilong/hikvision-sdk-config-ftp latest a02cd03b4342 4 months ago 830MB 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[[email protected]]-[~/ansible] └─$
開啓自動刷新
┌──[[email protected]]-[~/ansible] └─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache= imagecache.kubefledged.io/imagecache1 annotated ┌──[[email protected]]-[~/ansible] └─$
添加鏡像緩存
添加一個新的鏡像緩存
┌──[[email protected]]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 92, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20175233", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/my-busybox:latest", "liruilong/jdk1.8_191:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:43:32Z", "message": "All requested images pulled succesfully to respective nodes", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:40:34Z", "status": "Succeeded" } } ┌──[[email protected]]-[~/ansible] └─$
通過 ansible 確認
┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.101 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.102 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.100 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.103 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.105 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.106 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB ┌──[[email protected]]-[~/ansible] └─$
刪除鏡像緩存
┌──[[email protected]]-[~/ansible] └─$kubectl edit imagecaches imagecache1 -n kube-fledged imagecache.kubefledged.io/imagecache1 edited ┌──[[email protected]]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 94, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20175766", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/jdk1.8_191:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "message": "Image cache is being updated. Please view the status after some time", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:48:03Z", "status": "Processing" } }
通過 Ansible 確認，可以看到無論是 mastere 上的節點還是 work 的節點，對應的鏡像緩存都被清理
┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.102 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.101 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[[email protected]]-[~/ansible] └─$
這裏需要注意如果清除所有的鏡像緩存，那麼需要把 images 下的數組 寫成 "".
┌──[[email protected]]-[~/ansible] └─$kubectl edit imagecaches imagecache1 -n kube-fledged imagecache.kubefledged.io/imagecache1 edited ┌──[[email protected]]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[[email protected]]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 98, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20176849", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:52:16Z", "message": "All cached images succesfully deleted from respective nodes", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:51:47Z", "status": "Succeeded" } } ┌──[[email protected]]-[~/ansible] └─$
如果通過下面的方式刪除，直接註釋調對應的標籤
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster #- images: #- liruilong/my-busybox:latest # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - liruilong/hikvision-sdk-config-ftp:latest nodeSelector: kubernetes.io/hostname: vms105.liruilongs.github.io # Specifies a list of image pull secrets to pull images from private repositories into the cache #imagePullSecrets: #- name: myregistrykey ┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
那麼會報下面的錯
┌──[[email protected]]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl edit imagecaches imagecache1 -n kube-fledged error: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image lists You can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.
博文部分內容參考
© 文中涉及參考鏈接內容版權歸原作者所有，如有侵權請告知,如果你認可它不要吝嗇星星哦 :)
https://github.com/senthilrch/kube-fledged