解決Kubernetes1.5.1 coredns報錯CrashLoopBackOff

今天在使用K8s查看pod時發現,coredns出現了CrashLoopBackOff

[root@k8s-master01 flannel]# kubectl get pod -n kube-system
NAME                                   READY   STATUS             RESTARTS   AGE
coredns-5c98db65d4-f9rb7               0/1     CrashLoopBackOff   50         9d
coredns-5c98db65d4-xcd9s               0/1     CrashLoopBackOff   50         9d
etcd-k8s-master01                      1/1     Running            2          9d
kube-apiserver-k8s-master01            1/1     Running            2          9d
kube-controller-manager-k8s-master01   1/1     Running            3          9d
kube-flannel-ds-amd64-6h79p            1/1     Running            2          9d
kube-flannel-ds-amd64-bnvtd            1/1     Running            3          9d
kube-flannel-ds-amd64-bsq4j            1/1     Running            2          9d
kube-proxy-5fn9m                       1/1     Running            1          9d
kube-proxy-6hjvp                       1/1     Running            2          9d
kube-proxy-t47n9                       1/1     Running            2          9d
kube-scheduler-k8s-master01            1/1     Running            4          9d

使用kubectl logs命令查看, 報錯很奇怪

[root@k8s-master01 ~]# kubectl logs coredns-5c98db65d4-xcd9s -n kube-system
E0413 06:32:09.919666       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:317: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0413 06:32:09.919666       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:317: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host

原因:

查閱k8s官方文檔

coredns pods 有 CrashLoopBackOff 或者 Error 狀態
如果有些節點運行的是舊版本的 Docker，同時啓用了 SELinux，您或許會遇到 coredns pods 無法啓動的情況。 要解決此問題，您可以嘗試以下選項之一：

升級到 Docker 的較新版本。

禁用 SELinux.

修改 coredns 部署以設置 allowPrivilegeEscalation 爲 true：

kubectl -n kube-system get deployment coredns -o yaml | \
sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
kubectl apply -f -
CoreDNS 處於 CrashLoopBackOff 時的另一個原因是當 Kubernetes 中部署的 CoreDNS Pod 檢測 到環路時。有許多解決方法 可以避免在每次 CoreDNS 監測到循環並退出時，Kubernetes 嘗試重啓 CoreDNS Pod 的情況。

警告：
警告：禁用 SELinux 或設置 allowPrivilegeEscalation 爲 true 可能會損害集羣的安全性。

我這裏的原因可能是以前配置iptables時產生的

解決

1. 設置iptables爲空規則
   iptables -F && service iptables save
2. 刪除報錯的coredns pod

[root@k8s-master01 flannel]# kubectl delete pod coredns-5c98db65d4-xcd9s
Error from server (NotFound): pods "coredns-5c98db65d4-xcd9s" not found
[root@k8s-master01 flannel]# kubectl delete pod coredns-5c98db65d4-xcd9s -n kube-system
pod "coredns-5c98db65d4-xcd9s" deleted
[root@k8s-master01 flannel]# kubectl delete pod coredns-5c98db65d4-f9rb7  -n kube-system
pod "coredns-5c98db65d4-f9rb7" deleted

重新查看pod

[root@k8s-master01 flannel]# kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-54j5c               1/1     Running   0          13m
coredns-5c98db65d4-jmvbf               1/1     Running   0          14m
etcd-k8s-master01                      1/1     Running   2          9d
kube-apiserver-k8s-master01            1/1     Running   2          9d
kube-controller-manager-k8s-master01   1/1     Running   3          9d
kube-flannel-ds-amd64-6h79p            1/1     Running   2          9d
kube-flannel-ds-amd64-bnvtd            1/1     Running   3          9d
kube-flannel-ds-amd64-bsq4j            1/1     Running   2          9d
kube-proxy-5fn9m                       1/1     Running   1          9d
kube-proxy-6hjvp                       1/1     Running   2          9d
kube-proxy-t47n9                       1/1     Running   2          9d
kube-scheduler-k8s-master01            1/1     Running   4          9d
[root@k8s-master01 flannel]# 

狀態重新變成Running