原始的問題是這樣：
The connection to the server 192.168.122.200:6443 was refused - did you specify the right host or port?
lsof -i :6443
端口沒有監聽
master 的kubelet.service 是active的，docker容器api_server是down的
node的 kubelet.service 是 dead, docker容器都是down的
journalctl -xefu kubelet 看到的日誌都是連不上6443端口
後面找到容器kube-apiserver查看日誌：
似乎是證書過期了。。。
以下是解決步驟：
一. 檢查證書是否過期。
可以通過下面兩種方式檢查 Kubernetes 的證書是否過期。
1. kubeadm 命令查看
可以通過 kubeadm alpha certs check-expiration 命令查看相關證書是否過期。
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jul 08, 2023 14:11 UTC 364d no apiserver Jul 08, 2023 14:11 UTC 364d ca no apiserver-etcd-client Jul 08, 2023 14:11 UTC 364d etcd-ca no apiserver-kubelet-client Jul 08, 2023 14:11 UTC 364d ca no controller-manager.conf Jul 08, 2023 14:11 UTC 364d no etcd-healthcheck-client Jul 08, 2023 14:11 UTC 364d etcd-ca no etcd-peer Jul 08, 2023 14:11 UTC 364d etcd-ca no etcd-server Jul 08, 2023 14:11 UTC 364d etcd-ca no front-proxy-client Jul 08, 2023 14:11 UTC 364d front-proxy-ca no scheduler.conf Jul 08, 2023 14:11 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jul 02, 2031 14:04 UTC 8y no etcd-ca Jul 02, 2031 14:04 UTC 8y no front-proxy-ca Jul 02, 2031 14:04 UTC 8y no
2. openssl 命令查看
版本過低無法使用 kubeadm 命令時，可以通過 openssl 查看對應證書是否過期。
[root@master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not ' Not Before: Jul 4 14:04:27 2021 GMT Not After : Jul 8 14:11:17 2023 GMT
二. 自動更新證書
Kubenetes 在升級控制面板相關組件時會主動更新證書，因此如果保證 Kubernetes 能夠定期（一年以內）升級的話，證書會自動更新。
三. 手動更新證書
1. 證書備份
cp -rp /etc/kubernetes /etc/kubernetes.bak
2. 刪除舊的證書
將 /etc/kubernetes/pki 下要重新生成的證書刪除
sudo rm -rf /etc/kubernetes/pki/apiserver.key
3. 重新生成證書
主要通過 kubeadm alpha certs renew 命令生成，命令簡介如下
kubeadm alpha certs renew Usage: kubeadm alpha certs renew [flags] kubeadm alpha certs renew [command] Available Commands: all renew all available certificates apiserver Generates the certificate for serving the Kubernetes API apiserver-etcd-client Generates the client apiserver uses to access etcd apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet etcd-healthcheck-client Generates the client certificate for liveness probes to healtcheck etcd etcd-peer Generates the credentials for etcd nodes to communicate with each other etcd-server Generates the certificate for serving etcd front-proxy-client Generates the client for the front proxy
重新生成所有證書
[root@master ~]# kubeadm alpha certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration W0708 11:36:20.404836 6022 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed
4. 重新生成配置文件
備份舊的配置
mv /etc/kubernetes/*.conf /tmp/
生成新的配置
主要通過 kubeadm init phase kubeconfig 命令執行：
kubeadm init phase kubeconfig -h Usage: kubeadm init phase kubeconfig [flags] kubeadm init phase kubeconfig [command] Available Commands: admin Generates a kubeconfig file for the admin to use and for kubeadm itself all Generates all kubeconfig files controller-manager Generates a kubeconfig file for the controller manager to use kubelet Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes scheduler Generates a kubeconfig file for the scheduler to use
5. 後續操作
完成證書和配置文件的更新後，需要進行一系列後續操作保證更新生效，主要包括重啓 kubelet、更新管理配置。
- 重啓 kubelet
systemctl restart kubelet
- 更新 admin 配置
將新生成的 admin.conf 文件拷貝，替換 ~/.kube 目錄下的 config 文件。
cp /etc/kubernetes/admin.conf ~/.kube/config
至此master節點就恢復好了
恢復node節點
關於token失效, 重新生成
master集羣初始化後，token24小時後就會失效，如果到了token失效時間，node再加入集羣，需要重新生產token：
[root@master ~]# kubeadm token list
[root@master ~]#
token沒有了
重新生產token
[root@master ~]# kubeadm token create W0708 11:48:22.716513 15630 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] aju641.rs9sumbg24v80hld
獲取--discovery-token-ca-cert-hash值
[root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //' bb184b9cab8543539620b7736017514058ffe925ea04a8bed4c38465aaa004ae
加入集羣命令--在node節點執行
kubeadm join 192.168.122.200:6443 --token gkdzsa.xhh13svso84zie2p \ --discovery-token-ca-cert-hash sha256:bb184b9cab8543539620b7736017514058ffe925ea04a8bed4c38465aaa004ae
重啓node的kubelet服務
systemctl restart kubelet
https://blog.csdn.net/a1308422754/article/details/107157009/
https://blog.csdn.net/Ahri_J/article/details/107466921