rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4196772 vs. 4194304)
現象：kubernetes集羣不可用，所有work節點離線
問題定位：
執行kubectl get node 發現work節點都是NotReady狀態
登入到work節點查看日誌發現
Nov 1 10:32:34 izwz9a75ak59utsbrrj9crz kubelet: E1101 10:32:34.119157 1669 kuberuntime_container.go:323] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4196772 vs. 4194304)
Nov 1 10:32:34 izwz9a75ak59utsbrrj9crz kubelet: E1101 10:32:34.119174 1669 generic.go:197] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4196772 vs. 4194304)
發現/var/lib/docker/containers 下有上萬個容器文件
[root@k8s-master-2 ~]# docker ps -a|wc -l
10180
該問題是有k8s的一個bug https://github.com/kubernetes/kubernetes/issues/63858
解決方法：
登入到work節點清除不用的容器殘留
docker system prune
重啓docker和kubelet
systemctl restart docker&&systemctl restart kubelet
轉自：https://blog.csdn.net/u011181610/article/details/83623286