k8s 1.17.2高可用版本重启问题
- 开发环境平均 每10分钟重启一次。
- 测试环境6天平均 每天重启一次。
- cube环境 每小时重启一次。
- ramdisk 到目前18小时没有重启
1.测试的环境：192.168.1.135 密码：xxxx 配置：3master 8c32g etcd版本：3.2.20
1.1重启次数：
[root@portal135 ~]# kubectl get pods -n kube-system|grep "kube-.*bocloud.com"
kube-apiserver-portal134.bocloud.com 1/1 Running 2 5d23h
kube-apiserver-portal135.bocloud.com 1/1 Running 8 5d23h
kube-apiserver-portal136.bocloud.com 1/1 Running 3 5d23h
kube-controller-manager-portal134.bocloud.com 1/1 Running 2 5d23h
kube-controller-manager-portal135.bocloud.com 1/1 Running 7 5d23h
kube-controller-manager-portal136.bocloud.com 1/1 Running 5 5d23h
kube-scheduler-portal134.bocloud.com 1/1 Running 2 5d23h
kube-scheduler-portal135.bocloud.com 1/1 Running 7 5d23h
kube-scheduler-portal136.bocloud.com 1/1 Running 5 5d23h
1.2响应时间：
[root@portal134 upload]# export NODE_IPS="192.168.1.134 192.168.1.135 192.168.1.136"
[root@portal134 upload]# for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/etcd/ssl/ca.crt --cert=/etc/etcd/ssl/client.crt --key=/etc/etcd/ssl/client.key endpoint health; done
https://192.168.1.134:2379 is healthy: successfully committed proposal: took = 2.711272ms
https://192.168.1.135:2379 is healthy: successfully committed proposal: took = 2.089683ms
https://192.168.1.136:2379 is healthy: successfully committed proposal: took = 1.935061ms
2.开发环境：192.168.2.103 密码：xxxx 配置：3master 4c8g etcd版本：3.4.3
2.1重启次数：
[root@boc-108 ~]# kubectl get pods -n kube-system|grep "kube-.*.dev"
kube-apiserver-boc-103.dev 1/1 Running 63 4d4h
kube-apiserver-boc-104.dev 1/1 Running 67 4d4h
kube-apiserver-boc-108.dev 1/1 Running 82 4d4h
kube-controller-manager-boc-103.dev 1/1 Running 543 4d4h
kube-controller-manager-boc-104.dev 1/1 Running 561 4d4h
kube-controller-manager-boc-108.dev 1/1 Running 558 4d4h
kube-scheduler-boc-103.dev 1/1 Running 562 4d4h
kube-scheduler-boc-104.dev 1/1 Running 556 4d4h
kube-scheduler-boc-108.dev 1/1 Running 561 4d4h
2.2响应时间：
[root@boc-103 test]# export NODE_IPS="192.168.2.103 192.168.2.104 192.168.2.108"
[root@boc-103 test]# for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/etcd/ssl/ca.crt --cert=/etc/etcd/ssl/client.crt --key=/etc/etcd/ssl/client.key endpoint health; done
https://192.168.2.103:2379 is healthy: successfully committed proposal: took = 1.318040913s
https://192.168.2.104:2379 is healthy: successfully committed proposal: took = 1.557571675s
https://192.168.2.108:2379 is healthy: successfully committed proposal: took = 54.749774ms
3. 错误现象：
kube-controller-manager和kube-scheduler重启原因etcdserver: request timed out
E0224 03:54:36.433286 1 cronjob_controller.go:125] Failed to extract job list: etcdserver: request timed out
E0224 03:54:37.592575 1 leaderelection.go:331] error retrieving resource lock kube-system/kube-controller-manager: etcdserver: request timed out
I0224 03:54:38.047024 1 leaderelection.go:288] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition
I0224 03:54:38.047089 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' portal135.bocloud.com_46d96c59-fa2a-43d3-aa0e-4c969a287338 stopped leading
F0224 03:54:38.047158 1 controllermanager.go:279] leaderelection lost
apiserver也出现多次
E0224 03:54:37.591267 1 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}
4.cube环境 10.10.5.8 密码：xxxx 配置：3master 8c16g
4.1重启次数：
[root@master ~]# kubectl get pods -n kube-system|grep .novalocal
kube-apiserver-master.novalocal 1/1 Running 1 8h
kube-apiserver-node-1.novalocal 1/1 Running 1 8h
kube-apiserver-node-2.novalocal 1/1 Running 1 8h
kube-controller-manager-master.novalocal 1/1 Running 7 8h
kube-controller-manager-node-1.novalocal 1/1 Running 8 8h
kube-controller-manager-node-2.novalocal 1/1 Running 6 8h
kube-scheduler-master.novalocal 1/1 Running 8 8h
kube-scheduler-node-1.novalocal 1/1 Running 6 8h
kube-scheduler-node-2.novalocal 1/1 Running 8 8h
4.2响应时间：
https://10.10.5.8:2379 is healthy: successfully committed proposal: took = 15.970648ms
https://10.10.5.48:2379 is healthy: successfully committed proposal: took = 13.325127ms
https://10.10.5.38:2379 is healthy: successfully committed proposal: took = 18.190006ms
5.解决办法
5.1 扩容到32g6c，但io操作依旧很慢，关机在于硬盘的读写速度。
5.2 做一个ramdisk。把内存挂载到/var/lib/etcd下当做硬盘存储etcd数据。
mkdir /var/lib/etcd && mount -t tmpfs -o size=2G tmpfs /var/lib/etcd && echo "tmpfs /var/lib/etcd tmpfs defaults,size=2G 0 0" >> /etc/fstab
5.2.1 io操作对比结果：
- ramdisk 2.3 GB/s
- 数据盘 24.0 MB/s
[root@node-139 ~]# dd bs=1M count=1000 if=/dev/zero of=/media/tmp/a.txt conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.457043 s, 2.3 GB/s
[root@node-139 ~]# dd bs=1M count=1000 if=/dev/zero of=/home/a.txt conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 43.7001 s, 24.0 MB/s
注：/media/tmp是挂载ramdisk的路径
5.3 验证结果：etcd的响应速度有明显提升。
[root@node-218 ~]# for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/etcd/ssl/ca.crt --cert=/etc/etcd/ssl/client.crt --key=/etc/etcd/ssl/client.key endpoint health; done
https://192.168.2.218:2379 is healthy: successfully committed proposal: took = 11.960397ms
https://192.168.2.219:2379 is healthy: successfully committed proposal: took = 10.482817ms
https://192.168.2.220:2379 is healthy: successfully committed proposal: took = 10.786569ms
[root@node-218 ~]# kubectl get pods -n kube-system|grep .dev
kube-apiserver-node-218.dev 1/1 Running 0 10m
kube-apiserver-node-219.dev 1/1 Running 0 10m
kube-apiserver-node-220.dev 1/1 Running 0 11m
kube-controller-manager-node-218.dev 1/1 Running 1 10m
kube-controller-manager-node-219.dev 1/1 Running 1 10m
kube-controller-manager-node-220.dev 1/1 Running 1 11m
kube-scheduler-node-218.dev 1/1 Running 1 10m
kube-scheduler-node-219.dev 1/1 Running 0 11m
kube-scheduler-node-220.dev 1/1 Running 1 11m
[root@node-218 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 28G 3.5G 25G 13% /
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 0 5.8G 0% /dev/shm
tmpfs 5.8G 34M 5.8G 1% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/sda1 1014M 145M 870M 15% /boot
tmpfs 1.2G 0 1.2G 0% /run/user/0
tmpfs 2.0G 125M 1.9G 7% /var/lib/etcd
192.168.2.214:/opt/share 28G 2.5G 26G 9% /abcsys/upload
5.4 经过18小时后0次重启
[root@node-218 ~]# kubectl get pods -n kube-system|grep dev
kube-apiserver-node-218.dev 1/1 Running 0 18h
kube-apiserver-node-219.dev 1/1 Running 0 18h
kube-apiserver-node-220.dev 1/1 Running 0 18h
kube-controller-manager-node-218.dev 1/1 Running 1 18h
kube-controller-manager-node-219.dev 1/1 Running 1 18h
kube-controller-manager-node-220.dev 1/1 Running 1 18h
kube-scheduler-node-218.dev 1/1 Running 1 18h
kube-scheduler-node-219.dev 1/1 Running 0 18h
kube-scheduler-node-220.dev 1/1 Running 1 18h
5.5 使用ramdisk纯在的问题
由于数据是存储在内存中的，断电即丢失。
6 总结
开发环境需要提升硬盘的读写速度，现在的硬盘读写熟读只有24.0 MB/s，导致etcd数据库经常出现request timeout，健康检查超时，服务重启等问题。