k8s 一台master，两台node节点安装过程记录如下
一、安装环境
[root@k8sm ~]# cat /proc/version
Linux version 3.10.0-1062.12.1.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Tue Feb 4 23:02:59 UTC 2020
二、安装规划
docker版本：k8s v1.16.0 推荐的是docker-ce 18.09.9 这个版本，可以看下，k8s对应的docker版本，否则安装的时候会有告警。
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.03.1-ce. Latest validated version: 18.09
注意机器必须为2C，2g以上，否则安装的时候会报错
[root@k8sm ~]# docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:20:16 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:23:58 2018
OS/Arch: linux/amd64
Experimental: false
[root@k8sm ~]#
master： 192.168.47.129 k8sm
node1：192.168.47.132 k8ss1
node2：192.168.47.133 k8ss2
k8s版本：v1.16.0
三、安装前工作
1.确认是否已配置静态IP
2.修改hostname
3.安装docker
4.配置k8s安装环境
（1）关闭防火墙
systemctl disable firewalld
systemctl stop firewalld
（2）关闭selinux
setenforce 0
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
（3）禁用swap
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
（4）修改内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
（5）重新加载配置文件
sysctl --system
（6）配置阿里的k8s yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
四、安装kubeadm、kubelet、kubectl（仅master需要安装）
master：
yum install -y kubectl-1.16.0-0 kubeadm-1.16.0-0 kubelet-1.16.0-0
node：
yum install -y kubeadm-1.16.0-0 kubelet-1.16.0-0
安装完成后，重启k8s服务：
systemctl enable kubelet && systemctl start kubelet
五、初始化k8s
（1）master上：
使用阿里云的源registry.aliyuncs.com/google_containers
初始化k8s
kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.16.0 --apiserver-advertise-address 192.168.47.129 --pod-network-cidr=10.244.0.0/16 --token-ttl 0
–apiserver-advertise-address 192.168.47.129 根据master的ip修改
初始化安装完成后根据提示输入命令：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
获取node加入集群的命令
kubeadm token create --print-join-command
（2）node上
执行master中“kubeadm token create --print-join-command” 获取的命令
（3）验证
加入集群后，在master上验证。三台机器已在集群中（status为notready后续安装flannel）
[root@k8sm tmp]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8sm NotReady master 137m v1.16.0
k8ss1 NotReady <none> 96m v1.16.0
k8ss2 NotReady <none> 95m v1.16.0
六、安装flannel（master上）
已将kube-flannel.yml中的quay.io，修改为国内可以访问的地址quay-mirror.qiniu.com
配置文件放网盘了需要的自取：
链接：https://pan.baidu.com/s/18La-DOb9-iVLtqldMIh26g
提取码：q4iv
或者能翻墙的也可以直接wget：
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
安装fannel
kubectl apply -f kube-flannel.yml
安装完后验证：
[root@k8sm tmp]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-6l2bd 1/1 Running 0 177m
kube-system coredns-58cc8c89f4-lfr64 1/1 Running 0 177m
kube-system etcd-k8sm 1/1 Running 0 176m
kube-system kube-apiserver-k8sm 1/1 Running 0 176m
kube-system kube-controller-manager-k8sm 1/1 Running 0 176m
kube-system kube-flannel-ds-amd64-fg7qq 1/1 Running 0 17m
kube-system kube-flannel-ds-amd64-gmhhz 1/1 Running 0 17m
kube-system kube-flannel-ds-amd64-lzm74 1/1 Running 0 17m
kube-system kube-proxy-29p5w 1/1 Running 0 177m
kube-system kube-proxy-9bgr4 1/1 Running 0 135m
kube-system kube-proxy-gzhmw 1/1 Running 0 136m
kube-system kube-scheduler-k8sm 1/1 Running 0 176m
[root@k8sm tmp]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8sm Ready master 177m v1.16.0
k8ss1 Ready <none> 135m v1.16.0
k8ss2 Ready <none> 134m v1.16.0
三节点的status为ready，且pod均为runing,至此安装完成
七、安装过程中遇到的问题
1.没有禁用swap
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
没有禁用swap，禁用swap后解决
2.初始化的过程中遇到：
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
解决方法：
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
3.在装完flannel后 kubectl get nodes检查发现有一个节点还是notready
[root@k8sm tmp]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8sm Ready master 161m v1.16.0
k8ss1 NotReady <none> 120m v1.16.0
k8ss2 Ready <none> 119m v1.16.0
检查 pod发现：kube-system kube-flannel-ds-amd64-fg7qq 0/1 Init:ErrImagePull 0 79s
有报 Init:ErrImagePull的错误
[root@k8sm tmp]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-6l2bd 1/1 Running 0 161m
kube-system coredns-58cc8c89f4-lfr64 1/1 Running 0 161m
kube-system etcd-k8sm 1/1 Running 0 160m
kube-system kube-apiserver-k8sm 1/1 Running 0 160m
kube-system kube-controller-manager-k8sm 1/1 Running 0 160m
kube-system kube-flannel-ds-amd64-fg7qq 0/1 Init:ErrImagePull 0 79s
kube-system kube-flannel-ds-amd64-gmhhz 1/1 Running 0 79s
kube-system kube-flannel-ds-amd64-lzm74 1/1 Running 0 79s
kube-system kube-proxy-29p5w 1/1 Running 0 161m
kube-system kube-proxy-9bgr4 1/1 Running 0 119m
kube-system kube-proxy-gzhmw 1/1 Running 0 120m
kube-system kube-scheduler-k8sm 1/1 Running 0 160m
通过网上各种搜索，可能是Kubernetes无法找到本地的flannel镜像导致的。
分别在三台机器上执行docker images|grep fla
[root@k8sm tmp]# docker images|grep fla
quay-mirror.qiniu.com/coreos/flannel v0.11.0-amd64 ff281650a721 13 months ago 52.6MB
[root@k8ss1 ~]# docker images|grep fla
[root@k8ss2 ~]# docker images|grep fla
quay-mirror.qiniu.com/coreos/flannel v0.11.0-amd64 ff281650a721 13 months ago 52.6MB
发现确实k8ss1上没有这个本地镜像
直接在k8ss2上导出进行到共享文件夹并在k8ss1上导入
[root@k8ss2 mnt]# docker save -o /mnt/fannel.tar.gz quay-mirror.qiniu.com/coreos/flannel
[root@k8ss1 mnt]# docker load --input ops.v1.tar.gz
最后再检查下，状态都为running和ready