使用kubeadm离线安装kubernetes v1.9.1

kubeadm简介

kubeadm是一个工具包，可帮助您以简单，合理，安全和可扩展的方式安装Kubernetes群集。它还支持管理Bootstrap Tokens 和升/降级集群版本。
kubeadm默认情况下并不会安装一个网络解决方案，所以用kubeadm安装完之后 需要自己来安装一个网络的插件。一般的选用插件有Calico，Flannel，Romana， Weaves Net等。

环境

1.一台或多台运行deb / rpm兼容操作系统的机器，例如Ubuntu或CentOS
以下版本为官方所支持的版本：

1. Ubuntu 16.04+
2. Debian 9
3. CentOS 7
4. RHEL 7
5. Fedora 25/26 (best-effort)
6. HypriotOS v1.0.1+
7. Container Linux (tested with 1576.4.0)

本人在Debian8上也成功安装了k8s，但是有几点需要注意，因为内核问题，在运行kubeadm init时出现missing cgroups: memory，需要更新grub，在/etc/default/grub 添加GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1" 并执行update-grub && reboot

2.每台机器有2 GB或更多的内存
3.主站上有2个以上的CPU
4.集群中所有机器的公用或专用网络都是正常的

准备

本人在debian8和centos7都安装过k8s。此教程适用于以上两个系统及debian9，其他系统若有出入请自行修改。

1.关闭swap，执行swapoff -a 关闭swap分区
2.关闭selinux，在centos中执行setenforce 0
3.安装docker
Debian

#从Ubuntu 或者Debian仓库安装Docker：

$ apt-get update
$ apt-get install -y docker.io

#从Docker的Ubuntu或Debian仓库安装Docker CE 17.03
$ apt-get update
$ apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
$ add-apt-repository \
   "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
   $(lsb_release -cs) \
   stable"
$ apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')

centos7

$ yum install -y docker
$ systemctl enable docker && systemctl start docker

因为Docker未被墙，所以这里不提供docker的离线安装包，更多相关信息请参见官方的Docker安装指南
在RHEL/CentOS 7 系统上可能会路由失败，我们需要设置一下：

$ cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl --system

4.确保kubelet使用的cgroup驱动程序与Docker使用的相同。为了确保兼容性，你可以更新Docker,如下所示：

$ cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

也可改变kubernetes的配置文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf ，将Environment=”KUBELET_CGROUP_ARGS=–cgroup-driver=systemd” 替换成Environment=”KUBELET_CGROUP_ARGS=–cgroup-driver=cgroupfs”
5.下载需要的镜像文件及kubeadm，kubelet和kubectl
从以下地址下载所需要的镜像和包。

链接: https://pan.baidu.com/s/1hukuN6O 密码: 6ehe

因为linux的包依赖关系复杂，压缩包中只包含本人安装时的所缺依赖关系，若有其他依赖请自行下载。

安装

1.安装kubeadm，kubelet和kubectl，进入到下载的deb/rpm包目录下，运行dpkg -i *.deb 或者yum localinstall *.rpm 即可安装kubeadm，kubelet和kubectl，在centos下运行sudo systemctl enable kubelet && sudo systemctl start kubelet 启动kubelet。
2.加载所需镜像，进入到下载的images目录下，运行docker load -i k8s-1-9-1.tar 即可加载所以kubernetes需要的安装包，包括flannel，运行docker load -i calicov2.tar 可将calico镜像load进去。
3.使用kubeadm初始化master，我们在初始化的时候指定一下kubernetes版本，并设置一下pod-network-cidr（后面的flannel会用到）：kubeadm init --kubernetes-version=v1.9.1 --pod-network-cidr=10.244.0.0/16 若使用calico网络请将--pod-network-cidr=10.244.0.0/16 改成--pod-network-cidr=192.168.0.0/16 ，在这个过程中kubeadm执行了一系列的操作，包括一些pre-check，生成ca证书，安装etcd和其它控制组件等。
输出应该如下所示：

[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kubeadm-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.138.0.4]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 39.511972 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node master as master by adding a label and a taint
[markmaster] Master master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: <token>
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

最下面的这行kubeadm join什么的，就是用来让别的node加入集群的，可以看出非常方便。

4.执行如下命令，让kubelet控制集群

# 对于非root用户
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 对于root用户
$ export KUBECONFIG=/etc/kubernetes/admin.conf

5.安装cni插件
安装flannel插件，执行
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
安装calico插件，执行
kubectl apply -f https://docs.projectcalico.org/v3.0/gettingstarted/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
安装完network之后，你可以通过kubectl get pods --all-namespaces来查看kube-dns是否在running来判断network是否安装成功。
默认情况下，为了保证master的安全，master是不会被调度到app的。你可以取消这个限制通过输入：

kubectl taint nodes --all node-role.kubernetes.io/master-

加入node

1.如果你有多台机器，那么请将deb/rpm包安装到node节点上，然后运行kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash> 将其中的token换成你自己的。
输出应该如下所示：

[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.138.0.4:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.138.0.4:6443"
[discovery] Requesting info from "https://10.138.0.4:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.138.0.4:6443"
[discovery] Successfully established connection with API Server "10.138.0.4:6443"
[bootstrap] Detected server version: v1.8.0
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

2.然后在master上运行kubectl get nodes 即可看见加入进来的节点。

拆除节点

要撤消kubeadm的操作，首先应该关闭节点，并确保节点在关闭之前是空的。
1.在要撤销的节点上运行

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
#<node name>为本节点的名

2.在被删除的节点上，重置所有kubeadm安装状态

kubeadm reset