K8S之路(一) -- Kubeadm搭建K8S集群

一：摘要概述

Kubernetes简称K8S，8指代中间八个字母，希腊语舵手。大众理解的K8S都是对容器进行管理，Docker另外含义集装箱，命名K8S的初衷就是希望可以对这些集装箱进行管理。有伟大愿景的组织才能做出伟大的贡献，K8S的划时代后续正文中慢慢体会，本文将帮助小白新手搭建K8S集群。别被集群吓到，直白点就是K8S运行环境，只不过多加点节点机器

二：软硬件准备

硬件的相关要求官网上有明确指导，最低要求2核16G。其中Node即工作节点推荐4核，为了一帆风顺本文将使用超高标准操作。但是如果真是小白更建议用虚拟机操作，干净的配置环境对于搭建K8S集群来讲会少很多困扰

序列号	操作系统	CPU	内存	IP地址
1	CentOS7	8	128	192.168.15.12
2	CentOS7	8	128	192.168.15.13
3	CentOS7	8	128	192.168.15.14

软件准备无非就是安装包，但是这里只会阐述相关软件版本，下载前请先执行后续步骤再下载。

序列号	组件名称	版本	说明
1	kubeadm	1.16.0	初始化Master节点，仅仅需要Master节点安装
2	kubelet	1.16.0	Node节点运行管理，Master与Node都需要安装
3	kubectl	1.16.0	K8S命令工具，Master与Node都可以装上

安装K8S之前每台服务器保证安装Docker Engine，推荐版本18.09。安装完成后修改daemon.json文件，详细的Docker安装请移步Docker安装。修改内容如下，因为Docker默认采用systemd驱动，但是K8S采用cgroupfs驱动，需要将两者统一。统一的方式可以修改Docker，也可以修改K8S。K8S咱还没装好，就统一修改Docker

// 执行命令
vim /etc/docker/daemon.json

// 修改内容
 "exec-opts": [
        "native.cgroupdriver=cgroupfs"
    ]

三：环境准备

K8S搭建需要做许多相关配置，请务必按要求操作。不要存在侥幸心理，作者的经历就是血淋淋的教训

3.1 关闭防火墙

为这事儿个人绞尽脑汁想保留防火墙，按照公司规定这是必须的防护。但是，随着K8S安装后的应用扩展你会发现天真写你脑门上，还是关了吧。端口太多且有些应用组件通信端口不好找

// CentOS7检查firewall防火墙
systemctl status firewalld.service

// 关闭防火墙
systemctl stop firewalld.service

// 解除开机自启动
systemctl disable firewalld.service

3.2 主机名解析

保证每台机器可以根据主机名字解析出相应的IP地址，首先修改/etc/hostname文件为机器命名，然后修改/etc/hosts文件绑定主机名与IP

序列号	IP地址	主机名
1	192.168.15.12	k8s-master
2	192.168.15.13	k8s-node1
3	192.168.15.14	k8s-node2

// 方式1：文件修改
vim /etc/hostname

// 方式2：命令修改,CentOS7提供hostnamectl系列命令。下述命令执行完后查看/etc/hostname文件你就明白了
hostnamectl set-hostname k8s-master

// 最后结果查询,主机名有静态名、瞬态名等，自行了解
方式一：hostname
方拾二：hostnamectl status

3.3 关闭Swap

执行命令free -h可以查看主机内存使用情况，控制台输出形式如下：Mem是真正的物理内存，当这个内存达到设定使用最高比例后开始使用Swap内存，Swap内存是磁盘虚拟内存，详解请自行学习。这里操作就是必须关闭释放禁止使用Swap

// 临时关闭,开机失效
swapoff -a

// 永久关闭，但都是重启生效。机器不能重启请先使用临时命令操作
方式一：sed -i 's/.*swap.*/#&/' /etc/fstab
方式二：修改文件/etc/fstab，将Swap那行注释

3.4 关闭Selinux

Selinux增强型Linux，2.6集成为Linux模块，Linux的安全子系统。打开可能对后续目录挂载产生影响，部署时将其关闭

// 临时关闭
setenforce 0

// 永久关闭
方式一：sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
	   sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

方式二：vim /etc/selinux/config 和文件 vim /etc/sysconfig/selinux 将SELINUX参数设置为disabled

3.5 关闭Dnsmasp

Dnsmasp一个网络工具，可能导致Docker容器无法解析域名

// 关闭服务
systemctl stop dnsmasp

// 禁止开机自启动
systemctl disable dnsmasp

3.6 设置系统参数

// 执行命令
vim /etc/sysctl.d/k8s.conf 

// 文件内容
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100

// 查看效果
sysctl --system

3.7 设置系统时钟

// 调整系统时区
timedatectl set-timezone Asia/Shanghai

// UTC时钟与硬件一致
timedatectl set-local-rtc 0

3.8 配置阿里云K8S yum源

K8S默认的下载源是google，国外的肯定墙了过不去。所以yum下载前需要配置国内的yum源

// 执行命令 
vim /etc/yum.repos.d/kubernetes.repo

// 文件内容
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

四：下载安装

请仔细检查是否按照要求在所有主机上执行了三步骤，确保！确保！确保！很重要的事情提示三遍。上述三步骤一定要在所有主机上执行

4.1 Master主机操作

yum 下载安装相关组件

// 安装kubeadm、kubectl、kubelet
yum install -y kubeadm-1.16.0-0 kubectl-1.16.0-0 kubelet-1.16.0-0

启动kublet应用

systemctl enable kubelet && systemctl start kubelet

初始化Master

kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.16.0 --apiserver-advertise-address 自己的Mater主机IP地址 --pod-network-cidr=10.244.0.0/16 --token-ttl 0

后续操作，为了可以使用kubectl客户端命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装通信网络组件flanneld，因为官方的yml下载源在国外，国内请查看第六节的yml文件。这个应用组件用于通信，此时未安装使用命令kubectl get nodes 查看节点状态发现其为NotReady，装完后将会变成Ready才是正确的状态

// 任意位置创建文件
touch kube-flanneld.yml

// 将第五节内容复制粘贴到文件中，并执行下述命令
kubectl apply -f kube-flanneld.yml

4.2 Node操作

yum 下载安装相关组件

// 安装kubeadm、kubectl、kubelet
yum install -y kubectl-1.16.0-0 kubelet-1.16.0-0

加入集群，此时加入集群的命令可以去Master节点上获取。其实初始化完毕后的Master也会进行输出，如果没有记录就去查询即可

// Master节点执行命令，查询加入集群命令
kubeadm token create --print-join-command

// Node节点执行Master节点控制台输出内容即可

如果想在Node节点上操作kubectl命令请将Master节点的$HOME/.kube目录下的config文件拷贝到Node节点的相同位置即可，不然会提示8080什么的相关信息

五：踩坑小结

5.1 Node节点NotReady

做完所有操作后发现Node节点状态任然是NotReady，此时执行命令kubectl get pods -n kube-system发现有的状态显示不是Running，此时去Node节点查看并没有对应的Docker镜像，处理方式就是将Master节点下载好的相关镜像save然后Node节点load完毕后即可恢复Ready状态。因为本人已经安装好，所以下图所有都是Running

5.2 网络错误

错误输出

ROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

修改方法

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

六：kube-flanneld.yml文件

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-ppc64le
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - ppc64le
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-s390x
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - s390x
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg