一:摘要概述
Kubernetes簡稱K8S,8指代中間八個字母,希臘語舵手。大衆理解的K8S都是對容器進行管理,Docker另外含義集裝箱,命名K8S的初衷就是希望可以對這些集裝箱進行管理。有偉大願景的組織才能做出偉大的貢獻,K8S的劃時代後續正文中慢慢體會,本文將幫助小白新手搭建K8S集羣。別被集羣嚇到,直白點就是K8S運行環境,只不過多加點節點機器
二:軟硬件準備
硬件的相關要求官網上有明確指導,最低要求2核16G。其中Node即工作節點推薦4核,爲了一帆風順本文將使用超高標準操作。但是如果真是小白更建議用虛擬機操作,乾淨的配置環境對於搭建K8S集羣來講會少很多困擾
序列號 | 操作系統 | CPU | 內存 | IP地址 |
---|---|---|---|---|
1 | CentOS7 | 8 | 128 | 192.168.15.12 |
2 | CentOS7 | 8 | 128 | 192.168.15.13 |
3 | CentOS7 | 8 | 128 | 192.168.15.14 |
軟件準備無非就是安裝包,但是這裏只會闡述相關軟件版本,下載前請先執行後續步驟再下載。
序列號 | 組件名稱 | 版本 | 說明 |
---|---|---|---|
1 | kubeadm | 1.16.0 | 初始化Master節點,僅僅需要Master節點安裝 |
2 | kubelet | 1.16.0 | Node節點運行管理,Master與Node都需要安裝 |
3 | kubectl | 1.16.0 | K8S命令工具,Master與Node都可以裝上 |
安裝K8S之前每臺服務器保證安裝Docker Engine,推薦版本18.09。安裝完成後修改daemon.json文件,詳細的Docker安裝請移步Docker安裝。修改內容如下,因爲Docker默認採用systemd驅動,但是K8S採用cgroupfs驅動,需要將兩者統一。統一的方式可以修改Docker,也可以修改K8S。K8S咱還沒裝好,就統一修改Docker
// 執行命令
vim /etc/docker/daemon.json
// 修改內容
"exec-opts": [
"native.cgroupdriver=cgroupfs"
]
三:環境準備
K8S搭建需要做許多相關配置,請務必按要求操作。不要存在僥倖心理,作者的經歷就是血淋淋的教訓
3.1 關閉防火牆
爲這事兒個人絞盡腦汁想保留防火牆,按照公司規定這是必須的防護。但是,隨着K8S安裝後的應用擴展你會發現天真寫你腦門上,還是關了吧。端口太多且有些應用組件通信端口不好找
// CentOS7檢查firewall防火牆
systemctl status firewalld.service
// 關閉防火牆
systemctl stop firewalld.service
// 解除開機自啓動
systemctl disable firewalld.service
3.2 主機名解析
保證每臺機器可以根據主機名字解析出相應的IP地址,首先修改/etc/hostname
文件爲機器命名,然後修改/etc/hosts
文件綁定主機名與IP
序列號 | IP地址 | 主機名 |
---|---|---|
1 | 192.168.15.12 | k8s-master |
2 | 192.168.15.13 | k8s-node1 |
3 | 192.168.15.14 | k8s-node2 |
// 方式1:文件修改
vim /etc/hostname
// 方式2:命令修改,CentOS7提供hostnamectl系列命令。下述命令執行完後查看/etc/hostname文件你就明白了
hostnamectl set-hostname k8s-master
// 最後結果查詢,主機名有靜態名、瞬態名等,自行了解
方式一:hostname
方拾二:hostnamectl status
3.3 關閉Swap
執行命令free -h可以查看主機內存使用情況,控制檯輸出形式如下:Mem是真正的物理內存,當這個內存達到設定使用最高比例後開始使用Swap內存,Swap內存是磁盤虛擬內存,詳解請自行學習。這裏操作就是必須關閉釋放禁止使用Swap
// 臨時關閉,開機失效
swapoff -a
// 永久關閉,但都是重啓生效。機器不能重啓請先使用臨時命令操作
方式一:sed -i 's/.*swap.*/#&/' /etc/fstab
方式二:修改文件/etc/fstab,將Swap那行註釋
3.4 關閉Selinux
Selinux增強型Linux,2.6集成爲Linux模塊,Linux的安全子系統。打開可能對後續目錄掛載產生影響,部署時將其關閉
// 臨時關閉
setenforce 0
// 永久關閉
方式一:sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
方式二:vim /etc/selinux/config 和文件 vim /etc/sysconfig/selinux 將SELINUX參數設置爲disabled
3.5 關閉Dnsmasp
Dnsmasp一個網絡工具,可能導致Docker容器無法解析域名
// 關閉服務
systemctl stop dnsmasp
// 禁止開機自啓動
systemctl disable dnsmasp
3.6 設置系統參數
// 執行命令
vim /etc/sysctl.d/k8s.conf
// 文件內容
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
// 查看效果
sysctl --system
3.7 設置系統時鐘
// 調整系統時區
timedatectl set-timezone Asia/Shanghai
// UTC時鐘與硬件一致
timedatectl set-local-rtc 0
3.8 配置阿里雲K8S yum源
K8S默認的下載源是google,國外的肯定牆了過不去。所以yum下載前需要配置國內的yum源
// 執行命令
vim /etc/yum.repos.d/kubernetes.repo
// 文件內容
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
四:下載安裝
請仔細檢查是否按照要求在所有主機上執行了三步驟,確保!確保!確保!
很重要的事情提示三遍。上述三步驟一定要在所有主機上執行
4.1 Master主機操作
yum 下載安裝相關組件
// 安裝kubeadm、kubectl、kubelet
yum install -y kubeadm-1.16.0-0 kubectl-1.16.0-0 kubelet-1.16.0-0
啓動kublet應用
systemctl enable kubelet && systemctl start kubelet
初始化Master
kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.16.0 --apiserver-advertise-address 自己的Mater主機IP地址 --pod-network-cidr=10.244.0.0/16 --token-ttl 0
後續操作,爲了可以使用kubectl客戶端命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
安裝通信網絡組件flanneld,因爲官方的yml下載源在國外,國內請查看第六節的yml文件
。這個應用組件用於通信,此時未安裝使用命令kubectl get nodes
查看節點狀態發現其爲NotReady,裝完後將會變成Ready纔是正確的狀態
// 任意位置創建文件
touch kube-flanneld.yml
// 將第五節內容複製粘貼到文件中,並執行下述命令
kubectl apply -f kube-flanneld.yml
4.2 Node操作
yum 下載安裝相關組件
// 安裝kubeadm、kubectl、kubelet
yum install -y kubectl-1.16.0-0 kubelet-1.16.0-0
加入集羣,此時加入集羣的命令可以去Master節點上獲取。其實初始化完畢後的Master也會進行輸出,如果沒有記錄就去查詢即可
// Master節點執行命令,查詢加入集羣命令
kubeadm token create --print-join-command
// Node節點執行Master節點控制檯輸出內容即可
如果想在Node節點上操作kubectl命令請將Master節點的$HOME/.kube目錄下的config文件拷貝到Node節點的相同位置即可,不然會提示8080什麼的相關信息
五:踩坑小結
5.1 Node節點NotReady
做完所有操作後發現Node節點狀態任然是NotReady,此時執行命令kubectl get pods -n kube-system
發現有的狀態顯示不是Running,此時去Node節點查看並沒有對應的Docker鏡像,處理方式就是將Master節點下載好的相關鏡像save然後Node節點load完畢後即可恢復Ready狀態。因爲本人已經安裝好,所以下圖所有都是Running
5.2 網絡錯誤
錯誤輸出
ROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
修改方法
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
六:kube-flanneld.yml文件
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-ppc64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-s390x
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg