kubeadm部署Kubernetes 1.24版本集羣，已踩坑

原創2022-06-06 11:11·運維小弟

一、Kubernetes 1.24版本發佈及改動

1.1 Kubernetes 1.24 發佈

2022 年 5 月 3 日，Kubernetes 1.24 正式發佈，在新版本中，我們看到 Kubernetes 作爲容器編排的事實標準，正愈發變得成熟，有 12 項功能都更新到了穩定版本，同時引入了很多實用的功能，例如 StatefulSets 支持批量滾動更新，NetworkPolicy 新增 NetworkPolicyStatus 字段方便進行故障排查等

1.2 Kubernetes 1.24 改動

Kubernetes 正式移除對 Dockershim 的支持，討論很久的 “棄用 Dockershim” 也終於在這個版本畫上了句號。

想要清楚地瞭解docker 和 k8s 的關係，可以參考下這篇文章： https://i4t.com/5435.html

Kubernetes1.24 之前：

Kubernetes1.24 之後：

如還想繼續在k8s中使用docker，需要自行安裝cri-dockerd 組件；不然就使用containerd

二、Kubernetes 1.24版本集羣部署

2.1 Kubernetes 1.24版本集羣部署環境準備

2.1.1 主機操作系統說明

序號	操作系統及版本	備註
1	CentOS Linux release 7.9.2009

2.1.2 主機硬件配置說明

CPU	內存	硬盤	角色	IP地址	主機名
4C	8G	100GB	master	172.16.200.30	k8s-master01
4C	8G	100GB	node	172.16.200.31	k8s-node1
4C	8G	100GB	node	172.16.200.32	k8s-node2

2.1.3 主機配置

2.1.3.1 主機名配置

由於本次使用3臺主機完成kubernetes集羣部署，其中1臺爲master節點,名稱爲k8s-master01;其中2臺爲node節點，名稱分別爲：k8s-node1及k8s-node2

master節點
# hostnamectl set-hostname k8s-master01

node1節點
# hostnamectl set-hostname k8s-node1

node2節點
# hostnamectl set-hostname k8s-node2

2.1.3.2 主機名與IP地址解析

所有集羣主機均需要進行配置。

cat >> /etc/hosts <<EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.200.30 k8s-master01
172.16.200.31 k8s-node1
172.16.200.32 k8s-node2
EOF

2.1.3.3 關閉SWAP分區

修改完成後需要重啓操作系統，如不重啓，可臨時關閉，命令爲swapoff -a

#臨時關閉
# swapoff -a


#永遠關閉swap分區，需要重啓操作系統
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

2.1.3.4 防火牆配置

所有主機均需要操作。

關閉現有防火牆firewalld
# systemctl disable firewalld
# systemctl stop firewalld
# firewall-cmd --state
not running

2.1.3.5 SELINUX配置

所有主機均需要操作。修改SELinux配置需要重啓操作系統。

#臨時關閉
#setenforce 0

#永久生效
# sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

2.1.3.6 時間同步配置

所有主機均需要操作。最小化安裝系統需要安裝ntpdate軟件。

# crontab -l
0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com


#設置上海時區，東八區
# timedatectl set-timezone Asia/Shanghai

2.1.3.7 升級操作系統內核

所有主機均需要操作。

導入elrepo gpg key
$ rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

安裝elrepo YUM源倉庫
$ yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

安裝kernel-ml版本，ml爲長期穩定版本，lt爲長期維護版本
$ yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64

設置grub2默認引導爲0
$ grub2-set-default 0

重新生成grub2引導文件
$ grub2-mkconfig -o /boot/grub2/grub.cfg

更新後，需要重啓，使用升級的內核生效。
$ reboot

重啓後，需要驗證內核是否爲更新對應的版本
$ uname -r

2.1.3.8 配置內核轉發及網橋過濾

所有主機均需要操作。

添加網橋過濾及內核轉發配置文件

cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

#使之生效
$ sysctl -p /etc/sysctl.d/k8s.conf

#加載br_netfilter模塊
# modprobe br_netfilter

#查看是否加載
# lsmod | grep br_netfilter
br_netfilter           22256  0

2.1.3.9 安裝ipset及ipvsadm

所有主機均需要操作。

安裝ipset及ipvsadm
$ yum -y install ipset ipvsadm

#配置ipvsadm模塊加載方式.添加需要加載的模塊

cat > /etc/sysconfig/modules/ipvs.module <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack
EOF

授權、運行、檢查是否加載
chmod 755 /etc/sysconfig/modules/ipvs.module &&  /etc/sysconfig/modules/ipvs.module

2.2 Docker準備

2.2.1 Docker安裝環境準備

準備一塊單獨的磁盤，建議單獨把/var/lib/docker 掛載在一個單獨的磁盤上，所有主機均需要操作。

#格式化磁盤
$ mkfs.ext4 /dev/sdb

#創建docker工作目錄
$ mkdir /var/lib/docker

#寫入掛載信息到fstab中，永久掛載
$ echo "/dev/sdb /var/lib/docker ext4 defaults 0 0" >>  /etc/fstab

#使fstab掛載生效
$ mount -a

#查看磁盤掛載
$ df -h /dev/sdb


# 安裝一些必要工具
$ yum install -y yum-utils device-mapper-persistent-data lvm2

2.2.2 Docker安裝YUM源準備

使用阿里雲開源軟件鏡像站。

$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

2.2.2 Docker安裝

## 查看所有的可用版本
$ yum list docker-ce --showduplicates | sort -r

#安裝舊版本 yum install docker-ce-cli-19.03.15-3.el7  docker-ce-19.03.15-3.el7

# 安裝源裏最新版本
$ yum install docker-ce

2.2.3 啓動Docker服務

$  systemctl enable --now docker
$  systemctl start docker

2.2.4 配置docker加速，修改cgroup方式

/etc/docker/daemon.json 默認沒有此文件，需要單獨創建

在/etc/docker/daemon.json添加如下內容

tee /etc/docker/daemon.json <<-'EOF'
{                     
  "registry-mirrors" : [
    "http://hub-mirror.c.163.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# 啓動docker
    $ systemctl enable docker && systemctl restart docker

2.2.5 cri-dockerd安裝

2.2.5.1 下載cri-dockerd 二進制文件

項目地址：
https://github.com/Mirantis/cri-dockerd

下載
cri-dockerd-0.2.1.amd64.tgz 二進制版本

所有節點都安裝 cri-dockerd

# 拷貝二進制文件

# tar -xf cri-dockerd-0.2.1.amd64.tgz 
# cp cri-dockerd/cri-dockerd /usr/bin/
# chmod +x /usr/bin/cri-dockerd


# 配置啓動文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify

ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

EOF





# 生成socket 文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

EOF

【也可以直接下載
https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd 注意，需要修改cri-docker.service 中 ExecStart 啓動參數，這裏/usr/bin/cri-dockerd一定要加上參數--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7用來指定所用的pause鏡像是哪個，否則默認拉取k8s.gcr.io/pause:3.6，會導致安裝失敗。

2.2.5.2 啓動cri-docker

systemctl daemon-reload
systemctl start cri-docker
systemctl enable cri-docker
systemctl status cri-docker

2.3 kubernetes 1.24.1 集羣部署

2.3.1 集羣軟件及版本說明

截至本文發稿，2022.6.6 最新版是1.24.1版本

	kubeadm	kubelet	kubectl
版本	1.24.1	1.24.1	1.24.1
安裝位置	集羣所有主機	集羣所有主機	集羣所有主機
作用	初始化集羣、管理集羣等	用於接收api-server指令，對pod生命週期進行管理	集羣應用命令行管理工具

2.3.2 kubernetes YUM源準備

2.3.2.1 谷歌YUM源 [國外主機]

$  cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

2.3.2.2 阿里雲YUM源【國內主機】

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

$ yum clean all && yum makecache

2.3.3 集羣軟件安裝

所有節點均可安裝

# 查看所有的可用版本
$  yum list  kubeadm  kubelet kubectl --showduplicates | sort -r



# 默認安裝的版本就是最新版1.24.X，當然也可以指定版本安裝 ，如 yum install kubelet-1.16.2 kubeadm-1.16.2 kubectl-1.16.2
$ yum install  kubeadm  kubelet kubectl

#安裝後查看版本
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:38Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}


設置kubelet爲開機自啓動即可，由於沒有生成配置文件，集羣初始化後自動啓動
$ systemctl enable kubelet

2.3.4 配置kubelet

爲了實現docker使用的cgroupdriver與kubelet使用的cgroup的一致性，建議修改如下文件內容。

$ cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

2.3.5 初始化配置文件

只在master節點（k8s-master01）執行

$ kubeadm config print init-defaults > kubeadm.yaml

修改配置文件

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.16.200.30
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/cri-docker.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master01
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.24.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.224.0.0/16 # pod子網
scheduler: {}



---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

2.3.6 集羣鏡像準備

只在master節點（k8s-master01）執行

 # 查看需要使用的鏡像列表,若無問題，將得到如下列表
$ kubeadm config images list --config kubeadm.yaml 
registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
registry.aliyuncs.com/google_containers/pause:3.7
registry.aliyuncs.com/google_containers/etcd:3.5.3-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6


# 提前下載鏡像到本地
$ kubeadm config images pull --config kubeadm.yaml 
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6


$ docker images
REPOSITORY                                                        TAG       IMAGE ID       CREATED        SIZE
registry.aliyuncs.com/google_containers/kube-apiserver            v1.24.1   e9f4b425f919   7 days ago     130MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.24.1   b4ea7e648530   7 days ago     119MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.24.1   18688a72645c   7 days ago     51MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.24.1   beb86f5d8e6c   7 days ago     110MB
registry.aliyuncs.com/google_containers/etcd                      3.5.3-0   aebe758cef4c   6 weeks ago    299MB
registry.aliyuncs.com/google_containers/pause                     3.7       221177c6082a   2 months ago   711kB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6    a4ca41631cc7   7 months ago   46.8MB

備註：離線環境下載鏡像，可使用docker-save 實現

# cat image_download.sh
#!/bin/
images_list='
registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
registry.aliyuncs.com/google_containers/pause:3.7
registry.aliyuncs.com/google_containers/etcd:3.5.3-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6

for i in $images_list
do
        docker pull $i
done

docker save -o k8s-1-24-1.tar $images_list

2.3.7 集羣初始化

只在master節點（k8s-master01）執行

kubeadm init --config=kubeadm.yaml

# 重置 如果有需要，必須要指定--cri-socket，不然會報錯

kubeadm reset --cri-socket unix:///var/run/cri-docker.sock

初始化過程輸出
[root@localhost ~]# kubeadm config images pull --config=init.default.yaml --v=5
I0604 10:03:37.133902    2673 initconfiguration.go:255] loading configuration from "init.default.yaml"
I0604 10:03:37.136354    2673 kubelet.go:214] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6
[root@localhost ~]# kubeadm init --config=init.default.yaml
[init] Using Kubernetes version: v1.24.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-1] and IPs [10.96.0.1 172.16.201.30]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master-1] and IPs [172.16.201.30 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master-1] and IPs [172.16.201.30 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.503761 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.200.30:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:8a55d1074d4d74804ee493119a94902d816e2b185444b19398353585a1588120

2.3.7 集羣應用客戶端管理集羣文件準備

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master01 ~]# ls /root/.kube/
config

[root@k8s-master01 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf

2.3.8 集羣網絡準備

2.3.8.1 calico安裝

wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate


vim  calico.yaml 
................
  name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
................

[root@kubesphere ~]# kubectl apply -f calico.yaml

監視kube-system命名空間中pod運行情況
[root@k8s-master01 ~]# watch kubectl get pods -n kube-system

已經全部運行
[root@k8s-master01 ~]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-56cdb7c587-szkjr   1/1     Running   0             11m
calico-node-6xzg7                          1/1     Running   0             11m
coredns-74586cf9b6-bbhq6                   1/1     Running   2             35m
coredns-74586cf9b6-g6shr                   1/1     Running   2             35m
etcd-master-1                              1/1     Running   3             35m
kube-apiserver-master-1                    1/1     Running   3             35m
kube-controller-manager-master-1           1/1     Running   2             35m
kube-proxy-bbb2t                           1/1     Running   2             35m
kube-scheduler-master-1                    1/1     Running   2             35m

2.3.8.2 calico客戶端安裝

下載二進制文件
# curl -L https://github.com/projectcalico/calico/releases/download/v3.23.1/calicoctl-linux-amd64 -o calicoctl

安裝calicoctl
# mv calicoctl /usr/bin/

爲calicoctl添加可執行權限
# chmod +x /usr/bin/calicoctl

查看添加權限後文件
# ls /usr/bin/calicoctl
/usr/bin/calicoctl

查看calicoctl版本
# calicoctl  version
Client Version:    v3.23.1
Git commit:        967e24543
Cluster Version:   v3.23.1
Cluster Type:      k8s,bgp,kubeadm,kdd

通過~/.kube/config連接kubernetes集羣，查看已運行節點
$ DATASTORE_TYPE=kubernetes 
$ KUBECONFIG=~/.kube/config 
$ calicoctl get nodes
NAME
k8s-master01

2.3.9 集羣工作節點添加

因容器鏡像下載較慢，可能會導致報錯，主要錯誤爲沒有準備好cni（集羣網絡插件），如有網絡，請耐心等待即可。

[root@k8s-node1 ~]# kubeadm join 172.16.200.30:6443 --token 8x4o2u.hslo8xzwwlrncr8s \                              --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8s-node2 ~]# kubeadm join 172.16.200.30:6443 --token 8x4o2u.hslo8xzwwlrncr8s \
        --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock

注意：必須要加上 --cri-socket
unix:///var/run/cri-dockerd.sock ，不然會報錯

在master節點上操作，查看網絡節點是否添加
# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes
NAME
k8s-master01
k8s-node1
k8s-node2

2.3.10 驗證集羣可用性

查看所有的節點
[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   12h   v1.24.1
k8s-node1   Ready    <none>          12h   v1.24.1
k8s-node2   Ready    <none>          12h   v1.24.1

查看集羣健康情況
[root@k8s-master01 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

三、參考

https://www.bilibili.com/video/BV1uY411c7qU?p=3&spm_id_from=333.880.my_history.page.click
https://www.jianshu.com/p/a613f64ccab6
https://i4t.com/5435.html
https://www.bilibili.com/video/BV1gS4y1B7Ut?spm_id_from=333.880.my_history.page.click

轉自

kubeadm部署Kubernetes 1.24版本集羣，已踩坑-今日頭條
https://www.toutiao.com/article/7105957860210819623/?log_from=d253ecbcab6e2_1710747136432

kubeadm部署Kubernetes 1.24版本集羣【轉】

kubeadm部署Kubernetes 1.24版本集羣，已踩坑

一、Kubernetes 1.24版本發佈及改動

1.1 Kubernetes 1.24 發佈

1.2 Kubernetes 1.24 改動

二、Kubernetes 1.24版本集羣部署

2.1 Kubernetes 1.24版本集羣部署環境準備

2.1.1 主機操作系統說明

2.1.2 主機硬件配置說明

2.1.3 主機配置

2.1.3.1 主機名配置

2.1.3.2 主機名與IP地址解析

2.1.3.3 關閉SWAP分區

2.1.3.4 防火牆配置

2.1.3.5 SELINUX配置

2.1.3.6 時間同步配置

2.1.3.7 升級操作系統內核

2.1.3.8 配置內核轉發及網橋過濾

2.1.3.9 安裝ipset及ipvsadm

2.2 Docker準備

2.2.1 Docker安裝環境準備

2.2.2 Docker安裝YUM源準備

2.2.2 Docker安裝

2.2.3 啓動Docker服務

2.2.4 配置docker加速，修改cgroup方式

2.2.5 cri-dockerd安裝

2.2.5.1 下載cri-dockerd 二進制文件

2.2.5.2 啓動cri-docker

2.3 kubernetes 1.24.1 集羣部署

2.3.1 集羣軟件及版本說明

2.3.2 kubernetes YUM源準備

2.3.2.1 谷歌YUM源 [國外主機]

2.3.2.2 阿里雲YUM源【國內主機】

2.3.3 集羣軟件安裝

2.3.4 配置kubelet

2.3.5 初始化配置文件

2.3.6 集羣鏡像準備

2.3.7 集羣初始化

2.3.7 集羣應用客戶端管理集羣文件準備

2.3.8 集羣網絡準備

2.3.8.1 calico安裝

2.3.8.2 calico客戶端安裝

2.3.9 集羣工作節點添加

2.3.10 驗證集羣可用性

三、參考