k8s學習筆記-部署Flannel網絡

主機列表

本次實驗選擇5臺主機，3臺作爲master主機，2臺作爲node節點

節點ip	OS版本	hostname -f	安裝軟件
192.168.0.1	RHEL7.4	k8s-master01	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.2	RHEL7.4	k8s-master02	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.3	RHEL7.4	k8s-master03	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.4	RHEL7.4	k8s-node01	docker,flanneld,kubelet,kube-proxy
192.168.0.5	RHEL7.4	k8s-node02	docker,flanneld,kubelet,kube-proxy

kubernetes使用Flannel實現集羣內各節點能通過Pod 網段互聯互通

創建TLS 密鑰和證書

etcd 集羣啓用了雙向TLS 認證，所以需要爲flanneld 指定與etcd 集羣通信的CA 和密鑰。
創建flanneld 證書籤名請求：

cat > flanneld-csr.json <<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
   "algo": "rsa",
   "size": 2048
},
"names": [
   {
   "C": "CN",
   "ST": "BeiJing",
   "L": "BeiJing",
   "O": "k8s",
   "OU": "System"
   }
]
}
EOF

生成flanneld 證書和私鑰

# cfssl gencert -ca=/k8s/kubernetes/ssl/ca.pem -ca-key=/k8s/kubernetes/ssl/ca-key.pem -config=/k8s/kubernetes/ssl/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

# ls flanneld*
flanneld.csr flanneld-csr.json flanneld-key.pem flanneld.pem

向etcd 寫入集羣Pod 網段信息

# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem set /kubernetes/network/config '{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}'

輸出信息：
{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}

提示：

該步驟只需在第一次部署Flannel 網絡時執行，後續在其他節點上部署Flanneld 時無需再寫入該信息
寫入/kubernetes/network/config的 Pod 網段必須與kube-controller-manager 的 --cluster-cidr 選項值一致

安裝和配置flanneld

wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar xf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

創建flanneld的systemd unit 文件

cat << EOF > /lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/k8s/kubernetes/bin/flanneld \
--etcd-cafile=/k8s/kubernetes/ssl/ca.pem \
--etcd-certfile=/k8s/flanneld/ssl/flanneld.pem \
--etcd-keyfile=/k8s/flanneld/ssl/flanneld-key.pem \
--etcd-endpoints=https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379 \
--etcd-prefix=/kubernetes/network
ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh腳本將分配給flanneld 的Pod 子網網段信息寫入到/run/flannel/docker 文件中，後續docker 啓動時使用這個文件中的參數值爲 docker0 網橋
flanneld 使用系統缺省路由所在的接口和其他節點通信，對於有多個網絡接口的機器(內網和公網)，可以用 --iface 選項值指定通信接口(上面的 systemd unit 文件沒指定這個選項)

配置Docker啓動指定子網段

cat << EOF > /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

啓動flanneld

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl restart docker

檢查flanneld 服務

# 查看flannel.1
ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 100.100.31.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::9c42:ecff:fe23:8885 prefixlen 64 scopeid 0x20<link>
ether 9e:42:ec:23:88:85 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 55 overruns 0 carrier 0 collisions 0

# 查看集羣 Pod 網段(/16)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/config
{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}

# 查看已分配的 Pod 子網段列表(/24)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/100.100.31.0-24

# 查看某一 Pod 網段對應的 flanneld 進程監聽的 IP 和網絡參數
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/subnets/100.100.31.0-24
{"PublicIP":"192.168.0.2","BackendType":"vxlan","BackendData":{"VtepMAC":"9e:42:ec:23:88:85"}}

分發配置文件複製到其他節點

scp -r /k8s/kubernetes/bin/flanneld 192.168.0.2:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.3:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.4:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.5:/k8s/kubernetes/bin/
scp -r /k8s/flanneld/ssl/* 192.168.0.2:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.3:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.4:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.5:/k8s/flanneld/ssl/
scp /lib/systemd/system/flanneld.service 192.168.0.2:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.3:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.4:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.5:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/docker.service 192.168.0.2:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.3:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.4:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.5:/lib/systemd/system/docker.service

k8s學習筆記-部署Flannel網絡

主機列表

創建TLS 密鑰和證書

生成flanneld 證書和私鑰

向etcd 寫入集羣Pod 網段信息

安裝和配置flanneld

創建flanneld的systemd unit 文件

配置Docker啓動指定子網段

啓動flanneld

檢查flanneld 服務

分發配置文件複製到其他節點

AI模型 Llama 3體驗筆記

【面試準備】又一次失敗的面試經歷，題目離譜～資深軟件測試工程師

dotnet 8 版本與銀河麒麟V10和UOS系統的 glibc 兼容性

Ambari學習筆記-安裝Ambari2.6 on PostgreSQL

k8s學習筆記-部署Flannel網絡

k8s學習筆記-部署master節點

Ambari學習筆記-啓用Kerberos

k8s學習筆記-部署kubectl命令工具

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結