k8s学习笔记-部署Flannel网络

主机列表

本次实验选择5台主机，3台作为master主机，2台作为node节点

节点ip	OS版本	hostname -f	安装软件
192.168.0.1	RHEL7.4	k8s-master01	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.2	RHEL7.4	k8s-master02	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.3	RHEL7.4	k8s-master03	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.4	RHEL7.4	k8s-node01	docker,flanneld,kubelet,kube-proxy
192.168.0.5	RHEL7.4	k8s-node02	docker,flanneld,kubelet,kube-proxy

kubernetes使用Flannel实现集群内各节点能通过Pod 网段互联互通

创建TLS 密钥和证书

etcd 集群启用了双向TLS 认证，所以需要为flanneld 指定与etcd 集群通信的CA 和密钥。
创建flanneld 证书签名请求：

cat > flanneld-csr.json <<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
   "algo": "rsa",
   "size": 2048
},
"names": [
   {
   "C": "CN",
   "ST": "BeiJing",
   "L": "BeiJing",
   "O": "k8s",
   "OU": "System"
   }
]
}
EOF

生成flanneld 证书和私钥

# cfssl gencert -ca=/k8s/kubernetes/ssl/ca.pem -ca-key=/k8s/kubernetes/ssl/ca-key.pem -config=/k8s/kubernetes/ssl/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

# ls flanneld*
flanneld.csr flanneld-csr.json flanneld-key.pem flanneld.pem

向etcd 写入集群Pod 网段信息

# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem set /kubernetes/network/config '{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}'

输出信息：
{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}

提示：

该步骤只需在第一次部署Flannel 网络时执行，后续在其他节点上部署Flanneld 时无需再写入该信息
写入/kubernetes/network/config的 Pod 网段必须与kube-controller-manager 的 --cluster-cidr 选项值一致

安装和配置flanneld

wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar xf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

创建flanneld的systemd unit 文件

cat << EOF > /lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/k8s/kubernetes/bin/flanneld \
--etcd-cafile=/k8s/kubernetes/ssl/ca.pem \
--etcd-certfile=/k8s/flanneld/ssl/flanneld.pem \
--etcd-keyfile=/k8s/flanneld/ssl/flanneld-key.pem \
--etcd-endpoints=https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379 \
--etcd-prefix=/kubernetes/network
ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh脚本将分配给flanneld 的Pod 子网网段信息写入到/run/flannel/docker 文件中，后续docker 启动时使用这个文件中的参数值为 docker0 网桥
flanneld 使用系统缺省路由所在的接口和其他节点通信，对于有多个网络接口的机器(内网和公网)，可以用 --iface 选项值指定通信接口(上面的 systemd unit 文件没指定这个选项)

配置Docker启动指定子网段

cat << EOF > /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

启动flanneld

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl restart docker

检查flanneld 服务

# 查看flannel.1
ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 100.100.31.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::9c42:ecff:fe23:8885 prefixlen 64 scopeid 0x20<link>
ether 9e:42:ec:23:88:85 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 55 overruns 0 carrier 0 collisions 0

# 查看集群 Pod 网段(/16)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/config
{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}

# 查看已分配的 Pod 子网段列表(/24)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/100.100.31.0-24

# 查看某一 Pod 网段对应的 flanneld 进程监听的 IP 和网络参数
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl   --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/subnets/100.100.31.0-24
{"PublicIP":"192.168.0.2","BackendType":"vxlan","BackendData":{"VtepMAC":"9e:42:ec:23:88:85"}}

分发配置文件复制到其他节点

scp -r /k8s/kubernetes/bin/flanneld 192.168.0.2:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.3:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.4:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.5:/k8s/kubernetes/bin/
scp -r /k8s/flanneld/ssl/* 192.168.0.2:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.3:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.4:/k8s/flanneld/ssl/
scp -r /k8s/flanneld/ssl/* 192.168.0.5:/k8s/flanneld/ssl/
scp /lib/systemd/system/flanneld.service 192.168.0.2:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.3:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.4:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/flanneld.service 192.168.0.5:/lib/systemd/system/flanneld.service
scp /lib/systemd/system/docker.service 192.168.0.2:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.3:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.4:/lib/systemd/system/docker.service
scp /lib/systemd/system/docker.service 192.168.0.5:/lib/systemd/system/docker.service

k8s学习笔记-部署Flannel网络

主机列表

创建TLS 密钥和证书

生成flanneld 证书和私钥

向etcd 写入集群Pod 网段信息

安装和配置flanneld

创建flanneld的systemd unit 文件

配置Docker启动指定子网段

启动flanneld

检查flanneld 服务

分发配置文件复制到其他节点

Python 潮流周刊#50：我最喜欢的 Python 3.13 新特性！

Ambari學習筆記-安裝Ambari2.6 on PostgreSQL

k8s學習筆記-部署Flannel網絡

k8s學習筆記-部署master節點

Ambari學習筆記-啓用Kerberos

k8s學習筆記-部署kubectl命令工具

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結