Kubernetes(k8s)1.14 离线版集群 - 部署高可用的Etcd（3.3.17版）集群并搭建Flannel网络

声明：
如果您有更好的技术与作者分享，或者商业合作；
请访问作者个人网站 http://www.esqabc.com/view/message.html 留言给作者。
如果该案例触犯您的专利，请在这里：http://www.esqabc.com/view/message.html 留言给作者说明原由
作者一经查实，马上删除。

.
.

1、Etcd集群各节点的名称和ip如下：

前提提条件、服务器，请查看这个地址：https://blog.csdn.net/esqabc/article/details/102726771

内网IP	名称
172.26.16.249	k8s-01
172.26.16.250	k8s-02
172.26.16.251	k8s-03

2、下载etcd

a、官方下载地址：https://github.com/etcd-io/etcd/releases/tag/v3.3.17/etcd-v3.3.17-linux-amd64.tar.gz
下载后上传到：/opt/k8s/work

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# tar -xvf etcd-v3.3.17-linux-amd64.tar.gz

b、分发二进制文件到集群节点

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.17-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

3、创建etcd证书和私钥

a、创建证书json

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > etcd-csr.json <<EOF
添加下面内容：

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.26.16.249",
    "172.26.16.250",
    "172.26.16.251"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

说明一下：

hosts：字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

b、生成证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

[root@k8s-01 work]# ls etcd*pem

c、分发证书和私钥到etcd各个节点

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

d、创建etcd的启动文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > etcd.service.template <<EOF

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
  --data-dir=${ETCD_DATA_DIR} \\
  --wal-dir=${ETCD_WAL_DIR} \\
  --name=##NODE_NAME## \\
  --cert-file=/etc/etcd/cert/etcd.pem \\
  --key-file=/etc/etcd/cert/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-cert-file=/etc/etcd/cert/etcd.pem \\
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://##NODE_IP##:2380 \\
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \\
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://##NODE_IP##:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=6442450944 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

说明一下：

WorkDirectory、–data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录
–wal-dir 指定wal目录，为了提高性能，一般使用SSD和–data-dir不同的盘
–name
指定节点名称，当–initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster列表中
–cert-file、–key-file ETCD server与client通信时使用的证书和私钥
–trusted-ca-file 签名client证书的CA证书，用于验证client证书
–peer-cert-file、–peer-key-file ETCD与peer通信使用的证书和私钥
–peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书

e、分发启动文件到各个节点

(1）、分发会将配置文件中的#替换成ip

[root@k8s-01 ~]# cd /opt/k8s/work

for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

[root@k8s-01 ~]# ls *.service

(2）、分发会将配置文件中的#替换成ip

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

4、启动etcd服务

a、重命名etcd启动文件并启动etcd服务

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

b、检查etcd启动结果

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

如果启动正常，如下图：

如果etcd集群状态不是active (running)，请使用下面命令查看etcd日志

[root@k8s-01 ~]# journalctl -fu etcd

c、验证ETCD集群状态，在任意etcd节点执行

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
    --endpoints=https://${node_ip}:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health
  done

正常状态，如下图：

d、查看当前etcd集群leader

[root@k8s-01 ~]# cd /opt/k8s/work

ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
  -w table --cacert=/etc/kubernetes/cert/ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem \
  --endpoints=${ETCD_ENDPOINTS} endpoint status

正常，如下图：

.

5、部署Flannel网络

a、下载分发flanneld二进制文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# mkdir flannel
[root@k8s-01 work]# wget http://down.i4t.com/k8s1.14/flannel-v0.11.0-linux-amd64.tar.gz
[root@k8s-01 work]# tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel

b、分发二进制文件到所有集群的节点

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

c、创建Flannel证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > flanneld-csr.json <<EOF
添加下面内容：

{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

d、生成证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

[root@k8s-01 ~]# ls flanneld*pem

e、将生成的证书和私钥分发到所有节点

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
    scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert
  done

f、向etcd写入Pod网段信息

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/opt/k8s/work/ca.pem \
  --cert-file=/opt/k8s/work/flanneld.pem \
  --key-file=/opt/k8s/work/flanneld-key.pem \
  mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'

g、创建flanneld的启动文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > flanneld.service << EOF

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${IFACE} \\
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

h、分发启动文件到所有节点

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
  done

i、启动flanneld服务

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
  done

j、检查启动结果

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status flanneld|grep Active"
  done

正常结果：

k、检查分配给flanneld的Pod网段信息

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

l、查看已分配的Pod子网网段列表

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

m、查看节点flannel网络信息

[root@k8s-01 ~]# ip addr show

n、检查是否创建了 flannel 接口

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
  done

.
.
.

也可以参考这篇文章：https://i4t.com/4253.html

Kubernetes(k8s)1.14 离线版集群 - 部署高可用的Etcd（3.3.17版）集群并搭建Flannel网络

1、Etcd集群各节点的名称和ip如下：

2、下载etcd

3、创建etcd证书和私钥

4、启动etcd服务

5、部署Flannel网络

微服务实践之使用 Visual Studio 2022 调试Dapr 应用程序

wpf附加属性理解 WPF附加属性

Kubernetes(k8s)1.14 離線版集羣 - 搭建環境配置

Spring boot2.0.x+SpringCloud(Finchley版本)分佈式架構--分佈式配置中心(Spring Cloud Config)（五）

Kubernetes(k8s)1.14 離線版集羣 - 部署高可用的Etcd（3.3.17版）集羣並搭建Flannel網絡

Kubernetes(k8s)1.14 離線版集羣 - 部署master節點

Spring boot2.0.x+SpringCloud(Finchley版本)分佈式架構--搭建服務註冊中心與發現服務（一）

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結