k8s學習筆記-部署master節點

主機列表

本次實驗選擇5臺主機，3臺作爲master主機，2臺作爲node節點

節點ip	OS版本	hostname -f	安裝軟件
192.168.0.1	RHEL7.4	k8s-master01	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.2	RHEL7.4	k8s-master02	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.3	RHEL7.4	k8s-master03	docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler
192.168.0.4	RHEL7.4	k8s-node01	docker,flanneld,kubelet,kube-proxy
192.168.0.5	RHEL7.4	k8s-node02	docker,flanneld,kubelet,kube-proxy

kubernetes master 節點包含的組件：

• kube-apiserver
• kube-scheduler
• kube-controller-manager

目前這三個組件需要部署在同一臺機器上。

• kube-scheduler、kube-controller-manager 和 kube-apiserver三者的功能緊密相關；
• 同時只能有一個 kube-scheduler、kube-controller-manager 進程處於工作狀態，如果運行多個，則需要通過選舉產生一個 leader；

下載解壓二進制文件

# wget https://dl.k8s.io/v1.15.3/kubernetes-server-linux-amd64.tar.gz 
# tar xf kubernetes-server-linux-amd64.tar.gz

# cd kubernetes/server/bin/
# cp  kubeadm  kube-apiserver  kube-controller-manager  kubectl  kube-scheduler  /k8s/kubernetes/bin/

配置和啓動kube-apiserver  

創建kubernetes 證書

創建kubernetes 證書籤名請求：

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.0.1",
    "192.168.0.2",
    "192.168.0.3",
    "101.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF    

• 如果 hosts 字段不爲空，則需要指定授權使用該證書的 IP 或域名列表，所以上面分別指定了當前部署的 master 節點主機 IP，如有apiserver 負載地址或域名，也需一併指定
• 添加 kube-apiserver 註冊的名爲 kubernetes 的服務 IP (Service Cluster IP)，一般是 kube-apiserver --service-cluster-ip-range 選項值指定的網段的第一個IP，如 “101.254.0.1”

生成kubernetes 證書和私鑰

# cfssl gencert -ca=/k8s/kubernetes/ssl/ca.pem   -ca-key=/k8s/kubernetes/ssl/ca-key.pem   -config=/k8s/kubernetes/ssl/ca-config.json   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

# ls kub*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

# cp kubernetes*.pem /k8s/kubernetes/ssl/

創建kube-apiserver 使用的客戶端token 文件

kubelet 首次啓動時向kube-apiserver 發送TLS Bootstrapping 請求，kube-apiserver 驗證請求中的token 是否與它配置的token.csv 一致，如果一致則自動爲kubelet 生成證書和密鑰。

TLS Bootstrapping 使用的Token可以使用命令生成

# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
ef502f26a00ac235b04977cde1dc9916

cat << EOF > /k8s/kubernetes/cfg/token.csv
ef502f26a00ac235b04977cde1dc9916,kubelet-bootstrap,10001,"system:kubelet-bootstrap"    
EOF

創建apiserver配置文件

cat << EOF > /k8s/kubernetes/cfg/kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379 \
--bind-address=192.168.0.1 \
--secure-port=6443 \
--advertise-address=192.168.0.1 \
--allow-privileged=true \
--service-cluster-ip-range=101.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/kubernetes.pem  \
--tls-private-key-file=/k8s/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/kubernetes/ssl/ca.pem \
--etcd-certfile=/k8s/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/k8s/kubernetes/ssl/kubernetes-key.pem"
EOF    

提示：其他master節點需要將紅字部分替換 

創建kube-apiserver 的systemd unit文件

cat << EOF > /lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/k8s/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target    
EOF

啓動kube-apiserver服務

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

授予kubernetes證書訪問kubelet api權限

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

配置和啓動kube-controller-manager

創建kube-controller-manager配置文件

cat << EOF > /k8s/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=101.254.0.0/24 \
--cluster-cidr=100.100.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"    
EOF

 提示：

• 之前的kube-apiserver默認開啓監聽非安全端口8080，開啓了127.0.0.1的本地非安全認證，所以同一節點的kube-controller-manager可以直接配置本地認證，無需單獨爲其創建證書
• 如kube-apiserver配置--insecure-port=0，關閉監聽非安全端口(8080)，則需要單獨爲kube-controller-manager創建證書。

創建kube-controller-manager systemd unit 文件

cat << EOF > /lib/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/k8s/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target    
EOF

啓動 kube-controller-manager服務

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
systemctl status kube-controller-manager

配置和啓動kube-scheduler

創建kube-scheduler配置文件

cat << EOF > /k8s/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true"
EOF

創建kube-scheduler systemd unit 文件

cat << EOF > /lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/k8s/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target    
EOF

啓動kube-scheduler服務

systemctl daemon-reload
systemctl enable kube-scheduler.service 
systemctl restart kube-scheduler.service
systemctl status kube-scheduler.service

驗證master 節點

將可執行文件添加到 PATH 變量中

echo "export PATH=$PATH:/k8s/kubernetes/bin/" >>/etc/profile
source /etc/profile

#查看master服務狀態
# kubectl get cs 
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}     

分發文件至其他節點

scp -r /k8s/kubernetes/bin/* 192.168.0.2:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/* 192.168.0.3:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/ssl/* 192.168.0.2:/k8s/kubernetes/ssl/
scp -r /k8s/kubernetes/ssl/* 192.168.0.3:/k8s/kubernetes/ssl/
scp -r /k8s/kubernetes/cfg/* 192.168.0.2:/k8s/kubernetes/cfg/
scp -r /k8s/kubernetes/cfg/* 192.168.0.3:/k8s/kubernetes/cfg/
scp /lib/systemd/system/kube-* 192.168.0.2:/lib/systemd/system/
scp /lib/systemd/system/kube-* 192.168.0.3:/lib/systemd/system/