Kubernetes(k8s)1.14 離線版集羣 - 高可用kube-apiserver+部署KeepLived

聲明：
如果您有更好的技術與作者分享，或者商業合作；
請訪問作者個人網站 http://www.esqabc.com/view/message.html 留言給作者。
如果該案例觸犯您的專利，請在這裏：http://www.esqabc.com/view/message.html 留言給作者說明原由
作者一經查實，馬上刪除。

.
.

1、搭建前說明：

.
a、前提提條件、服務器，請查看這個地址：https://blog.csdn.net/esqabc/article/details/102726771
.

.

2、搭建kube-apiserver 高可用

• 使用Nginx 4層實現k8s節點(master節點和worker節點)高可用，訪問kube-apiserver的步驟
• 注意：搭建服務器，沒有特殊說明，一般默認在：k8s-01 操作

a、下載編譯nginx
（1）把下載好的文件上傳到：/opt/k8s/work

[root@k8s-01 ~]# cd /opt/k8s/work
解壓
[root@k8s-01 work]# tar -xzvf nginx-1.15.3.tar.gz
編譯
[root@k8s-01 work]# cd /opt/k8s/work/nginx-1.15.3
[root@k8s-01 nginx-1.15.3]# mkdir nginx-prefix
.
注意，下面命令有一個點的
.
[root@k8s-01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module
[root@k8s-01 nginx-1.15.3]# make && make install

說明一下：

• without-http_scgi_module --without-http_fastcgi_module
• with-stream：開啓 4 層透明轉發(TCP Proxy)功能；
• without-xxx：關閉所有其他功能，這樣生成的動態鏈接二進制程序依賴最小；

（2）創建目錄結構

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  done

（3）分發到其他主機

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    sleep 3
  done

注意：如果出現下面錯誤，執行上面命令重複執行一次就OK：

出現下圖就代表成功：

（4）配置Nginx文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > kube-nginx.conf <<EOF
添加下面內容：

worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
        hash $remote_addr consistent;
        server 172.26.16.249:6443        max_fails=3 fail_timeout=30s;
        server 172.26.16.250:6443        max_fails=3 fail_timeout=30s;
        server 172.26.16.251:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen *:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

注意：只需要修改server 內容即可
.
（5）分發配置文件

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  done

（6）配置Nginx啓動文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > kube-nginx.service <<EOF
添加下面內容：

[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

（7）分發nginx啓動文件

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done

（8）啓動 kube-nginx 服務

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl start kube-nginx"
  done

（9）檢查 kube-nginx 服務運行狀態

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done

出現下圖，就說明正常

3、KeepLived 部署

在所有master節點安裝keeplived
a、安裝keepalived

[root@k8s-01 ~]# yum install -y keepalived

b、配置keeplive服務

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > /etc/keepalived/keepalived.conf <<EOF
添加下面內容

! Configuration File for keepalived
global_defs {
   router_id 172.26.16.249
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 172.26.16.249
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        172.26.16.252
    }
}
EOF	

注意：## 172.26.16.249 爲當前節點，172.26.16.253爲node節點
.
c、將配置拷貝到其他節點，並替換相關IP

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in 172.26.16.249 172.26.16.250 172.26.16.251
  do
    echo ">>> ${node_ip}"
    scp  /etc/keepalived/keepalived.conf $node_ip:/etc/keepalived/keepalived.conf
  done

d、替換IP

	ssh root@172.26.16.250 sed -i 's#172.26.16.249#172.26.16.250#g'  /etc/keepalived/keepalived.conf

	ssh root@172.26.16.251 sed -i 's#172.26.16.249#172.26.16.251#g'  /etc/keepalived/keepalived.conf

注意：不需修改172.26.16.249，只需修改其他IP即可

e、創建健康檢查腳本

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# vi /opt/check_port.sh
添加下面內容：

CHK_PORT=$1
 if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
 else
        echo "Check Port Cant Be Empty!"
 fi

f、啓動keeplived

[root@k8s-01 ~]# cd /opt/k8s/work

for NODE in k8s-01 k8s-02 k8s-03; do
    echo "--- $NODE ---"
    scp -r /opt/check_port.sh  $NODE:/etc/keepalived/
    ssh $NODE 'systemctl enable --now keepalived'
done

g、查看是否成功

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# ping 172.26.16.252

h、檢查是否啓動成功（分別在其他服務器執行）

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# ps -ef|grep keep

如果沒有啓動，請執行下面的命令：
[root@k8s-01 ~]# systemctl start keepalived