上一篇文章我們已經將k8s集羣部署成功,但是我們只是部署了單節master點的k8s,接下來我們要部署master的高可用節點,此時我們用到了lb節點,那麼lb節點需要部署多節點避免單點故障
架構
環境
系統 角色 IP
centos7.4 master-1 10.10.25.149
centos7.4 master-2 10.10.25.112
centos7.4 node-1 10.10.25.150
centos7.4 node-2 10.10.25.151
centos7.4 lb-1 backup 10.10.25.111
centos7.4 lb-2 master 10.10.25.110
centos7.4 VIP 10.10.25.113部署master02 節點
拷貝master01上面的 /opt/kubernetes/目錄
scp -r /opt/kubernetes/ [email protected]:/opt
拷貝master01上的相關服務
scp /usr/lib/systemd/system/{kube-apiserver,kube-scheduler,kube-controller-manager}.service [email protected]:/usr/lib/systemd/system
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--bind-address=10.10.25.112 \
--insecure-bind-address=127.0.0.1 \
--authorization-mode=Node,RBAC \
--runtime-config=rbac.authorization.k8s.io/v1 \
--kubelet-https=true \
--anonymous-auth=false \
--basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
--service-cluster-ip-range=10.1.0.0/16 \
--service-node-port-range=20000-40000 \
--tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
--etcd-servers=https://10.10.25.149:2379,https://10.10.25.150:2379,https://10.10.25.151:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/log/api-audit.log \
--event-ttl=1h \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
啓動apiserver
systemctl start kube-apiserver
# ps -aux | grep kube
systemctl start kube-scheduler kube-controller-manager
加入系統path
vim /root/.bash_profile
添加
PATH=$PATH:$HOME/bin:/opt/kubernetes/bin
source .bash_profile
查看組件狀態
# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
此時已經說明可以連接到ETCD集羣查看node狀態
# /opt/kubernetes/bin/kubectl get node
NAME STATUS ROLES AGE VERSION
10.10.25.150 NotReady <none> 14d v1.10.3
10.10.25.151 NotReady <none> 14d v1.10.3
說明master02 還無法與node通信配置單節點LB負載均衡
注:做高可用集羣時間上需要同步
lb02節點配置
配置nginx yum源,使用4層代理做
vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=https://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
yum install -y nginx修改Nginx配置文件
vim /etc/nginx/nginx.conf
stream {
log_format main "remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.25.149:6443;
server 10.10.25.112:6443;
}
server {
listen 10.10.25.110:6443;
proxy_pass k8s-apiserver;
}
}
修改node節點
cd /opt/kubernetes/cfg/
vim bootstrap.kubeconfig
修改 server: https://10.10.25.149:6443 爲
server: https://10.10.25.110:6443
vim kubelet.kubeconfig
修改 server: https://10.10.25.149:6443 爲
server: https://10.10.25.110:6443
vim kube-proxy.kubeconfig
修改 server: https://10.10.25.149:6443 爲
server: https://10.10.25.110:6443
systemctl restart kubelet
systemctl restart kube-proxy
此時啓動以後會發現master01 master02 都無法與node節點通訊,查看node日誌發現,提示證書錯誤,大致意思是kube-proxy證書是master01節點的而不是LB節點的.所以接下來我們需要重新生成Kube-proxy證書
master01重新生成api-server證書
編輯證書json文件
[root@master ssl]# cat kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.10.25.149",
"10.10.25.112",
"10.10.25.110",
"10.10.25.111",
"10.10.25.113",
"10.1.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
說明:json文件中的IP包括master01 master02節點IP,所有LB節點IP和VIP 的地址,因爲我們最終需要實現 Nginx + Keepalive 0單節點的負載均衡架構
生成證書
cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
拷貝到相應節點
cp kubernetes*.pem /opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.112:/opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.150:/opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.151:/opt/kubernetes/ssl/重啓master節點的服務
systemctl start kube-scheduler kube-controller-manager kube-apiserver
重啓node節點服務
systemctl restart kube-proxy kubelet驗證
# kubectl get node
NAME STATUS ROLES AGE VERSION
10.10.25.150 Ready <none> 15d v1.10.3
10.10.25.151 Ready <none> 15d v1.10.3
說明已經實現了單節點負載均衡.
這裏有個地方需要注意,在以上配置都完成並且沒有錯誤的情況下,有可能出現獲取到的node狀態是notready,有可能出現此問題的有原因是在日誌裏面發現node無法註冊,此時我們需要手動註冊,在master01上執行一下命令即可
kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
lb01節點配置
同樣安裝nginx,過程不贅述，nginx配置也相同。需要改變的只是綁定的IP
vim /etc/nginx/nginx.conf
stream {
log_format main "remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.25.149:6443;
server 10.10.25.112:6443;
}
server {
listen 10.10.25.111:6443;
proxy_pass k8s-apiserver;
}
}
使用Keepalive實現LB節點的高可用
安裝keepalive兩個節點都需要
yum install keepalived -y
設置lb02爲keepalived爲master節點
修改lb02keepalived配置文件
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh" #腳本檢查ngixn狀態
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.25.113/24
}
track_script {
check_nginx
}
}
編寫nginx狀態檢測腳本
cat /etc/keepalived/check_nginx.sh
#!/bin/sh
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")
#獲取nginx進程數
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
授予腳本執行權限
chmod +x /etc/keepalived/check_nginx.sh
啓動keepalived
systemctl start keepalived查看是VIP是否生效
ip addr
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0c:29:2e:86:82 brd ff:ff:ff:ff:ff:ff
inet 10.10.25.110/24 brd 10.10.25.255 scope global dynamic ens192
valid_lft 71256sec preferred_lft 71256sec
inet 10.10.25.113/32 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::58b8:49be:54a7:4c43/64 scope link
valid_lft forever preferred_lft forever
配置lb01keepalived
修改爲backup的keepalived配置文件
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens192
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.25.113
}
track_script {
check_nginx
}
}
編寫nginx狀態檢測腳本
cat /etc/keepalived/check_nginx.sh
#!/bin/sh
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")
#獲取nginx進程數
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
授予腳本執行權限
chmod +x /etc/keepalived/check_nginx.sh
啓動lb01keepalived
systemctl start keepalived
Keepalive故障切換
做keepalived故障切換，測試方法
1 打開一個窗口一直ping VIP
2 kill master節點nginx
3 觀察VIP是否遷移到備份和VIP的丟包情況
4 啓動master節點的nginx 和keepalive
5 觀察VIP時候漂移回到master節點 接入K8s集羣
將node節點的接入VIP
cd /opt/kubernetes/cfg/
vim bootstrap.kubeconfig
修改 server: https://10.10.25.110:6443 爲
server: https://10.10.25.113:6443
vim kubelet.kubeconfig
修改 server: https://10.10.25.110:6443 爲
server: https://10.10.25.113:6443
vim kube-proxy.kubeconfig
修改 server: https://10.10.25.110:6443 爲
server: https://10.10.25.113:6443
systemctl restart kubelet
systemctl restart kube-proxy
重啓服務
systemctl restart kubelet
systemctl restart kube-proxy修改nginx配置文件（兩個節點都需要）
cat /etc/nginx/nginx.conf
user nginx;
worker_processes 2;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
stream {
log_format main "remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.25.149:6443;
server 10.10.25.112:6443;
}
server {
listen 0.0.0.0:6443;
proxy_pass k8s-apiserver;
}
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
重啓Nginx
systemctl restart nginx驗證VIP接入
kubectl get node
NAME STATUS ROLES AGE VERSION
10.10.25.150 Ready <none> 15d v1.10.3
10.10.25.151 Ready <none> 15d v1.10.3
此時說明接入VIP成功