k8s master 負載均衡
1. 服務器規劃
說明：只實現master負載均衡
| 服務器名稱 | IP | 角色 |
| k8s-master1 | 192.168.1.107 | k8s-master1、etcd |
| k8s-master2 | 192.168.1.108 | k8s-master2 |
| k8s-node1 | 192.168.1.109 | k8s-node1 |
| nginx | 192.168.1.55 | nginx負載 |
2.k8s-master1 部署
1.安裝Docker
# 關閉防火牆
ufw disable && ufw status
# 執行腳本安裝docker
curl -s https://raw.githubusercontent.com/jy1779/docker/master/install/aliyun_docker_install.sh | bash
# 修改docker.server參數
LINE=$(grep -n ExecStart /lib/systemd/system/docker.service|awk -F : '{print $1}')
EXECSTARTPOST='ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT'
sed "$LINE a$EXECSTARTPOST" -i /lib/systemd/system/docker.service
# 重新加載docker.server及重啓docker服務
systemctl daemon-reload && service docker restart
service docker status
2. 生成配置文件及根證書
# 添加內核參數
/etc/sysctl.d/k8s.conf
# 參數說明：
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Enable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
# 執行命令，添加內核參數 cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF # 使內核參數生效 sysctl -p /etc/sysctl.d/k8s.conf
# 如提示以下報錯，則執行：modprobe br_netfilter
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
# 獲取k8s二進制文件及配置文件
# kubernetes.git 並非官網的文件，是自定義安裝k8s集羣所需的文件 git clone https://code.aliyun.com/jy1779/kubernetes.git tar xf ./kubernetes/kubernetes-bins.tar.gz -C /usr/local/sbin/ && rm -f ./kubernetes/kubernetes-bins.tar.gz echo 'export PATH=$PATH:/usr/local/sbin/kubernetes-bins' >> /etc/profile && source /etc/profile
# 檢測環境變量
which kubectl /usr/local/sbin/kubernetes-bins/kubectl
# 生成配置文件
cd /root/kubernetes/kubernetes-starter/
# 修改配置文件
vim config.properties
#kubernetes二進制文件目錄,eg: /home/michael/bin
BIN_PATH=/usr/local/sbin/kubernetes-bins
#當前節點ip, eg: 192.168.1.102
NODE_IP=192.168.1.107
#etcd服務集羣列表, eg: http://192.168.1.102:2379
#如果已有etcd集羣可以填寫現有的。沒有的話填寫：http://${MASTER_IP}:2379 （MASTER_IP自行替換成自己的主節點ip）
ETCD_ENDPOINTS=https://192.168.1.107:2379
#kubernetes主節點ip地址, eg: 192.168.1.102
MASTER_IP=192.168.1.107
# 執行腳本生成配置文件
./gen-config.sh with-ca
====替換變量列表====
BIN_PATH=/usr/local/sbin/kubernetes-bins
NODE_IP=192.168.1.107
ETCD_ENDPOINTS=https://192.168.1.107:2379
MASTER_IP=192.168.1.107
====================
====替換配置文件====
all-node/kube-calico.service
ca/admin/admin-csr.json
ca/ca-config.json
ca/ca-csr.json
ca/calico/calico-csr.json
ca/etcd/etcd-csr.json
ca/kube-proxy/kube-proxy-csr.json
ca/kubernetes/kubernetes-csr.json
master-node/etcd.service
master-node/kube-apiserver.service
master-node/kube-controller-manager.service
master-node/kube-scheduler.service
services/kube-dashboard.yaml
services/kube-dns.yaml
worker-node/10-calico.conf
worker-node/kubelet.service
worker-node/kube-proxy.service
=================
配置生成成功，位置: /root/kubernetes/kubernetes-starter/target
# 安裝cfssl
wget -q --show-progress --https-only --timestamping \ https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \ https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 # 修改爲可執行權限 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 # 移動到bin目錄 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson # 驗證 cfssl version
# 生成根證書
# 創建目錄存放ca證書
mkdir -p /etc/kubernetes/ca # 提示：ca-config.json、ca-csr.json事先已經準備好，可修改，也可以自己生成 # 複製ca文件 cp ~/kubernetes/kubernetes-starter/target/ca/ca-config.json /etc/kubernetes/ca cp ~/kubernetes/kubernetes-starter/target/ca/ca-csr.json /etc/kubernetes/ca # 生成證書和密鑰 cd /etc/kubernetes/ca cfssl gencert -initca ca-csr.json | cfssljson -bare ca # 查看證書和密鑰 ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
3. 部署Etcd
etcd節點需要提供給其他服務訪問，就要驗證其他服務的身份，所以需要一個標識自己監聽服務的server證書，當有多個etcd節點的時候也需要client證書與etcd集羣其他節點交互，當然也可以client和server使用同一個證書因爲它們本質上沒有區別。
# 創建存放etcd證書的目錄
mkdir -p /etc/kubernetes/ca/etcd
# 複製etcd證書配置
cp ~/kubernetes/kubernetes-starter/target/ca/etcd/etcd-csr.json /etc/kubernetes/ca/etcd/
cd /etc/kubernetes/ca/etcd/
# 修改etcd-csr.json配置文件
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.1.107",
"192.168.1.108", #添加k8s-master2的IP
"192.168.1.55" #添加nginx負載均衡的IP
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
# 複製etcd證書配置
cp ~/kubernetes/kubernetes-starter/target/ca/etcd/etcd-csr.json /etc/kubernetes/ca/etcd/
cd /etc/kubernetes/ca/etcd/
# 使用根證書(ca.pem)簽發etcd證書
cfssl gencert \
-ca=/etc/kubernetes/ca/ca.pem \
-ca-key=/etc/kubernetes/ca/ca-key.pem \
-config=/etc/kubernetes/ca/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
# 跟之前類似生成三個文件etcd.csr是個中間證書請求文件，我們最終要的是etcd-key.pem和etcd.pem
ls
etcd.csr etcd-csr.json etcd-key.pem etcd.pem
# 創建工作目錄(保存數據的地方)
mkdir -p /var/lib/etcd
# 把etcd服務配置文件copy到系統服務目錄
cp ~/kubernetes/kubernetes-starter/target/master-node/etcd.service /lib/systemd/system/
# 創建etcd服務
systemctl enable etcd.service
# 啓動etcd服務
service etcd start
# 查看服務日誌，看是否有錯誤信息，確保服務正常
journalctl -f -u etcd.service
# 測試etcd服務是否正常
ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.1.107:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd/etcd.pem \
--key=/etc/kubernetes/ca/etcd/etcd-key.pem \
endpoint health
# 顯示以下則爲部署成功。
https://192.168.1.107:2379 is healthy: successfully committed proposal: took = 10.408412ms
4.部署APIServer
# 創建存放api證書目錄
mkdir -p /etc/kubernetes/ca/kubernetes
# 複製apiserver證書配置
cp ~/kubernetes/kubernetes-starter/target/ca/kubernetes/kubernetes-csr.json /etc/kubernetes/ca/kubernetes/
# 使用根證書(ca.pem)簽發kubernetes證書
cd /etc/kubernetes/ca/kubernetes/
# 修改kubernetes-csr.json配置文件
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.1.107",
"192.168.1.108", #添加k8s-master2的IP
"192.168.1.55", #添加nginx負載均衡的iP
"10.68.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
cfssl gencert \
-ca=/etc/kubernetes/ca/ca.pem \
-ca-key=/etc/kubernetes/ca/ca-key.pem \
-config=/etc/kubernetes/ca/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
# 跟之前類似生成三個文件kubernetes.csr是個中間證書請求文件，我們最終要的是kubernetes-key.pem和kubernetes.pem
ls
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
# 生成token認證文件
# 生成隨機token
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
97e8c07dce2b2bab69cfd3162d5383c9
# 寫入token.csv文件
echo "97e8c07dce2b2bab69cfd3162d5383c9,kubelet-bootstrap,10001,"system:kubelet-bootstrap"" > /etc/kubernetes/ca/kubernetes/token.csv
# 把apiservice服務配置文件copy到系統服務目錄
cp ~/kubernetes/kubernetes-starter/target/master-node/kube-apiserver.service /lib/systemd/system/
# 創建kube-apiserver服務
systemctl enable kube-apiserver.service
# 啓動kube-apiserver服務
service kube-apiserver start
# 查看kube-apiserver日誌
journalctl -f -u kube-apiserver
# 默認api允許服務端口，可以修改，比如80端口，如果不修改就無法映射
cat ~/kubernetes/kubernetes-starter/target/master-node/kube-apiserver.service |grep port-range
--service-node-port-range=20000-40000 \
5.部署Controller-manager
controller-manager一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。 # 把kube-controller-manager.service 服務配置文件copy到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/master-node/kube-controller-manager.service /lib/systemd/system/ # 創建kube-controller-manager.service 服務 systemctl enable kube-controller-manager.service # 啓動kube-controller-manager.service 服務 service kube-controller-manager start # 查看kube-controller-manager.service 日誌 journalctl -f -u kube-controller-manager
6. 部署Scheduler
Scheduler一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。 # 把scheduler 服務配置文件copy到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/master-node/kube-scheduler.service /lib/systemd/system/ # 創建kube-scheduler.service 服務 systemctl enable kube-scheduler.service # 啓動kube-scheduler.service 服務 service kube-scheduler start # 查看kube-scheduler.service 日誌 journalctl -f -u kube-scheduler
7.配置Kubectl管理
# 創建存放kubectl證書目錄
mkdir -p /etc/kubernetes/ca/admin
# 準備admin證書配置 - kubectl只需客戶端證書，因此證書請求中 hosts 字段可以爲空
# 複製kubectl證書配置
cp ~/kubernetes/kubernetes-starter/target/ca/admin/admin-csr.json /etc/kubernetes/ca/admin/
# 使用根證書(ca.pem)簽發admin證書
cd /etc/kubernetes/ca/admin/
cfssl gencert \
-ca=/etc/kubernetes/ca/ca.pem \
-ca-key=/etc/kubernetes/ca/ca-key.pem \
-config=/etc/kubernetes/ca/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
# 我們最終要的是admin-key.pem和admin.pem
ls
admin.csr admin-csr.json admin-key.pem admin.pem
# 配置kubectl文件
# 指定apiserver的地址和證書位置
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ca/ca.pem \
--embed-certs=true \
--server=https://192.168.1.107:6443
# 設置客戶端認證參數，指定admin證書和祕鑰
kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/ca/admin/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/ca/admin/admin-key.pem
# 關聯用戶和集羣
kubectl config set-context kubernetes \
--cluster=kubernetes --user=admin
# 設置當前上下文
kubectl config use-context kubernetes
# 設置結果就是一個配置文件，可以看看內容
cat ~/.kube/config
# 驗證master組件
root@k8s-master1:/etc/kubernetes/ca/calico# kubectl get cs
NAME STATUS MESSAGE ERROR
etcd-0 Healthy {"health": "true"}
controller-manager Healthy ok
scheduler Healthy ok
# 創建kubelet-bootstrap綁定
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
8.部署Calico網絡
Calico實現了CNI接口，是kubernetes網絡方案的一種選擇，它一個純三層的數據中心網絡方案（不需要Overlay），並且與OpenStack、Kubernetes、AWS、GCE等IaaS和容器平臺都有良好的集成。 Calico在每一個計算節點利用Linux Kernel實現了一個高效的vRouter來負責數據轉發，而每個vRouter通過BGP協議負責把自己上運行的workload的路由信息像整個Calico網絡內傳播——小規模部署可以直接互聯，大規模下可通過指定的BGP route reflector來完成。 這樣保證最終所有的workload之間的數據流量都是通過IP路由的方式完成互聯的。
# calico證書用在四個地方：
# calico/node: 這個docker 容器運行時訪問 etcd 使用證書
# cni 配置文件中 cni 插件: 需要訪問 etcd 使用證書
# calicoctl: 操作集羣網絡時訪問 etcd 使用證書
# calico/kube-controllers: 同步集羣網絡策略時訪問 etcd 使用證書
# 創建存放calico證書 mkdir -p /etc/kubernetes/ca/calico # 準備calico證書配置 - calico只需客戶端證書，因此證書請求中 hosts 字段可以爲空 cp ~/kubernetes/kubernetes-starter/target/ca/calico/calico-csr.json /etc/kubernetes/ca/calico/ cd /etc/kubernetes/ca/calico/ cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes calico-csr.json | cfssljson -bare calico # 我們最終要的是calico-key.pem和calico.pem ls calico.csr calico-csr.json calico-key.pem calico.pem # 啓動kube-calico.service 服務 cp ~/kubernetes/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ systemctl enable kube-calico.service # 啓動kube-calico服務需要下載鏡像 service kube-calico start
3.k8s-master2 部署
1.安裝Docker
# 關閉防火牆
ufw disable && ufw status
# 執行腳本安裝docker
curl -s https://raw.githubusercontent.com/jy1779/docker/master/install/aliyun_docker_install.sh | bash
# 修改docker.server參數
LINE=$(grep -n ExecStart /lib/systemd/system/docker.service|awk -F : '{print $1}')
EXECSTARTPOST='ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT'
sed "$LINE a$EXECSTARTPOST" -i /lib/systemd/system/docker.service
# 重新加載docker.server及重啓docker服務
systemctl daemon-reload && service docker restart
service docker status
2. 生成配置文件及根證書
# 添加內核參數
/etc/sysctl.d/k8s.conf
# 參數說明：
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Enable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
# 執行命令，添加內核參數 cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF # 使內核參數生效 sysctl -p /etc/sysctl.d/k8s.conf
# 如提示以下報錯，則執行：modprobe br_netfilter
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
# 獲取k8s二進制文件及配置文件
# kubernetes.git 並非官網的文件，是自定義安裝k8s集羣所需的文件
root@master:~# git clone https://code.aliyun.com/jy1779/kubernetes.git
# 解壓k8s二進制文件，並添加到系統環境變量
root@master:~# tar xf ./kubernetes/kubernetes-bins.tar.gz -C /usr/local/sbin/ && rm -f ./kubernetes/kubernetes-bins.tar.gz
root@master:~# echo 'export PATH=$PATH:/usr/local/sbin/kubernetes-bins' >> /etc/profile && source /etc/profile
# 檢測環境變量
root@master:~# which kubectl
/usr/local/sbin/kubernetes-bins/kubectl
# 生成配置文件
cd /root/kubernetes/kubernetes-starter/
# 修改配置文件
vim config.propertie
#kubernetes二進制文件目錄,eg: /home/michael/bin
BIN_PATH=/usr/local/sbin/kubernetes-bins
#當前節點ip, eg: 192.168.1.102
NODE_IP=192.168.1.108
#etcd服務集羣列表, eg: http://192.168.1.102:2379
#如果已有etcd集羣可以填寫現有的。沒有的話填寫：http://${MASTER_IP}:2379 （MASTER_IP自行替換成自己的主節點ip）
ETCD_ENDPOINTS=https://192.168.1.107
#kubernetes主節點ip地址, eg: 192.168.1.102
MASTER_IP=192.168.1.108
./gen-config.sh with-ca
====替換變量列表====
BIN_PATH=/usr/local/sbin/kubernetes-bins
NODE_IP=192.168.1.72
ETCD_ENDPOINTS=https://192.168.1.72:2379,https://192.168.1.73:2379,https://192.168.1.74:2379
MASTER_IP=192.168.1.72
====================
====替換配置文件====
all-node/kube-calico.service
ca/admin/admin-csr.json
ca/ca-config.json
ca/ca-csr.json
ca/calico/calico-csr.json
ca/etcd/etcd-csr.json
ca/kube-proxy/kube-proxy-csr.json
ca/kubernetes/kubernetes-csr.json
master-node/etcd.service
master-node/kube-apiserver.service
master-node/kube-controller-manager.service
master-node/kube-scheduler.service
services/kube-dashboard.yaml
services/kube-dns.yaml
worker-node/10-calico.conf
worker-node/kubelet.service
worker-node/kube-proxy.service
=================
配置生成成功，位置: /root/kubernetes/kubernetes-starter/target
# 安裝cfssl
wget -q --show-progress --https-only --timestamping \ https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \ https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 # 修改爲可執行權限 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 # 移動到bin目錄 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson # 驗證 cfssl version
# 生成根證書
# 創建目錄存放ca證書
mkdir -p /etc/kubernetes/ca # 從k8s-master1獲取證書 rsync -av 192.168.1.107:/etc/kubernetes/ca/ca.pem /etc/kubernetes/ca/ rsync -av 192.168.1.107:/etc/kubernetes/ca/ca-key.pem /etc/kubernetes/ca/ rsync -av 192.168.1.107:/etc/kubernetes/ca/ca-config.json /etc/kubernetes/ca/
# 創建存放etcd證書的目錄
# 從k8s-master1獲取證書
mkdir -p /etc/kubernetes/ca/etcd rsync -av 192.168.1.107:/etc/kubernetes/ca/etcd/etcd-key.pem /etc/kubernetes/ca/etcd/ rsync -av 192.168.1.107:/etc/kubernetes/ca/etcd/etcd.pem /etc/kubernetes/ca/etcd/
# 測試連接etcd服務
ETCDCTL_API=3 etcdctl \ --endpoints=https://192.168.1.107:2379 \ --cacert=/etc/kubernetes/ca/ca.pem \ --cert=/etc/kubernetes/ca/etcd/etcd.pem \ --key=/etc/kubernetes/ca/etcd/etcd-key.pem \ endpoint health # 提示以下，則正常 https://192.168.1.107:2379 is healthy: successfully committed proposal: took = 341.160166ms
3.部署APIServer
# 創建存放api證書目錄
# 從k8s-master1獲取證書
mkdir -p /etc/kubernetes/ca/kubernetes cd /etc/kubernetes/ca/kubernetes/ rsync -av 192.168.1.107:/etc/kubernetes/ca/kubernetes/kubernetes-key.pem /etc/kubernetes/ca/kubernetes/ rsync -av 192.168.1.107:/etc/kubernetes/ca/kubernetes/kubernetes.pem /etc/kubernetes/ca/kubernetes/ rsync -av 192.168.1.107:/etc/kubernetes/ca/kubernetes/token.csv /etc/kubernetes/ca/kubernetes/ # 把apiservice服務配置文件copy到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/master-node/kube-apiserver.service /lib/systemd/system/ # 創建kube-apiserver服務 systemctl enable kube-apiserver.service # 啓動kube-apiserver服務 service kube-apiserver start # 查看kube-apiserver日誌 journalctl -f -u kube-apiserver
4. 部署Controller-manager
controller-manager一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。 # 把kube-controller-manager.service 服務配置文件copy到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/master-node/kube-controller-manager.service /lib/systemd/system/ # 創建kube-controller-manager.service 服務 systemctl enable kube-controller-manager.service # 啓動kube-controller-manager.service 服務 service kube-controller-manager start # 查看kube-controller-manager.service 日誌 journalctl -f -u kube-controller-manager
5.部署Scheduler
Scheduler一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。
# 把scheduler 服務配置文件copy到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/master-node/kube-scheduler.service /lib/systemd/system/ # 創建kube-scheduler.service 服務 systemctl enable kube-scheduler.service # 啓動kube-scheduler.service 服務 service kube-scheduler start # 查看kube-scheduler.service 日誌 journalctl -f -u kube-scheduler
6.配置Kubectl管理
# 創建存放kubectl證書目錄
mkdir -p /etc/kubernetes/ca/admin
# 準備admin證書配置 - kubectl只需客戶端證書，因此證書請求中 hosts 字段可以爲空
# 複製kubectl證書配置
cp ~/kubernetes/kubernetes-starter/target/ca/admin/admin-csr.json /etc/kubernetes/ca/admin/
# 使用根證書(ca.pem)簽發admin證書
cd /etc/kubernetes/ca/admin/
cfssl gencert \
-ca=/etc/kubernetes/ca/ca.pem \
-ca-key=/etc/kubernetes/ca/ca-key.pem \
-config=/etc/kubernetes/ca/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
# 配置kubectl文件
# 指定apiserver的地址和證書位置
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ca/ca.pem \
--embed-certs=true \
--server=https://192.168.1.108:6443
# 設置客戶端認證參數，指定admin證書和祕鑰
kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/ca/admin/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/ca/admin/admin-key.pem
# 關聯用戶和集羣
kubectl config set-context kubernetes \
--cluster=kubernetes --user=admin
# 設置當前上下文
kubectl config use-context kubernetes
# 查看認證
cat ~/.kube/config
# 查看master 組件
kubectl get componentstatus
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
7.部署Calico網絡
# 創建存放calico證書 mkdir -p /etc/kubernetes/ca/calico rsync -av 192.168.1.107:/etc/kubernetes/ca/calico/calico.pem /etc/kubernetes/ca/calico rsync -av 192.168.1.107:/etc/kubernetes/ca/calico/calico-key.pem /etc/kubernetes/ca/calico # 啓動kube-calico.service 服務 cp ~/kubernetes/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ systemctl enable kube-calico.service # 啓動kube-calico服務需要下載鏡像 service kube-calico start journalctl -f -u kube-calico # 日誌查看 calicoctl node status
4.部署nginx
服務器：192.168.1.55
docker-compose 部署nginx
# 查看nginx的docker-compose結構
tree -L 2 nginx/
nginx/
├── conf
│ ├── conf.d
│ ├── fastcgi_params
│ ├── koi-utf
│ ├── koi-win
│ ├── mime.types
│ ├── modules -> /usr/lib/nginx/modules
│ ├── nginx.conf
│ ├── scgi_params
│ ├── uwsgi_params
│ └── win-utf
├── docker-compose.yml
└── html
├── 50x.html
└── index.html
# docker-compose.yaml配置文件
cd nginx
cat docker-compose.yml
version: '2.0'
services:
nginxs:
image: nginx
container_name: nginxs
network_mode: host
volumes:
- "./conf:/etc/nginx"
- "./html:/usr/share/nginx/html"
# 查看nginx配置文件
cat conf/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
# 4層轉發
stream {
log_format ws "$remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s.log ws;
server {
listen 6443;
proxy_pass app_server;
}
upstream app_server{
server 192.168.1.107:6443; #k8s-master1
server 192.168.1.108:6443; #k8s-master2
}
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
docker-compose ps
Name Command State Ports
---------------------------------------------
nginxs nginx -g daemon off; Up
# 查看端口
netstat -nutlp|grep 6443
tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 5706/nginx: master
5.部署Node節點
1.安裝Docker
# 關閉防火牆
ufw disable && ufw status
# 執行docker安裝腳本
curl -s https://raw.githubusercontent.com/jy1779/docker/master/install/aliyun_docker_install.sh | bash
# 獲取二進制文件
git clone https://code.aliyun.com/jy1779/kubernetes.git
# 解壓kubernetes-bins,添加到環境變量
tar xf ./kubernetes/kubernetes-bins.tar.gz -C /usr/local/sbin/
echo 'export PATH=$PATH:/usr/local/sbin/kubernetes-bins' >> /etc/profile && source /etc/profile
# 修改docker.server
LINE=$(grep -n ExecStart /lib/systemd/system/docker.service|awk -F : '{print $1}')
EXECSTARTPOST='ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT'
sed "$LINE a$EXECSTARTPOST" -i /lib/systemd/system/docker.service
# 重啓docker
systemctl daemon-reload && service docker restart
service docker status2.生成配置文件
# 添加內核參數
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# 使內核參數生效
sysctl -p /etc/sysctl.d/k8s.conf
# 修改配置文件config.properties
cd /root/kubernetes/kubernetes-starter/
# 查看配置文件
cat config.properties
#kubernetes二進制文件目錄,eg: /home/michael/bin
BIN_PATH=/usr/local/sbin/kubernetes-bins
#當前節點ip, eg: 192.168.1.102
NODE_IP=192.168.1.109
#etcd服務集羣列表, eg: http://192.168.1.102:2379
#如果已有etcd集羣可以填寫現有的。沒有的話填寫：http://${MASTER_IP}:2379 （MASTER_IP自行替換成自己的主節點ip）
ETCD_ENDPOINTS=https://192.168.1.107:2379
#kubernetes主節點ip地址, eg: 192.168.1.102
MASTER_IP=192.168.1.55
# 生成配置文件
cd ~/kubernetes/kubernetes-starter && ./gen-config.sh with-ca
====替換變量列表====
BIN_PATH=/usr/local/sbin/kubernetes-bins
NODE_IP=192.168.1.109
ETCD_ENDPOINTS=https://192.168.1.107:2379
MASTER_IP=192.168.1.55
====================
====替換配置文件====
all-node/kube-calico.service
ca/admin/admin-csr.json
ca/ca-config.json
ca/ca-csr.json
ca/calico/calico-csr.json
ca/etcd/etcd-csr.json
ca/kube-proxy/kube-proxy-csr.json
ca/kubernetes/kubernetes-csr.json
master-node/etcd.service
master-node/kube-apiserver.service
master-node/kube-controller-manager.service
master-node/kube-scheduler.service
services/kube-dashboard.yaml
services/kube-dns.yaml
worker-node/10-calico.conf
worker-node/kubelet.service
worker-node/kube-proxy.service
=================
配置生成成功，位置: /root/kubernetes/kubernetes-starter/target
# 安裝cfssl
wget -q --show-progress --https-only --timestamping \ https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \ https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson cfssl version
# 創建存放證書目錄
# 從k8s-master1獲取
mkdir -p /etc/kubernetes/ca/ mkdir -p /etc/kubernetes/ca/calico/ rsync -av 192.168.1.107:/etc/kubernetes/ca/ca.pem /etc/kubernetes/ca/ rsync -av 192.168.1.107:/etc/kubernetes/ca/ca-key.pem /etc/kubernetes/ca/ rsync -av 192.168.1.107:/etc/kubernetes/ca/ca-config.json /etc/kubernetes/ca/ rsync -av 192.168.1.107:/etc/kubernetes/ca/calico/calico.pem /etc/kubernetes/ca/calico/ rsync -av 192.168.1.107:/etc/kubernetes/ca/calico/calico-key.pem /etc/kubernetes/ca/calico/
3.部署Calico網絡
# 複製calico啓動文件到系統服務目錄 cp ~/kubernetes/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ # 創建kube-calico.service systemctl enable kube-calico.service # 啓動kube-calico.service 服務 service kube-calico start # 查看calico節點，可以看到master節點的calico calicoctl node status
4.部署Kubelet
cd /etc/kubernetes/ # 創建bootstrap.kubeconfig kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.1.55:6443 \ --kubeconfig=bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap \ --token=97e8c07dce2b2bab69cfd3162d5383c9 \ --kubeconfig=bootstrap.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig kubectl config use-context default --kubeconfig=bootstrap.kubeconfig # 準備cni mkdir -p /etc/cni/net.d/ cp ~/kubernetes/kubernetes-starter/target/worker-node/10-calico.conf /etc/cni/net.d/ # 創建存放kubelet工作目錄 mkdir /var/lib/kubelet # 將kubelet.service 複製到系統目錄 cp ~/kubernetes/kubernetes-starter/target/worker-node/kubelet.service /lib/systemd/system/ # 創建kubelet服務 systemctl enable kubelet # 啓動kubelet服務 service kubelet start
5.Master簽發證書
# 在master服務器執行
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-Vuj62TUED4foaVjOmsbvMLJpfDsy1RBHbKMAhgtuoyE 16s kubelet-bootstrap Pending
# 執行指令簽發
kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
certificatesigningrequest "node-csr-Vuj62TUED4foaVjOmsbvMLJpfDsy1RBHbKMAhgtuoyE" approved
# 再次查看
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-Vuj62TUED4foaVjOmsbvMLJpfDsy1RBHbKMAhgtuoyE 1m kubelet-bootstrap Approved,Issued
# 驗證節點
kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.1.109 Ready <none> 52s v1.9.0
6.部署kube-proxy
# 創建kube-proxy工作目錄及存放證書目錄 mkdir -p /var/lib/kube-proxy mkdir -p /etc/kubernetes/ca/kube-proxy # 複製kube-proxy服務配置文件 cp ~/kubernetes/kubernetes-starter/target/ca/kube-proxy/kube-proxy-csr.json /etc/kubernetes/ca/kube-proxy/ cd /etc/kubernetes/ca/kube-proxy/ # 使用根證書(ca.pem)簽發calico證書 cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy cd /etc/kubernetes/ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.1.55:6443 \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem \ --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig # 複製kube-proxy啓動文件到系統目錄 cp ~/kubernetes/kubernetes-starter/target/worker-node/kube-proxy.service /lib/systemd/system/ # 創建kube-proxy服務 systemctl enable kube-proxy # 啓動kube-proxy服務 service kube-proxy start
7.在k8s-master1創建deployment
# nginx-depolyment.yaml 配置文件 cat nginx-depolyment.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx annotations: nginx.ingress.kubernetes.io/secure-backends: "true" spec: replicas: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.cn-hangzhou.aliyuncs.com/jonny/nginx:1.9.14 ports: - containerPort: 80 # nginx-service.yaml配置文件 cat nginx-service.yaml apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 20001 type: NodePort # 查看nginx pod kubectl get pods NAME READY STATUS RESTARTS AGE nginx-65dbdf6899-z8cp5 1/1 Running 0 2m kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 2h nginx-service NodePort 10.68.169.183 <none> 80:20001/TCP 2m # 驗證nginx curl -I http://192.168.1.109:20001/ HTTP/1.1 200 OK Server: nginx/1.9.14 Date: Thu, 18 Apr 2019 09:04:03 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Wed, 21 Sep 2016 08:11:20 GMT Connection: keep-alive ETag: "57e240a8-264" Accept-Ranges: bytes
5.添加kubectl遠程客戶端
在nginx服務器上添加kubectl客戶端
從k8s-master獲取
rsync -av 192.168.1.107:/usr/local/sbin/kubernetes-bins/kubectl /usr/bin/kubectl
rsync -av 192.168.1.107:/etc/kubernetes/ca/admin/ /root/kubectl/ca/
rsync -av 192.168.1.107:/etc/kubernetes/ca/ca.pem /root/kubectl/ca/
kubectl config set-cluster kubernetes --server=https://192.168.1.55:6443 --certificate-authority=ca.pem
# 設置用戶項中cluster-admin用戶證書認證字段
kubectl config set-credentials cluster-admin --certificate-authority=ca.pem --client-key=admin-key.pem --client-certificate=admin.pem
# 設置環境項中名爲default的默認集羣和用戶
kubectl config set-context default --cluster=kubernetes --user=cluster-admin
# 設置默認環境項爲default
kubectl config use-context default
cat /root/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority: /root/kubectl/ca/ca.pem
server: https://192.168.1.55:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: cluster-admin
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: cluster-admin
user:
as-user-extra: {}
client-certificate: /root/kubectl/ca/admin.pem
client-key: /root/kubectl/ca/admin-key.pem
# 驗證
kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.1.109 Ready <none> 1h v1.9.06.模擬關閉一個k8s-master1
# 在k8s-master1操作 service kube-apiserver stop service kube-controller-manager stop service kube-scheduler stop # 本地已經無法執行 kubectl get node The connection to the server 192.168.1.107:6443 was refused - did you specify the right host or port? # 在nginx服務器的遠程的客戶端執行，不受影響 kubectl get node NAME STATUS ROLES AGE VERSION 192.168.1.109 Ready <none> 48m v1.9.0 kubectl get pods NAME READY STATUS RESTARTS AGE nginx-65dbdf6899-z8cp5 1/1 Running 0 42m # 可以查看nginx的日誌，當192.168.1.107 master關閉後，請求轉發到另外一個master192.168.1.108 192.168.1.55 192.168.1.108:6443 18/Apr/2019:10:19:19 +0000 200