preflight

安裝包

kubeadm
kubelet
kubectl
kubernetes-cni
socat

需要翻牆下載

鏡像

啓動集羣最少需要以下鏡像(以1.8.1爲例)

# basic
gcr.io/google_containers/pause-amd64:3.0

# kubernetes
gcr.io/google_containers/kube-apiserver-amd64:v1.8.1
gcr.io/google_containers/kube-controller-manager-amd64:v1.8.1
gcr.io/google_containers/kube-scheduler-amd64:v1.8.1
gcr.io/google_containers/kube-proxy-amd64:v1.8.1

# kube-dns
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5

# cni network
quay.io/calico/node:v2.6.2
quay.io/calico/kube-controllers:v1.0.0
quay.io/calico/cni:v1.11.0

etcd 集羣

etcd 版本：3.2.7

walker-1:

[root@walker-1 dashboard]# cat /etc/etcd/etcd.conf | egrep -v "^($|#)"
ETCD_NAME=walker-1
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.16.6.47:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.6.47:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.6.47:2380"
ETCD_INITIAL_CLUSTER="walker-2=http://172.16.6.249:2380,walker-1=http://172.16.6.47:2380,walker-4=http://172.16.17.119:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.6.47:2379"

walker-2:

[root@walker-2 kubernetes]#  cat /etc/etcd/etcd.conf | egrep -v "^($|#)"
ETCD_NAME=walker-2
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.16.6.249:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.6.249:2379, http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.6.249:2380"
ETCD_INITIAL_CLUSTER="walker-1=http://172.16.6.47:2380,walker-2=http://172.16.6.249:2380,walker-4=http://172.16.17.119:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.6.249:2379"

walker-4:

[root@walker-4 ~]# cat /etc/etcd/etcd.conf | egrep -v "^($|#)"
ETCD_NAME=walker-4
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.16.17.119:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.17.119:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.17.119:2380"
ETCD_INITIAL_CLUSTER="walker-2=http://172.16.6.249:2380,walker-1=http://172.16.6.47:2380,walker-4=http://172.16.17.119:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.17.119:2379"

搭建kubernetes集羣

拓撲

graph TD
    nodex --> lvs
    lvs --> controller1
    lvs --> controller2

server	ip	hostname
lvs	172.16.6.56
controller1	172.16.6.47	walker-1
controller2	172.16.6.249	walker-2
node1	172.16.17.119	walker-4

其中 controller1, controller2, node1 組成 etcd集羣

通過kubeadm初始化

準備一個初始化文件，kubeadm-init.yaml

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - http://walker-1:2379
  - http://walker-2:2379
  - http://walker-4:2379
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 192.168.0.0/16
kubernetesVersion: v1.8.1
apiServerCertSANs:
- walker-1.novalocal
- walker-2
- 172.16.6.47
- 172.16.6.249
- 172.16.6.79 # vip

執行

[root@walker-1 kubernetes]# kubeadm init --config ./kubeadm-init.yaml --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Skipping pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [walker-1.novalocal kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local walker-1.novalocal walker-2.novalocal] and IPs [10.96.0.1 172.16.6.47 172.16.6.47 172.16.6.249 172.16.6.79]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
...

You can now join any number of machines by running the following on each node
as root:

   kubeadm join --token 19f284.da47998c9abb01d3 172.16.6.47:6443 --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538

執行後，可以發現kube-dns無法創建。提示：

Events:
  Type     Reason                  Age               From                         Message
  ----     ------                  ----              ----                         -------
  Normal   Scheduled               15s               default-scheduler            Successfully assigned kube-dns-8bb5c479-brk2t to walker-1.novalocal
  Normal   SuccessfulMountVolume   15s               kubelet, walker-1.novalocal  MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   15s               kubelet, walker-1.novalocal  MountVolume.SetUp succeeded for volume "kube-dns-token-4jcng"
  Warning  FailedCreatePodSandBox  5s (x2 over 11s)  kubelet, walker-1.novalocal  Failed create pod sandbox.
  Warning  FailedSync              5s (x2 over 11s)  kubelet, walker-1.novalocal  Error syncing pod
  Normal   SandboxChanged          4s (x2 over 10s)  kubelet, walker-1.novalocal  Pod sandbox changed, it will be killed and re-created.

Pod sandbox changed, it will be killed and re-created 基本上是因爲網絡插件異常導致，因此需要檢查網絡插件健康狀態。

[root@walker-1 ~]# kubectl get ds --namespace=kube-system
NAME          DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
calico-node   0         0         0         0            0           <none>          21d
kube-proxy    1         1         1         1            1           <none>          21d

發現calico-node的個數爲0.
原因是kubernetes的taint 和toleration 機制導致的。因此可以參考kube-proxy中的配置，在和calico幾個相關的容器中添加。

tolerations:
- effect: NoSchedule
  key: node-role.kubernetes.io/master
- effect: NoSchedule
  key: node.cloudprovider.kubernetes.io/uninitialized
  value: "true"

然後calico容器就能在master上運行了。kube-dns也順利啓動。

[root@walker-1 kubernetes]# kubectl get nodes
NAME                 STATUS    ROLES     AGE       VERSION
walker-1.novalocal   Ready     master    11m       v1.8.1

添加備份節點

方便起見，將controller1 上 /etc/kubernetes的所有文件copy到 controller2 上

[root@walker-1 kubernetes]# scp -r /etc/kubernetes/* walker-2:/etc/kubernetes

這裏先介紹一下k8s集羣的原理。
apiserver在集羣中充當了一個類似gateway的組件，即所有的請求都通過它(包括多controller和多sheduler之間的選舉)。因爲是集羣間通信的唯一入口，所以首先要保證apiserver的冗餘。接下來第一步就是在controller2上先把apiserver配置好。

k8s中爲了安全考慮，添加了多種認證方式。比如x509證書、service account、static token 等等。以kubeadm創建的集羣，默認是以證書的形式來驗證apiserver的合法性。因此在/etc/kubernetes/pki/ 下面有 apiserver.crt和 apiserver.key 這兩個服務端的證書文件。kubeadm 會根據配置文件中的內容來生成apiserver證書。可以通過

[root@walker-1 kubernetes]# openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8163852820871759178 (0x714bd4e2faa1c54a)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Dec 19 07:17:25 2017 GMT
            Not After : Dec 19 07:17:26 2018 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:

                    ...

                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:walker-1.novalocal, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:walker-1.novalocal, DNS:walker-2, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:172.16.6.47, IP Address:172.16.6.47, IP Address:172.16.6.249, IP Address:172.16.6.79, IP Address:127.0.0.1, IP Address:10.96.0.1
    Signature Algorithm: sha256WithRSAEncryption
         ...

來查看apiserver證書的內容。可以看到裏面包含了controller1, controller2 以及要使用的vip信息。所以在controller2上我們可以直接使用該證書，而不用在手動生成。
值得一提的是，在 pki 目錄下所有的證書都可以被重用(大部分是客戶端認證的證書，有一份就好了)。

唯一需要修改的是有關 controller2 上kubelet 的證書信息。

一般我們執行命令，行令流程如下：

client --> kubelet --> apiserver --> etcd

如 kubectl get po 的時候，kubectl 作爲客戶端程序，向本地的kubelet 監聽的端口發起了請求。一般情況下，請求被視爲匿名的。這個時候，kubelet在向apiserver請求的時候，apiserver會覈對用戶權限，然後拒絕掉該請求。並返回401。所以在/etc/kubernetes/ 目錄下面會有一個 admin.conf。 kubectl 會從該文件中讀取請求的用戶信息，並以該用戶身份發出請求。爲了防止中間人攻擊，協議一般採用https。但是需要客戶端和服務器之間的雙向認證。因此客戶端也需要ca 頒發的證書。(因爲證書信息是基於service account 的，因此證書可重用。kube-controller 和 kube-shudler 也是如此)

在kubernetes 1.7.x 即其以上的版本中，爲了安全考慮，在apiserver的啓動參數--admission-control中，啓動了NodeRestriction 策略。而且設置集羣認證方式爲- --authorization-mode=Node,RBAC。在k8s中，所有節點間(可以認爲是各個節點上的kubelet進程)是通過service account + 證書的形式來進行認證的，節點對應的service account 名爲system:node:$(hostname)，且都歸屬與system:nodes service account 組。

因爲每個節點的hostname不同，所以需要單獨配置證書。
可使用如下腳本來生成kubelet的客戶端證書

set -e

cat > $HOSTNAME-csr.conf << EOF
[ v3_ext ]
# Extensions to add to a certificate request
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF

# generate client private key file
echo "generate client private key file"
openssl genrsa -out $HOSTNAME.key 2048

# generate client CSR(certificate signing request)
echo "generate client CSR(certificate signing request)"
openssl req -new -sha256 -key $HOSTNAME.key -out $HOSTNAME.csr -subj "/O=system:nodes/CN=system:node:$HOSTNAME"

# generate client certification
echo "generate client certification"
#openssl dgst -sha256 x509 -req -in $HOSTNAME.key -CA ca.crt -CAkey ca.key -CAcreateserial -out $HOSTNAME.crt -days 1000 -extensions v3_ext -extfile $HOSTNAME-crt.conf
openssl x509 -sha256 -req -in $HOSTNAME.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out $HOSTNAME.crt -days 1000 -extensions v3_ext -extfile $HOSTNAME-csr.conf

# show created info
echo "show created info"
openssl x509 -noout -text -in $HOSTNAME.crt


# show crt with base64 encode
echo "show crt with base64 encode"
cat $HOSTNAME.crt | base64 -w 0 && echo

# show key with base64 encode
echo "show key with base64 encode"
cat $HOSTNAME.key | base64 -w 0 && echo

將最後打印出來的證書和key的base64編碼，替換掉 /etc/kubernetes/kubelet.conf 中的 client-certificate-data 和 client-key-data。具體如下所示

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $(cat /etc/kubernetes/pki/ca.crt | base64 -w 0)
    server: https://172.16.6.249:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:$HOSTNAME
  name: system:node:$HOSTNAME@kubernetes
current-context: system:node:$HOSTNAME@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:$HOSTNAME
  user:
    client-certificate-data: $(cat /etc/kubernetes/pki/$HOSTNAME.crt | base64 -w 0)
    client-key-data: $(cat /etc/kubernetes/pki/$HOSTNAME.key | base64 -w 0)

同時修改 admin.conf, controller-manager.conf, scheduler.conf 將apiserver地址指向controller2.
重啓kubelet服務即可

[root@walker-2 kubernetes]# kubectl get nodes
NAME                 STATUS     ROLES     AGE       VERSION
walker-1.novalocal   NotReady   master    4h        v1.8.1
walker-2             Ready      <none>    2h        v1.8.1

[root@walker-1 k8s]# kubectl get po -o wide --namespace=kube-system
NAME                                         READY     STATUS    RESTARTS   AGE       IP                NODE
calico-kube-controllers-86f67c8fd7-ln78x     1/1       Running   0          1h        172.16.6.47       walker-1.novalocal
calico-node-7rqmc                            2/2       Running   0          8m        172.16.6.249      walker-2
calico-node-zqc56                            2/2       Running   0          1h        172.16.6.47       walker-1.novalocal
calico-policy-controller-566dc8d645-5dnv8    1/1       Running   0          1h        172.16.6.249      walker-2
kube-apiserver-walker-1.novalocal            1/1       Running   3          1h        172.16.6.47       walker-1.novalocal
kube-apiserver-walker-2                      1/1       Running   0          7m        172.16.6.249      walker-2
kube-controller-manager-walker-1.novalocal   1/1       Running   0          1h        172.16.6.47       walker-1.novalocal
kube-controller-manager-walker-2             1/1       Running   0          8m        172.16.6.249      walker-2
kube-dns-8bb5c479-7rsjv                      3/3       Running   0          1h        192.168.187.196   walker-1.novalocal
kube-proxy-66c6x                             1/1       Running   0          8m        172.16.6.249      walker-2
kube-proxy-k5zwm                             1/1       Running   0          1h        172.16.6.47       walker-1.novalocal
kube-scheduler-walker-1.novalocal            1/1       Running   0          1h        172.16.6.47       walker-1.novalocal
kube-scheduler-walker-2                      1/1       Running   0          7m        172.16.6.249      walker-2

至此搭建成功，完成的一大半。

Note: 值得一提的是，最好將 etcd 服務器地址用ip表示。使用域名的時候，在controller2 上的apiserver會出現連接etcd超時的情況。儘管在hosts文件裏面添加了相關記錄，也不行。但是在controller1上卻不會受此影響，很詫異。

lvs 搭建

toplogy:
        lvs(172.16.6.56)
             | ^
             | | vip:172.16.6.79
             v |
    -----------------------
        | ^         | ^
        | |         | |
        v |         v |
rs1(172.16.6.47)  rs2(172.16.6.249)

lvs

添加vip

[root@lvs ~]# ip addr add 172.16.6.79 dev eth1
[root@lvs ~]# ip addr
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:c2:e8:12 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.56/24 brd 172.16.6.255 scope global dynamic eth1
       valid_lft 67967sec preferred_lft 67967sec
    inet 172.16.6.79/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec2:e812/64 scope link 
       valid_lft forever preferred_lft forever

lvs配置

[root@lvs ~]# ipvsadm -A -t 172.16.6.79:6443 -s rr
[root@lvs ~]# ipvsadm -a -t 172.16.6.79:6443 -r 172.16.6.47:6443
[root@lvs ~]# ipvsadm -a -t 172.16.6.79:6443 -r 172.16.6.249:6443
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.6.79:6443 rr
  -> 172.16.6.47:6443             Route   1      0          0         
  -> 172.16.6.249:6443            Route   1      1          0

設置調度方法爲輪詢。

Note: lvs不會對realserver做健康檢查，這意味着當rs2服務失敗後，請求仍會被調度到rs2上。爲了剔除失敗的rs,需要配合keepalived來完成

keepalived 配置如下：

[root@lvs ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id LVS_DEVEL
}

vrrp_instance lvsgroup {
    state MASTER                # 標識該主機爲MASTER
    interface eth1              # 主機網卡，vip綁定的網卡
    virtual_router_id 81
    priority 100                    # 優先級，要比BACKUP主機的大
    advert_int 1                    # VRRP multicast 廣播週期秒數
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.6.79             #定義vip,可有多個，換行添加
    }
}

virtual_server 172.16.6.79 6443 {
    delay_loop 5                # 每隔6s查看realserver狀態
    lb_algo rr                  # realserver調度算法
    lb_kind DR                  # lvs工作模式
    # persistence_timeout 3         # 同一ip的連接，在50s分配到同一臺realserver
    protocol TCP                    # 使用tcp檢測realserver狀態

    real_server 172.16.6.47 6443 {  
        weight 1                
        TCP_CHECK {
            cennect_timeout 10
        }
    }

    real_server 172.16.6.47 6443 {
        weight 1                
        TCP_CHECK {
            cennect_timeout 10
        }
    }
}

配置好keepalived之後就不用對lvs做任何操作了。keepalived進程會接管lvs，動態更新lvs後端rs.

rs{1,2}

編寫realserver 配置腳本lvs_rs.sh，內容如下:

#!/bin/bash
SNS_VIP=172.16.6.79

case "$1" in
start)
   ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
   route add -host $SNS_VIP dev lo:0
   echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
   echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
   echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
   echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
   sysctl -p >/dev/null 2>&1
   echo "LVS RealServer Start OK"
;;
stop)
   ifconfig lo:0 down
   route del $SNS_VIP >/dev/null 2>&1
   echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
   echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
   echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
   echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
   echo "LVS RealServer Stoped"
;;
*)
   echo "Usage: $0 {start|stop}"
   exit 1
esac

爲了防止rs重啓後失效，可加入到開機自啓:

[root@rs1 ~]# echo "sh /root/lvs_rs.sh start" >> /etc/rc.d/rc.local
[root@rs1 ~]# chmod +x /etc/rc.d/rc.local

centos7 中，降低了rc.local的執行權限。推薦使用systemd來管理。

測試

[root@controller ~]# telnet 172.16.6.79 6443
Trying 172.16.6.79...
Connected to 172.16.6.79.
Escape character is '^]'.

搭好lvs後，需要修改kubectl, kube-proxy以及cluster-info的cm配置，將apiserver地址指向 vip。

至此集羣搭建完畢

參考文檔：

https://www.cnblogs.com/keithtt/p/7896948.html
https://kubernetes.io/docs/admin/authentication/
https://kubernetes.io/docs/admin/kubelet-authentication-authorization/
https://kubernetes.io/docs/concepts/cluster-administration/certificates/

discsthnew

發佈了31 篇原創文章 · 獲贊 6 · 訪問量 7萬+

私信關注

kubernetes 1.8.1 高可用

preflight

安裝包

鏡像

etcd 集羣

搭建kubernetes集羣

拓撲

通過kubeadm初始化

添加備份節點

lvs 搭建

lvs

rs{1,2}

測試

釘釘打卡速度慢

Nginx R31 doc 官方文檔-01-nginx 如何安裝

Qt/C++音視頻開發74-合併標籤圖形/生成yolo運算結果圖形/文字和圖形合併成一個/水印濾鏡

挑戰程序設計競賽 2.2章習題 POJ - 3617 Best Cow Line 貪心

字節面試：MySQL什麼時候鎖表？如何防止鎖表？

.NET8連接SQL SERVER 2008 R2 報：證書鏈是由不受信任的頒發機構頒發的

golang開發環境搭建(win10)

python計算機視覺學習筆記——PIL庫的用法

Golang初學：獲取程序內存使用情況，std runtime

k8s 身份驗證

安裝harbor 鏡像倉庫

Calico 網絡互連架構

k8s獲取client ip

Ceph ARCHITECTURE

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結