kubernetes-1.11.0集羣部署之master集羣 (二)

kubernetes-1.11.0集羣部署之master集羣 (二)

Master

master 的組件有以下: 

1. API Server
   Api server提供HTTP/HTTPS RESTful  API,既kubernetes API,API server 是kubernetes Cluster的前端接口，各種客戶端工具(CLI或UI)以及kubernetes其他組件可以通過它管理cluster各種資源

2. Scheduler(kube-scheduler)
   scheduler 負責決定Pod放在哪個node 上運行.scheduler 在調度時會充分考慮cluster 的拓撲結構，當前各個節點的負載，以及應用對高可用、性能、數據親和性的需求. 

3. Controller Manager(kube-controller-manager)      
   Controller Manager 負責管理CLuster 各種資源，保障資源處於預期的狀態，Controller Manager 由多種controller 組成，包括replication  controller、endpoints  controller 、namespace controller、serviceaccounts controller 等。

  不同的controller 管理不同的資源，例如,replication controller管理Deployment、SatefulSet、DaemonSet 的生命週期,namespace controller 管理namespace 資源

4. etcd 
   etcd負責保存kubernetes Cluster的配置信息和各種資源的狀態信息.當數據發生變化時，etcd快速通知kubernetes 相關組件

5.Pod 
  Pod 要能夠相互通信，kubernetes Cluster 必須部署Pod 網絡，網絡組件可選擇fannel calico  
  (kubectl 安裝在所有需要進行操作的機器上）

6.master 需要安裝的組建
kube-apiserver，kube-controller-manager，kube-scheduler，kubectl，kubelet，kubeadm

安裝組件

將之前編譯好的二進制文件,kube-apiserver，kube-controller-manager，kube-scheduler，kubectl，kubelet，kubeadm 拷貝到/usr/local/bin/  所有master 進行同步

創建admin 證書
kubectl 與 kube-apiserver 的安全端口通信，需要爲安全通信提供TLS 證書和祕鑰

vim  admin-csr.json
  {
    "CN": "admin",
    "hosts": [],
    "key": {
    "algo": "rsa",
    "size": 2048
    },
  "names": [
  {
   "C": "CN",
   "ST": "BeiJing",
   "L": "BeiJing",
   "O": "system:masters",
   "OU": "System"
   }
  ]
 }

生成admin 證書和祕鑰
cd /opt/ssl/ 
   cfssl gencert -ca=ca.pem  -ca-key=ca-key.pem  -config=config.json  -profile=kubernetes admin-csr.json | cfssljson -bare admin 

證書拷貝到master 的機器上，154，156.159  
scp ssl/admin*   [email protected]:/etc/kubernetes/ssl/ 
...........

生成kubernetes配置文件

#生成證書相關配置文件存儲於/root/.kube 目錄中

配置kubernetes 集羣

在10.39.10.154 這臺機器上操作
kubectl config   set-cluster   kubernetes   --certificate-authority=/etc/kubernetes/ssl/ca.pem  --embed-certs=true   --server=https://127.0.0.1:6443 

配置客戶端認證
kubectl config set-credentials  admin  --client-certificate=/etc/kubernetes/ssl/admin.pem  --embed-certs=true  --client-key=/etc/kubernetes/ssl/admin-key.pem 

kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin

kubectl config use-context kubernetes

創建kubernetes 證書

vim kubernetes-csr.json 
 {
  "CN": "kubernetes",
  "hosts": [
  "127.0.0.1",
  "10.39.10.154",
  "10.39.10.156",
  "10.39.10.159",
  "10.254.0.1",
  "kubernetes",
  "kubernetes.default",
  "kubernetes.default.svc",
  "kubernetes.default.svc.cluster",
  "kubernetes.default.svc.cluster.local"
    ],
   "key": {
   "algo": "rsa",
   "size": 2048
   },
   "names": [
   {
     "C": "CN",
     "ST": "BeiJing",
     "L": "BeiJing",
     "O": "k8s",
     "OU": "System"
     }
    ]
  } 

# 這裏 hosts 字段中 三個 IP 分別爲 127.0.0.1 本機，10.39.10.154 和 10.39.10.156，10.39.10.159 爲 Master 的IP，多個Master需要寫多個。  10.254.0.1 爲 kubernetes SVC 的 IP， 一般是 部署網絡的第一個IP , 如: 10.254.0.1 ， 在啓動完成後，我們使用   kubectl get svc ， 就可以查看到 

生成kubernetes 證書和私鑰 
cfssl gencert -ca=ca.pem  -ca-key=ca-key.pem  -config=config.json  -profile=kubernetes   kubernetes-csr.json | cfssljson  -bare kubernetes

拷貝kubernetes*.pem  /etc/kubernetes/ssl/     
scp kubernetes*.pem [email protected]:/etc/kubernetes/ssl/
........

配置kube-apiserver

[root@devops ssl]# head -c 16 /dev/urandom  | od -An -t x | tr -d ' '
da4090c6baadef99e577a9ac5da6f684
創建encryption-config.yaml 配置
 cat > encryption-config.yaml <<EOF
    kind: EncryptionConfig
    apiVersion: v1
    resources:
       - resources:
        - secrets
providers:
  - aescbc:
      keys:
        - name: key1
          secret: 40179b02a8f6da07d90392ae966f7749
  - identity: {}
EOF

#拷貝
scp encryption-config.yaml [email protected]:/etc/kubernetes/  
.......  

#生成高級審覈配置文件 

[https://kubernetes.io/docs/tasks/debug-application-cluster/audit/]()

 cat >> audit-policy.yaml <<EOF
    # Log all requests at the Metadata level.
    apiVersion: audit.k8s.io/v1beta1
    kind: Policy
   rules:
   - level: Metadata
 EOF 
拷貝到所有master 機器上
scp audit-policy.yaml   [email protected]:/etc/kubernetes/    
創建kube-apiserver.service 文件
# 自定義系統
# master 機器上都需要配置
vi /etc/systemd/system/kube-apiserver.service

[Unit]
 Description=Kubernetes API Server
 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--anonymous-auth=false \
--experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
--advertise-address=10.39.10.154 \
--allow-privileged=true \
--apiserver-count=3 \
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kubernetes/audit.log \
--authorization-mode=Node,RBAC \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--kubelet-client-certificate=/etc/kubernetes/ssl/kubernetes.pem \
--kubelet-client-key=/etc/kubernetes/ssl/kubernetes-key.pem \
--enable-swagger-ui=true \
--etcd-cafile=/etc/kubernetes/ssl/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
--etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
--etcd-servers=https://10.39.10.154:2379,https://10.39.10.156:2379,https://10.39.10.159:2379 \
--event-ttl=1h \
--kubelet-https=true \
--insecure-bind-address=127.0.0.1 \
--insecure-port=8080 \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-cluster-ip-range=10.254.0.0/18 \
--service-node-port-range=30000-32000 \
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--enable-bootstrap-token-auth \
--v=1
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

啓動kube-apiserver

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver        

配置kube-controller-manager

vim /etc/systemd/system/kube-controller-manager.service
   [Unit]
   Description=Kubernetes Controller Manager
   Documentation=https://github.com/GoogleCloudPlatform/kubernetes

   [Service]
    ExecStart=/usr/local/bin/kube-controller-manager \
    --address=0.0.0.0 \
    --master=http://127.0.0.1:8080 \
    --allocate-node-cidrs=true \
    --service-cluster-ip-range=10.254.0.0/18 \
   --cluster-cidr=10.254.64.0/18 \
   --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
   --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
   --feature-gates=RotateKubeletServerCertificate=true \
   --controllers=*,tokencleaner,bootstrapsigner \
   --experimental-cluster-signing-duration=86700h0m0s \
   --cluster-name=kubernetes \
   --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
   --root-ca-file=/etc/kubernetes/ssl/ca.pem \
   --leader-elect=true \
   --node-monitor-grace-period=40s \
   --node-monitor-period=5s \
   --pod-eviction-timeout=5m0s \
   --v=2
    Restart=on-failure
    RestartSec=5

    [Install]
    WantedBy=multi-user.target

啓動kube-controller-manager

  systemctl daemon-reload
  systemctl enable kube-controller-manager
  systemctl start kube-controller-manager
  systemctl status kube-controller-manager

配置kube-scheduler

vi /etc/systemd/system/kube-scheduler.service  
  [Unit]
    Description=Kubernetes Scheduler
   Documentation=https://github.com/GoogleCloudPlatform/kubernetes

 [Service]
 ExecStart=/usr/local/bin/kube-scheduler \
 --address=0.0.0.0 \
 --master=http://127.0.0.1:8080 \
 --leader-elect=true \
 --v=1
 Restart=on-failure
 RestartSec=5

 [Install]
 WantedBy=multi-user.target  

啓動kube-scheduler

 systemctl daemon-reload
 systemctl enable kube-scheduler
 systemctl start kube-scheduler
 systemctl status kube-scheduler    

驗證master 節點

每臺master 上執行kubectl get componentstatuses 

配置 kubelet 認證

 kubelet 授權 kube-apiserver 的一些操作 exec run logs 等
 # RBAC 只需創建一次就可以
 kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

創建 bootstrap kubeconfig 文件

#注意: token 生效時間爲 1day , 超過時間未創建自動失效，需要重新創建 token 
# 創建 集羣所有 kubelet 的 token
在master 上執行
kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-154 --kubeconfig ~/.kube/config 

kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-156 --kubeconfig ~/.kube/config

 kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-159 --kubeconfig ~/.kube/config
# 查看生成的token  
kubeadm token list --kubeconfig ~/.kube/config 

以下爲了區分會先生成node名稱加bootstrap.kubeconfig

 生成kubernetes-master-154 bootstrap.kubeconfig
 kubectl config set-cluster kubernetes   --certificate-authority=/etc/kubernetes/ssl/ca.pem  --embed-certs=true  --server=https://127.0.0.1:6443 --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig

 #配置客戶端認證
 kubectl config set-credentials  kubelet-bootstrap  --token=aaa8j5.4nvwg82imbrzb7r2   --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig

 #配置關聯
kubectl config set-context default  --cluster=kubernetes  --user=kubelet-bootstrap  --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig

 #配置默認關鍵
 kubectl config use-context default  --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig

 拷貝生成的kubernetes-master-154-bootstrap.kubeconfig 文件
 mv kubernetes-master-154-bootstrap.kubeconfig  /etc/kubernetes/bootstrap.kubeconfig

 生成kubernetes-master-156 bootstrap.kubeconfig    
 kubectl config set-cluster kubernetes  --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true  --server=https://127.0.0.1:6443   --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig

#配置客戶端認證
 kubectl config set-credentials kubelet-bootstrap   --token=rz2col.l8x1x9dg5kg7jjw6  --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig

#配置關聯
kubectl config set-context default \
 --cluster=kubernetes \
 --user=kubelet-bootstrap \
 --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig

#配置默認關聯
kubectl config use-context default --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig

#拷貝生成的kubernetes-master-156-bootstrap.kubeconfig  
 mv kubernetes-master-156-bootstrap.kubeconfig  /etc/kubernetes/bootstrap.kubeconfig

# 生成159的bootstrap.kubeconfig  

#配置集羣參數
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=kubernetes-master-159-bootstrap.kubeconfig 

#配置客戶端認證
kubectl config set-credentials kubelet-bootstrap \
--token=9ocdef.pjd1s7twtro2ho8a \
--kubeconfig=kubernetes-master-159-bootstrap.kubeconfig

#配置關聯
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
 --kubeconfig=kubernetes-master-159-bootstrap.kubeconfig

#配置默認關聯
kubectl config use-context default --kubeconfig=kubernetes-master-159-bootstrap.kubeconfig

#拷貝生成
 mv kubernetes-master-159-bootstrap.kubeconfig  /etc/kubernetes/bootstrap.kubeconfig 

配置bootstrap RBAC 權限

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

創建自動批准相關CSR請求ClusterRole

vim /etc/kubernetes/tls-instructs-csr.yaml 

 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
 rules:
 - apiGroups: ["certificates.k8s.io"]
   resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]

 #導入yaml 文件
 kubectl apply -f /etc/kubernetes/tls-instructs-csr.yaml 

 #查看  
 kubectl describe ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver 

 #將ClusterRole 綁定到適當的用戶組
 #自動批准 system:bootstrappers 組用戶 TLS bootstrapping 首次申請證書的 CSR 請求

  kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers

 #自動批准 system:nodes 組用戶更新 kubelet 自身與 apiserver 通訊證書的 CSR 請求

  kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes  

#自動批准 system:nodes 組用戶更新 kubelet 10250 api 端口證書的 CSR 請求

 kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes

創建 kubelet.service 文件

關於kubectl get  node 中的ROLES 的標籤
單 Master 打標籤 kubectl label node kubernetes-master-154   node-role.kubernetes.io/master=''    
這裏需要將 單Master 更改爲 NoSchedule
更新標籤命令爲 kubectl taint nodes kubernetes-64 node-role.kubernetes.io/master=:NoSchedule
既 Master 又是 node 打標籤 kubectl label node kubernetes-65 node-role.kubernetes.io/master=””
單 Node 打標籤 kubectl label node kubernetes-66 node-role.kubernetes.io/node=””
關於刪除 label 可使用 - 號相連 如: kubectl label nodes kubernetes-65 node-role.kubernetes.io/node-

 #創建kubelet.service
 vi /etc/systemd/system/kubelet.service 
 [Unit]
 Description=Kubernetes Kubelet
 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
 After=docker.service
 Requires=docker.service

 [Service]
 WorkingDirectory=/var/lib/kubelet
 ExecStart=/usr/local/bin/kubelet \
 --hostname-override=kubernetes-master-154 \
 --pod-infra-container-image=jicki/pause-amd64:3.1 \
 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
 --config=/etc/kubernetes/kubelet.config.json \
 --cert-dir=/etc/kubernetes/ssl \
 --logtostderr=true \
 --v=2

[Install]
WantedBy=multi-user.target

# 創建 kubelet config 配置文件
vi /etc/kubernetes/kubelet.config.json

  {
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
  },
"webhook": {
  "enabled": true,
  "cacheTTL": "2m0s"
},
"anonymous": {
  "enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
  "cacheAuthorizedTTL": "5m0s",
  "cacheUnauthorizedTTL": "30s"
}
},
"address": "10.39.10.154",
"port": 10250,
"readOnlyPort": 0,
"cgroupDriver": "cgroupfs",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"RotateCertificates": true,
"featureGates": {
 "RotateKubeletClientCertificate": true,
 "RotateKubeletServerCertificate": true
},
"MaxPods": "512",
"failSwapOn": false,
"containerLogMaxSize": "10Mi",
"containerLogMaxFiles": 5,
"clusterDomain": "cluster.local.",
"clusterDNS": ["10.254.0.2"]
}  

啓動kubelet

 systemctl daemon-reload
  systemctl enable kubelet
 systemctl start kubelet
 systemctl status kubelet

總結一下

master 安裝的有
systemctl status kubelet 
systemctl status kube-scheduler 
systemctl status kube-controller-manager
systemctl status kube-apiserver