k8s三部曲(1)——kubernetes 完整集羣部署(二進制安裝)

原創 小兔爷 2020-06-23 19:18

kubernetes 完整集羣部署（二進制安裝）

1.準備環境

  1. 安裝環境 CentOS7 服務器 3臺

  2. 準備安裝包

    etcd-v3.3.15-linux-amd64.tar.gz

    kubernetes-server-linux-amd64.tar.gz

    docker-17.03.2-ce.tgz

    flanneld 0.7.1

    cfssl 1.2.0

2.安裝規劃

節點IP 角色 安裝的組件
192.168.0.111 Master etcd、kube-apiserver、kube-controller-manager、kube-scheduler、cfssl、kubectl
192.168.0.112 Node1 docker 、kubelet、kube-proxy、flanneld 、cfssl、kubectl
192.168.0.113 Node2 docker 、kubelet、kube-proxy flanneld、cfssl 、kubectl

3.預備SSL/TSL證書

  1. 安裝cfssl(所有節點)

    $ yum install -y wget
#下載
$ wget -q --show-progress --https-only --timestamping \
  https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \
  https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
#修改爲可執行權限
$ chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
#移動到bin目錄
$ mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
#驗證
$ cfssl version

  2. 準備生成證書的配置文件ca-config.json(主節點)

    {
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

  3. 準備生成證書的配置文件ca-csr.json(主節點)

    {
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

  4. 生成根證書（主節點）

    1.根證書是證書信任鏈的根，各個組件通訊的前提是有一份大家都信任的證書(根證書)，每個人使用的證書都是由這個根證書籤發的。
    2.生成根證書後將根證書複製到其他節點以便生成子證書或者在主節點上生成資子證書然後複製到其他節點

    #所有證書相關的東西都放在這
$ mkdir -p /etc/kubernetes/ca
#準備生成證書的配置文件
$ cp ca-config.json /etc/kubernetes/ca
$ cp ca-csr.json /etc/kubernetes/ca
#生成證書和祕鑰
$ cd /etc/kubernetes/ca
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
#生成完成後會有以下文件（我們最終想要的就是ca-key.pem和ca.pem，一個祕鑰，一個證書）
$ ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

4.etcd部署

  1. 準備證書

    0.1 編寫etcd-csr.json 文件(主節點)

    {
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.0.111				
"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

    0.2 簽發etcd證書(主節點)

    #etcd證書放在這
$ mkdir -p /etc/kubernetes/ca/etcd
#準備etcd證書配置
$ cp etcd-csr.json /etc/kubernetes/ca/etcd/
$ cd /etc/kubernetes/ca/etcd/
#使用根證書(ca.pem)簽發etcd證書
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
#跟之前類似生成三個文件etcd.csr是個中間證書請求文件，我們最終要的是etcd-key.pem和etcd.pem
$ ls
etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

  2. 解壓etcd-v3.3.15-linux-amd64.tar.gz

    tar -zxvf etcd-v3.3.15-linux-amd64.tar.gz

  3. 解壓後，將etcd、etcdctl 文件複製到/usr/bin目錄

    cd etcd-v3.3.15-linux-amd64
cp etcd etcdctl /usr/bin

  4. 設置etcd.service服務文件

    在/etc/systemd/system/目錄下創建etcd.service文件，內容如下：

    [Unit]
Description=etcd.service
[Service]
Type=notify
TimeoutStartSec=0
Restart=always
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd 
[Install]
WantedBy=multi-user.target

    創建/var/lib/etcd 目錄

    mkdir /var/lib/etcd

  5. 創建配置/etc/etcd/etcd.conf文件，內容如下

    ETCD_NAME=ETCD Server
ETCD_DATA_DIR="/var/lib/etcd/"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.111:2379,http://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.111:2379"
ETCD_LISTEN_PEER_URLS="https://192.168.0.111:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.111:2380"
ETCD_CERT_FILE="/etc/kubernetes/ca/etcd/etcd.pem"
ETCD_KEY_FILE="/etc/kubernetes/ca/etcd/etcd-key.pem"
ETCD_PEER_CERT_FILE="/etc/kubernetes/ca/etcd/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/kubernetes/ca/etcd/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/ca/ca.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/ca/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_AUTO_TLS="true"

  6. 配置開機啓動

    systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service

  7. 檢驗etcd是否按照成功

    ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.0.111:2379  \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd/etcd.pem \
--key=/etc/kubernetes/ca/etcd/etcd-key.pem \
endpoint health

#顯示health代表正常
https://192.168.0.111:2379 is healthy: successfully committed proposal: took = 23.87696ms

5.kube-apiserver部署

  1. 準備證書

    0.1 編寫kubernetes-csr.json 文件(主節點)

    {
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.0.111",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

    0.2 生成證書(主節點)

    #api-server證書放在這，api-server是核心，文件夾叫kubernetes吧，如果想叫apiserver也可以，不過相關的地方都需要修改哦
$ mkdir -p /etc/kubernetes/ca/kubernetes
#準備apiserver證書配置
$ cp kubernetes-csr.json /etc/kubernetes/ca/kubernetes/
$ cd /etc/kubernetes/ca/kubernetes/
#使用根證書(ca.pem)簽發kubernetes證書
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
#跟之前類似生成三個文件kubernetes.csr是個中間證書請求文件，我們最終要的是kubernetes-key.pem和kubernetes.pem
$ ls
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

  2. 解壓kubernetes-server-linux-amd64.tar.gz

    tar -zxvf kubernetes-server-linux-amd64.tar.gz

  3. 解壓後，將kube-apiserver 文件複製到/usr/bin目錄

    cp kube-apiserver /usr/bin

  4. 設置kube-apiserver.service服務文件

    在/etc/systemd/system/目錄下創建kube-apiserver.service，內容如下：

    –advertise-address : 提供給其他組件使用的內部kube-apiserver地址
    –bind-address: 安全端口https監聽的host
    –service-cluster-ip-range：集羣的ip範圍

    [Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/usr/bin/kube-apiserver \
	--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
	--insecure-bind-address=127.0.0.1 \
	--kubelet-https=true \
	--advertise-address=192.168.0.111 \	
	--bind-address=192.168.0.111 \
	--authorization-mode=Node,RBAC \
	--runtime-config=rbac.authorization.k8s.io/v1 \
	--enable-bootstrap-token-auth  \
	--token-auth-file=/etc/kubernetes/ca/kubernetes/token.csv  \
	--tls-cert-file=/etc/kubernetes/ca/kubernetes/kubernetes.pem \
	--tls-private-key-file=/etc/kubernetes/ca/kubernetes/kubernetes-key.pem  \
	--client-ca-file=/etc/kubernetes/ca/ca.pem \
	--service-account-key-file=/etc/kubernetes/ca/ca-key.pem \
	--etcd-cafile=/etc/kubernetes/ca/ca.pem \
	--etcd-certfile=/etc/kubernetes/ca/kubernetes/kubernetes.pem  \
	--etcd-keyfile=/etc/kubernetes/ca/kubernetes/kubernetes-key.pem \
	--service-cluster-ip-range=192.168.0.0/24  \
	--service-node-port-range=20000-40000 \
	--etcd-servers=http://127.0.0.1:2379  \
	--enable-swagger-ui=true \
	--allow-privileged=true \
	--audit-log-maxage=30 \
	--audit-log-maxbackup=3 \
	--audit-log-maxsize=100 \
	--audit-log-path=/var/lib/audit.log \
	--event-ttl=1h
	--v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target:

  5. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable kube-apiserver.service
systemctl start kube-apiserver.service
systemctl status kube-apiserver.service

  6. 生成隨機token

    後期需要通過token認證方式接入集羣

    #生成隨機token
$ head -c 16 /dev/urandom | od -An -t x | tr -d ' '
64e3bafc3a02f2a56aa3570b9f6c86ef

#按照固定格式寫入token.csv，注意替換token內容
$ echo "64e3bafc3a02f2a56aa3570b9f6c86ef,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"" > /etc/kubernetes/ca/kubernetes/token.csv

6.kube-controller-manager部署

  1. 無需證書(主節點)

    controller-manager一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。

  2. 解壓kubernetes-server-linux-amd64.tar.gz

    tar -zxvf kubernetes-server-linux-amd64.tar.gz

  3. 解壓後，將kube-controller-manager 文件複製到/usr/bin目錄

    cp kube-controller-manager /usr/bin

  4. 設置kube-controller-manager.service服務文件

    在/etc/systemd/system/目錄下創建kube-controller-manager.service，內容如下：

    [Unit]
Description=Kubernetes Scheduler
After=kube-apiserver.service 
Requires=kube-apiserver.service
 
[Service]
ExecStart=/usr/bin/kube-controller-manager \
        --master=http://127.0.0.1:8080 \
        --logtostderr=true --log-dir /var/log/kubernetes --v=2
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

  5. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable kube-controller-manager.service
systemctl start kube-controller-manager.service
systemctl status kube-controller-manager.service

7.kube-scheduler部署

  1. 無需證書(主節點)

    controller-manager一般與api-server在同一臺機器上，所以可以使用非安全端口與api-server通訊，不需要生成證書和私鑰。

  2. 解壓kubernetes-server-linux-amd64.tar.gz

    tar -zxvf kubernetes-server-linux-amd64.tar.gz

  3. 解壓後，將kube-scheduler 文件複製到/usr/bin目錄

    cp kube-scheduler /usr/bin

  4. 設置kube-scheduler.service服務文件

    在/etc/systemd/system/目錄下創建kube-scheduler.service，內容如下：

    [Unit]
Description=Kubernetes Scheduler
After=kube-apiserver.service 
Requires=kube-apiserver.service
 
[Service]
User=root
ExecStart=/usr/bin/kube-scheduler --master=http://127.0.0.1:8080 --logtostderr=true --log-dir /var/log/kubernetes --v=2
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

  5. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable kube-scheduler.service
systemctl start kube-scheduler.service
systemctl status kube-scheduler.service

8.kubectl部署

  1. 準備證書

    前言：
    1.kubectl不部署則跳過此步驟，只需將kubectl複製到主節點的/usr/bin 即可（但是之後創建的一些認證配置文件需要自行遷移到其他主機）
    2.kubectl部署在所有節點，需要配置連接信息，則如下所示（可以直接在當前主機創建配置文件）

    0.1 編寫admin-csr.json文件

    {
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

    0.2 創建證書

    #kubectl證書放在這，由於kubectl相當於系統管理員，我們使用admin命名
$ mkdir -p /etc/kubernetes/ca/admin
#準備admin證書配置 - kubectl只需客戶端證書，因此證書請求中 hosts 字段可以爲空
$ cp admin-csr.json /etc/kubernetes/ca/admin/
$ cd /etc/kubernetes/ca/admin/
#使用根證書(ca.pem)簽發admin證書
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes admin-csr.json | cfssljson -bare admin
#我們最終要的是admin-key.pem和admin.pem
$ ls
admin.csr  admin-csr.json  admin-key.pem  admin.pem

  2. 配置kubectl

    #指定apiserver的地址和證書位置（ip自行修改）
$ kubectl config set-cluster kubernetes \
        --certificate-authority=/etc/kubernetes/ca/ca.pem \
        --embed-certs=true \
        --server=https://192.168.0.111:6443
#設置客戶端認證參數，指定admin證書和祕鑰
$ kubectl config set-credentials admin \
        --client-certificate=/etc/kubernetes/ca/admin/admin.pem \
        --embed-certs=true \
        --client-key=/etc/kubernetes/ca/admin/admin-key.pem
#關聯用戶和集羣
$ kubectl config set-context kubernetes \
        --cluster=kubernetes --user=admin
#設置當前上下文
$ kubectl config use-context kubernetes

#設置結果就是一個配置文件，可以看看內容
$ cat ~/.kube/config

  3. 驗證master節點

    kubectl get componentstatus

NAME                 AGE
scheduler            <unknown>
controller-manager   <unknown>
etcd-0               <unknown>

9.kube-proxy部署

  1. 準備證書(工作節點)

    0.1 編寫kube-proxy-csr.json文件

    {
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

    0.2 生成證書

    #proxy證書放在這
$ mkdir -p /etc/kubernetes/ca/kube-proxy
#準備proxy證書配置 - proxy只需客戶端證書，因此證書請求中 hosts 字段可以爲空。
#CN 指定該證書的 User 爲 system:kube-proxy，預定義的 ClusterRoleBinding system:node-proxy 將User system:kube-proxy 與 Role system:node-proxier 綁定，授予了調用 kube-api-server proxy的相關 API 的權限
$ cp kube-proxy-csr.json /etc/kubernetes/ca/kube-proxy/
$ cd /etc/kubernetes/ca/kube-proxy/

#使用根證書(ca.pem)簽發kubr-proxy證書
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
#我們最終要的是kube-proxy-key.pem和kube-proxy.pem
$ ls
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

    0.3 生成kube-proxy.kubeconfig配置

    #設置集羣參數（注意替換ip）
$ kubectl config set-cluster kubernetes \
        --certificate-authority=/etc/kubernetes/ca/ca.pem \
        --embed-certs=true \
        --server=https://192.168.0.111:6443 \
        --kubeconfig=kube-proxy.kubeconfig
#設置客戶端認證參數
$ kubectl config set-credentials kube-proxy \
        --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem \
        --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem \
        --embed-certs=true \
        --kubeconfig=kube-proxy.kubeconfig
#設置上下文參數
$ kubectl config set-context default \
        --cluster=kubernetes \
        --user=kube-proxy \
        --kubeconfig=kube-proxy.kubeconfig
#選擇上下文
$ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
#移動到合適位置
$ mv kube-proxy.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig

  2. 解壓kubernetes-server-linux-amd64.tar.gz

    tar -zxvf kubernetes-server-linux-amd64.tar.gz

  3. 解壓後，將kube-proxy 文件複製到/usr/bin目錄

    cp kube-proxy /usr/bin

  4. 設置kube-proxy.service服務文件

    在/etc/systemd/system/目錄下創建kube-proxy.service，內容如下：

    [Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
ExecStart=/usr/bin/kube-proxy \
   --hostname-override=192.168.0.112 \
   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
   --logtostderr=true
Restart=on-failure

[Install]
WantedBy=multi-user.target

  5. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable kube-proxy.service
systemctl start kube-proxy.service
systemctl status kube-proxy.service

10.kubelet部署

  1. 部署準備

    0.1 創建角色綁定（主節點）

    引導token的方式要求客戶端向api-server發起請求時告訴他你的用戶名和token，並且這個用戶是具有一個特定的角色：system:node-bootstrapper，所以需要先將bootstrap token 文件中的 kubelet-bootstrap 用戶賦予這個特定角色，然後 kubelet纔有權限發起創建認證請求。 在主節點執行下面命令

    #可以通過下面命令查詢clusterrole列表
kubectl -n kube-system get clusterrole

#可以回顧一下token文件的內容
cat /etc/kubernetes/ca/kubernetes/token.csv
64e3bafc3a02f2a56aa3570b9f6c86ef,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

#創建角色綁定（將用戶kubelet-bootstrap與角色system:node-bootstrapper綁定）
kubectl create clusterrolebinding kubelet-bootstrap \
         --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

    0.2 創建bootstrap.kubeconfig（工作節點）

    這個配置是用來完成bootstrap token認證的，保存了用戶，token等重要的認證信息，這個文件可以藉助kubectl命令生成：（也可以自己寫配置）

    #設置集羣參數(注意替換ip)
$ kubectl config set-cluster kubernetes \
        --certificate-authority=/etc/kubernetes/ca/ca.pem \
        --embed-certs=true \
        --server=https://192.168.0.111:6443 \
        --kubeconfig=bootstrap.kubeconfig
#設置客戶端認證參數(注意替換token)
$ kubectl config set-credentials kubelet-bootstrap \
        --token=64e3bafc3a02f2a56aa3570b9f6c86ef \
        --kubeconfig=bootstrap.kubeconfig
#設置上下文
$ kubectl config set-context default \
        --cluster=kubernetes \
        --user=kubelet-bootstrap \
        --kubeconfig=bootstrap.kubeconfig
#選擇上下文
$ kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#將剛生成的文件移動到合適的位置
$ mv bootstrap.kubeconfig /etc/kubernetes/

  2. 解壓kubernetes-server-linux-amd64.tar.gz

    tar -zxvf kubernetes-server-linux-amd64.tar.gz

  3. 解壓後，將kubelet 文件複製到/usr/bin目錄

    cp kubelet /usr/bin

  4. 設置kubelet.service服務文件

    在/etc/systemd/system/目錄下創建kubelet.service，內容如下：
    cluster-dns ：kube-dns服務的ip地址，稍後創建相應的kube-dns服務

    [Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/bin/kubelet \
  --hostname-override=192.168.0.112\
  --kubeconfig=/etc/kubernetes/bootstrap.kubeconfig\
  --cluster-dns=192.168.0.2 \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ca \
  --hairpin-mode hairpin-veth \
  --cluster-domain=cluster.local \
  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0\
  --logtostderr=true
#kubelet cAdvisor 默認在所有接口監聽 4194 端口的請求, 以下iptables限制內網訪問
ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP
Restart=on-failure
KillMode=process

    記得創建 /var/lib/kubelet 目錄

  5. 禁用swap分區

    sudo swapoff -a

#要永久禁掉swap分區，打開如下文件註釋掉swap那一行
sudo vi /etc/fstab

  6. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable kubelet.service
systemctl start kubelet.service
systemctl status kubelet.service

    #啓動kubelet之後到 master節點 允許worker加入(批准worker的tls證書請求)
#--------*在主節點執行*---------
$ kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
#-----------------------------

#檢查日誌
$ journalctl -f -u kubelet

11.docker 安裝

  1. 解壓docker-17.03.2-ce.tgz

    tar -zxvf docker-17.03.2-ce.tgz

  2. 解壓後，將docker 文件複製到/usr/bin目錄

    cp docker* /usr/bin

  3. 設置docker.service服務文件

    在/etc/systemd/system/目錄下創建docker.service，內容如下：

    [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
  
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
  
[Install]
WantedBy=multi-user.target

  4. 配置開機啓動並查看狀態

    systemctl daemon-reload
systemctl enable docker
systemctl start docker
systemctl status docker

12. flanneld 安裝

  1. 準備證書

    0.1 編寫flanneld-csr.json文件

    	{
	  "CN": "flanneld",
	  "hosts": [],
	  "key": {
	    "algo": "rsa",
	    "size": 2048
	  },
	  "names": [
	    {
	      "C": "CN",
	      "ST": "Beijing",
	      "L": "XS",
	      "O": "k8s",
	      "OU": "System"
	    }
	  ]
	}

    0.2 生成證書

    #flanneld證書放在這
$ mkdir -p /etc/kubernetes/ca/flanneld
#準備flanneld證書配置 - flanneld只需客戶端證書，因此證書請求中 hosts 字段可以爲空
$ cp flanneld-csr.json /etc/kubernetes/ca/flanneld/
$ cd /etc/kubernetes/ca/flanneld/
#使用根證書(ca.pem)簽發flanneld證書
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes calico-csr.json | cfssljson -bare flanneld
#我們最終要的是flanneld-key.pem和flanneld.pem
$ ls
flanneld.csr  flanneld-csr.json  flanneld-key.pem  flanneld.pem

  2. 安裝之前首先在Master節點的etcd上存儲網關值

    172.66.0.0 爲docker網段(可以自定義)

    ETCDCTL_API=2 etcdctl \
--endpoints=https://192.168.0.111:2379  \
--ca-file=/etc/kubernetes/ca/ca.pem \
--cert-file=/etc/kubernetes/ca/etcd/etcd.pem \
--key-file=/etc/kubernetes/ca/etcd/etcd-key.pem \
mk /atomic.io/network/config '{"Network": "172.66.0.0/16"}'

  3. 使用yum安裝flanneld

    yum install -y flanneld

  4. service配置文件/usr/lib/systemd/system/flanneld.service

    [Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
WantedBy=docker.service

  5. /etc/sysconfig/flanneld配置文件

    解釋：
1. --ifce=eth1 這裏是網卡，一般爲當前所在ip的網卡
2. FLANNEL_ETCD_PREFIX 這是etcd存儲的相關網絡信息的key
3. FLANNEL_ETCD_ENDPOINTS  這是etcd服務的地址

    # etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="https://192.168.0.111:2379"
# # etcd config key.  This is the configuration key that flannel queries
# # For address range assignment
FLANNEL_ETCD_PREFIX="/atomic.io/network"
# # Any additional options that you want to pass
FLANNEL_OPTIONS="--iface=eth1 --etcd-prefix=/atomic.io/network --etcd-endpoints=https://192.168.0.111:2379"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/etc/kubernetes/ca/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/etc/kubernetes/ca/calico/calico.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/etc/kubernetes/ca/calico/calico-key.pem"

  6. 配置docker文件，將flanneld綁定到docker上面

    最重要的就是 --bip=${FLANNEL_SUBNET}
原理就是通過綁定這個網段參數，讓docker啓動的時候按照這個參數去啓動

    Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
#import flannel configuration
EnvironmentFile=-/etc/sysconfig/flanneld
EnvironmentFile=-/run/flannel/subnet.env
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock --bip=${FLANNEL_SUBNET}
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

~

  7. 啓動flanneld

    systemctl daemon-reload
systemctl enable flanneld.service
systemctl start flanneld.service
systemctl status  flanneld.service

  8. 重啓docker

    systemctl daemon-reload
systemctl restart flanneld.service
systemctl status  flanneld.service

13.安裝kube-dns

  1. 創建kube-dns.yml文件

     vim kube-dns.yaml

  2. 修改部分配置
    clusterIP ：與上面的kubelet 的cluster-dns遙遙呼應(也就是說這裏的clusterIP就是上邊kubelet的dns的ip)

```bash
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile

---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 192.168.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: registry.cn-hangzhou.aliyuncs.com/imooc/k8s-dns-kube-dns-amd64:1.14.5
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: registry.cn-hangzhou.aliyuncs.com/imooc/k8s-dns-dnsmasq-nanny-amd64:1.14.5
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --log-facility=-
        - --server=/cluster.local./127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: registry.cn-hangzhou.aliyuncs.com/imooc/k8s-dns-sidecar-amd64:1.14.5
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      serviceAccountName: kube-dns

```
  1. 創建相應服務資源
```bash
kubectl create -f kube-dns.yml
```

13.安裝ingress

  1. 四層負載與七層負載

    四層負載：根據 ip:port 進行負載
    七層負載：基於四層代理之上，可以根據請求的內容、例如url、head等信息進行負載

    在這裏插入圖片描述

  2. 安裝相應資源服務

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

  3. 暴露相應NodePort端口

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

  4. 創建實例測試ingress.yml

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-test
spec:
  rules:
    - host: www1.liliguang.top
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-pod
            servicePort: 80

  5. 主機添加host 訪問測試

14.可能遇到的坑

  1. 外部無法訪問(NodePort只能一個節點訪問，其他節點不行)

    #核中的forward功能開啓(立即生效，重啓後效果不再)
echo "1" > /proc/sys/net/ipv4/ip_forward
#打開iptables的外網訪問
iptables -P FORWARD ACCEPT

  2. 配置服務文件時日誌報Unknown lvalue或 Missing ‘=’ 錯誤

    問題的原因是因爲配置文件有可能是從window複製而來，可能會有一些隱藏格式讓linux無法識別，可以使用vi命令重新創建一個文件，將配置信息複製到這個新創建文件即可解決

    [vagrant@Master system]$ sudo systemctl daemon-reload
[vagrant@Master system]$ sudo systemctl start kube-apiserver
Failed to start kube-apiserver.service: Unit is not loaded properly: Bad message.
See system logs and 'systemctl status kube-apiserver.service' for details.
[vagrant@Master system]$ sudo systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: error (Reason: Bad message)
   Active: inactive (dead) since Thu 2019-11-21 08:02:21 UTC; 8min ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 2208 (code=exited, status=0/SUCCESS)

Nov 21 08:08:50 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:6] Trailing garbage, ignoring.
Nov 21 08:08:50 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:7] Missing '='.
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:6] Trailing garbage, ignoring.
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:7] Unknown lvalue '--admission-control' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:8] Unknown lvalue '--insecure-bind-address' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:9] Unknown lvalue '--kubelet-https' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:10] Unknown lvalue '--bind-address' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:11] Unknown lvalue '--authorization-mode' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:12] Unknown lvalue '--runtime-config' in section 'Service'
Nov 21 08:10:27 Master systemd[1]: [/etc/systemd/system/kube-apiserver.service:13] Missing '='.

  3. etcdctl異常x509: certificate signed by unknown authority

    $ etcdctl mk /atomic.io/network/config '{"Network": "172.66.0.0/16"}'
Error:  x509: certificate signed by unknown authority

    etcdctl工具是一個可以對etcd數據進行管理的命令行工具，這個工具在兩個不同的etcd版本下的行爲方式也完全不同。
    export ETCDCTL_API=2
    export ETCDCTL_API=3

    # 使用 ETCDCTL_API=2  
ETCDCTL_API=2 etcdctl \
--endpoints=https://192.168.0.111:2379  \
--ca-file=/etc/kubernetes/ca/ca.pem \
--cert-file=/etc/kubernetes/ca/etcd/etcd.pem \
--key-file=/etc/kubernetes/ca/etcd/etcd-key.pem \
mk /atomic.io/network/config '{"Network": "172.66.0.0/16"}'

#使用  ETCDCTL_API=3
etcdctl set /atomic.io/network/config '{"Network": "172.66.0.0/16"}'

  4. 整個k8s集羣崩潰

    往往是由於etcd的數據庫崩潰爲主要原因，這時候只需要刪除數據庫，重新導入數據即可，之後會發現etcd啓動正常，其他服務陸續啓動正常，未啓動的可以手動啓動

     sudo rm -rf /var/lib/etcd/*
 systemctl daemon-reload && systemctl restart etcd

  5. service account 無法訪問apiserver
    此問題可以參考https://blog.csdn.net/kozazyh/article/details/88541533
    此問題是由於我們的default route 並不是我們當前主機的ip 造成了 如圖所示
    Endpoints 的 ip 並不是我們集羣的master的ip
    在這裏插入圖片描述

    解決方案：

    通過在kube-apiserver配置文件上配置 --advertise-address = 192.168.0.111

15.參考資料

慕課網 ：https://coding.imooc.com/class/198.html
知乎： https://zhuanlan.zhihu.com/p/64777456

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

電腦宕機後恢復K8s Pod啓動

1. 先開機看看集羣狀態是不是正常的 kubectl get cs 2. 看看pod狀態： kubectl -n xxx get pod 3. 進到nfs目錄內modeling mariab的文件夾 /dockerdata-nfs 4

liuskyter 2020-07-08 11:16:26

k8s(二)

佔位

6moji6 2020-07-08 10:26:30

利用 kubeadm 簡單搭建k8s（已更新爲V1.13.0版本）

1. 基本系統環境 1.1 系統內核 查看當前系統內核（我這裏是5.0.5-1.el7.elrepo.x86_64）： uname -a 版本必須大於等於3.10，否則需要升級內核： # ELRepo 倉庫(可以先看一下 /et

jacksonary 2020-07-08 10:04:33

3.2 控制器——副本控制器（ReplicationController）

3.2 副本控制器（ReplicationController）  ReplicationController（在kubectl命令中經常縮寫爲rc或rcs）是實際確保特定數量的Pod副本在任意時刻的運行。如果Pod副本超過指定數

jacksonary 2020-07-08 10:04:33

3.5 控制器——DaemonSet（守護線程集）

3.5 控制器——DaemonSet（守護線程集）  每個DaemonSet可以確保某些甚至全部節點運行一個Pod的副本，當node加入集羣時，Pod就會加入這些節點，同樣的，當節點從集羣中移除時，這些pods被垃圾回收。刪除一個

jacksonary 2020-07-08 10:04:33

3.3 控制器——Deployments（部署）

3.3 Deployments（部署）  Deployments控制器（Deployment controller，Deployment應該也是控制器的一種吧）提供了Pod和ReplicaSets的聲明式更新。在Deploymen

jacksonary 2020-07-08 10:04:33

3.1 控制器——ReplicaSet

3. 控制器（Controller） 3.1 副本集（ReplicaSet） 定義：副本集（ReplicaSet）的目的是爲了保證一組穩定的Pod副本在任意給定時刻都在運行。因此，它通常用於保證特定數量的相同Pod的可用性。  副

jacksonary 2020-07-08 10:04:33

centos7搭建kubernetes v1.17.1集羣

機器 hostname 10.211.55.64 k8sMaster 10.211.55.65 k8sNode1 10.211.55.66 k8sNode2 所有機器執行以下操作 (1-13) 配置y

夜幕.思年华 2020-07-08 06:16:44

雲原生Tekton之觸發器Trigger

背景 前面的文章講了tekton中pipeline的教程和使用案例，大家有沒有想過，每次都要運行taskrun或者pipelineRun才能真正運行流水線。那怎麼做到自動化執行taskrun和pipelineRun呢？我想了下有

云原生手记 2020-07-08 03:27:17

kubernetes -- 結點調度控制的幾種方式

簡介 調度器通過 kubernetes 的 watch 機制來發現集羣中新創建且尚未被調度到 Node上的 Pod。調度器會將發現的每一個未調度的 Pod 調度到一個合適的 Node 上來運行。 kube-scheduler 是

生命热力٩( 'ω' )و 2020-07-08 02:17:36

k8s創建資源的兩種方式、訪問pod

創建資源 1.用kubectl命令直接創建，   #kubectl run httpd-app --image=reg.yunwei.edu/learn/httpd:latest --replicas=2 在命令行中通過參數指定資源的

PpikachuP 2020-07-08 01:52:27

O-RAN notes(3)---Bronze SMO deployment (2)

(continued) Most of the problems are related to 'docker pull'. Before you execute the install script(./dep/smo/bin/inst

zhenggao2 2020-07-07 23:48:07

centos安裝k8s集羣

一、集羣方式 機器配置：centos 4.4內核以上，cpu大於1核 1.主機配置 配置 規格 內存配置 2G CPU配置 2核 系統版本 Centos7.7 kubelet版本 1.5.1 do

意林飞笑 2020-07-07 18:12:18

kubernetes存儲 -- Volumes管理（三）StatefulSet控制器、StatefulSet部署mysql主從集羣

StatefulSet控制器 StatefulSet可以通過Headless Service維持Pod的拓撲狀態. StatefulSet將應用狀態抽象成了兩種情況: 拓撲狀態:應用實例必須按照某種順序啓動。新創建的Pod必須

生命热力٩( 'ω' )و 2020-07-07 02:59:20

使用kubeadm安裝kubenetes1.17.3

我們使用centos7的系統，內核升級到5.5.4，爲什麼使用升級後的內核，這是因爲centos7.4的內核是3.10，在docker運行時，有內核bug，導致運行緩慢，出現一堆錯誤異常，可以查閱我另外一個博客文章。 一、安裝前的準備工作

Blue_Tear 2020-07-07 01:26:46