一、概述
kubernetes 1.13 已發佈，這是 2018 年年內第四次也是最後一次發佈新版本。Kubernetes 1.13 是迄今爲止發佈間隔最短的版本之一（與上一版本間隔十週），主要關注 Kubernetes 的穩定性與可擴展性，其中存儲與集羣生命週期相關的三項主要功能已逐步實現普遍可用性。
Kubernetes 1.13 的核心特性包括：利用 kubeadm 簡化集羣管理、容器存儲接口（CSI ）以及將 CoreDNS 作爲默認 DNS 。
利用 kubeadm 簡化集羣管理功能
大多數與 Kubernetes 接觸頻繁的人或多或少都會親自動手使用 kubeadm ，它是管理集羣生命週期的重要工具，能夠幫助從創建到配置再到升級的整個流程。;隨着 1.13 版本的發佈，kubeadm 功能進入 GA 版本，正式普遍可用。kubeadm 處理現有硬件上的生產集羣的引導，並以最佳實踐方式配置核心 Kubernetes 組件，以便爲新節點提供安全而簡單的連接流程並支持輕鬆升級。
該 GA 版本中最值得注意的是已經畢業的高級功能，尤其是可插拔性和可配置性。kubeadm 旨在爲管理員與高級自動化系統提供一套工具箱，如今已邁出重要一步。
容器存儲接口（CSI）
容器存儲接口最初於 1.9 版本中作爲 alpha 測試功能引入，在 1.10 版本中進入 beta 測試，如今終於進入 GA 階段正式普遍可用。在 CSI 的幫助下，Kubernetes 卷層將真正實現可擴展性。通過 CSI ，第三方存儲供應商將可以直接編寫可與 Kubernetes 互操作的代碼，而無需觸及任何 Kubernetes 核心代碼。事實上，相關規範也已經同步進入 1.0 階段。
隨着 CSI 的穩定，插件作者將能夠按照自己的節奏開發核心存儲插件，詳見 CSI 文檔。
CoreDNS 成爲 Kubernetes 的默認 DNS 服務器
在 1.11 版本中，開發團隊宣佈 CoreDNS 已實現基於 DNS 服務發現的普遍可用性。在最新的 1.13 版本中，CoreDNS 正式取代 kuber-dns 成爲 Kubernetes 中的默認 DNS 服務器。CoreDNS 是一種通用的、權威的 DNS 服務器，能夠提供與 Kubernetes 向下兼容且具備可擴展性的集成能力。由於 CoreDNS 自身單一可執行文件與單一進程的特性，因此 CoreDNS 的活動部件數量會少於之前的 DNS 服務器，且能夠通過創建自定義 DNS 條目來支持各類靈活的用例。此外，由於 CoreDNS 採用 Go 語言編寫，它具有強大的內存安全性。
CoreDNS 現在是 Kubernetes 1.13 及後續版本推薦的 DNS 解決方案，Kubernetes 已將常用測試基礎設施架構切換爲默認使用 CoreDNS ，因此，開發團隊建議用戶也儘快完成切換。KubeDNS 仍將至少支持一個版本，但現在是時候開始規劃遷移了。另外，包括 1.11 中 Kubeadm 在內的許多 OSS 安裝工具也已經進行了切換。
1、安裝環境準備：
部署節點說明
IP地址 主機名 CPU 內存 磁盤
172.16.8.100 qas-k8s-master01 4C 4G 50G
172.16.8.101 qas-k8s-node01 4C 4G 50G
172.16.8.102 qas-k8s-node02 4C 4G 50G
k8s安裝包下載
鏈接：https://pan.baidu.com/s/1wO6T7byhaJYBuu2JlhZvkQ
提取碼：pm9u
部署網絡說明
2、架構圖
Kubernetes 架構圖
Flannel網絡架構圖
數據從源容器中發出後，經由所在主機的docker0虛擬網卡轉發到flannel0虛擬網卡，這是個P2P的虛擬網卡，flanneld服務監聽在網卡的另外一端。
Flannel通過Etcd服務維護了一張節點間的路由表，在稍後的配置部分我們會介紹其中的內容。
源主機的flanneld服務將原本的數據內容UDP封裝後根據自己的路由表投遞給目的節點的flanneld服務，數據到達以後被解包，然後直接進入目的節點的flannel0虛擬網卡，
然後被轉發到目的主機的docker0虛擬網卡，最後就像本機容器通信一下的有docker0路由到達目標容器。
3、 Kubernetes工作流程
集羣功能各模塊功能描述：
Master節點：
Master節點上面主要由四個模塊組成，APIServer，schedule,controller-manager,etcd
APIServer: APIServer負責對外提供RESTful的kubernetes API的服務，它是系統管理指令的統一接口，任何對資源的增刪該查都要交給APIServer處理後再交給etcd，如圖，kubectl(kubernetes提供的客戶端工具，該工具內部是對kubernetes API的調用）是直接和APIServer交互的。
schedule: schedule負責調度Pod到合適的Node上，如果把scheduler看成一個黑匣子，那麼它的輸入是pod和由多個Node組成的列表，輸出是Pod和一個Node的綁定。 kubernetes目前提供了調度算法，同樣也保留了接口。用戶根據自己的需求定義自己的調度算法。
controller manager: 如果APIServer做的是前臺的工作的話，那麼controller manager就是負責後臺的。每一個資源都對應一個控制器。而control manager就是負責管理這些控制器的，比如我們通過APIServer創建了一個Pod，當這個Pod創建成功後，APIServer的任務就算完成了。
etcd：etcd是一個高可用的鍵值存儲系統，kubernetes使用它來存儲各個資源的狀態，從而實現了Restful的API。
Node節點：
每個Node節點主要由三個模板組成：kublet, kube-proxy
kube-proxy: 該模塊實現了kubernetes中的服務發現和反向代理功能。kube-proxy支持TCP和UDP連接轉發，默認基Round Robin算法將客戶端流量轉發到與service對應的一組後端pod。服務發現方面，kube-proxy使用etcd的watch機制監控集羣中service和endpoint對象數據的動態變化，並且維護一個service到endpoint的映射關係，從而保證了後端pod的IP變化不會對訪問者造成影響，另外，kube-proxy還支持session affinity。
kublet：kublet是Master在每個Node節點上面的agent，是Node節點上面最重要的模塊，它負責維護和管理該Node上的所有容器，但是如果容器不是通過kubernetes創建的，它並不會管理。本質上，它負責使Pod的運行狀態與期望的狀態一致。
二、Kubernetes 安裝及配置
1、初始化環境
1.1、設置關閉防火牆及SELINUX
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
1.2、關閉Swap
swapoff -a && sysctl -w vm.swappiness=0
vi /etc/fstab
#UUID=7bff6243-324c-4587-b550-55dc34018ebf swap swap defaults 0 0
1.3、設置Docker所需參數
cat << EOF | tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
1.4、安裝 Docker
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install docker-ce -y
systemctl start docker && systemctl enable docker
1.5、創建安裝目錄
mkdir /k8s/etcd/{bin,cfg,ssl} -p
mkdir /k8s/kubernetes/{bin,cfg,ssl} -p
1.6、安裝及配置CFSSL
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
1.7、創建認證證書
創建 ETCD 證書
cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
創建 ETCD CA 配置文件
cat << EOF | tee ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Shenzhen"
}
]
}
EOF
創建 ETCD Server 證書
cat << EOF | tee server-csr.json
{
"CN": "etcd",
"hosts": [
"172.16.8.100",
"172.16.8.101",
"172.16.8.102"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Shenzhen"
}
]
}
EOF
生成 ETCD CA 證書和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
創建 Kubernetes CA 證書
cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat << EOF | tee ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Shenzhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
生成API_SERVER證書
cat << EOF | tee server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"172.16.8.100",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Shenzhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
創建 Kubernetes Proxy 證書
cat << EOF | tee kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Shenzhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
1.8、 ssh-key認證
ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:FQjjiRDp8IKGT+UDM+GbQLBzF3DqDJ+pKnMIcHGyO/o root@qas-k8s-master01
The key's randomart image is:
+---[RSA 2048]----+
|o.==o o. .. |
|ooB+o+ o. . |
|B++@o o . |
|=X*o . |
|o=O. . S |
|..+ |
|oo . |
| . |
|o+E |
+----[SHA256]-----+
ssh-copy-id 172.16.8.101
ssh-copy-id 172.16.8.102
2 、部署ETCD
解壓安裝文件
tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/
cp etcd etcdctl /k8s/etcd/bin/
vim /k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.8.100:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.8.100:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.8.100:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.8.100:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.8.100:2380,etcd02=https://172.16.8.101:2380,etcd03=https://172.16.8.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
拷貝證書文件
cp capem serverpem /k8s/etcd/ssl
啓動ETCD服務
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
將啓動文件、配置文件拷貝到 節點1、節點2
cd /k8s/
scp -r etcd 172.16.8.101:/k8s/
scp -r etcd 172.16.8.102:/k8s/
scp /usr/lib/systemd/system/etcd.service 172.16.8.101:/usr/lib/systemd/system/etcd.service
scp /usr/lib/systemd/system/etcd.service 172.16.8.102:/usr/lib/systemd/system/etcd.service
vim /k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.8.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.8.101:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.8.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.8.101:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.8.100:2380,etcd02=https://172.16.8.101:2380,etcd03=https://172.16.8.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
vim /k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.8.102:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.8.102:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.8.102:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.8.102:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.8.100:2380,etcd02=https://172.16.8.101:2380,etcd03=https://172.16.8.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
驗證集羣是否正常運行
./etcdctl \
--ca-file=/k8s/etcd/ssl/ca.pem \
--cert-file=/k8s/etcd/ssl/server.pem \
--key-file=/k8s/etcd/ssl/server-key.pem \
--endpoints="https://172.16.8.100:2379,\
https://172.16.8.101:2379,\
https://172.16.8.102:2379" cluster-health
member 5db3ea816863435 is healthy: got healthy result from https://172.16.8.102:2379
member 991b5845cecb31b is healthy: got healthy result from https://172.16.8.101:2379
member c67ee2780d64a0d4 is healthy: got healthy result from https://172.16.8.100:2379
cluster is healthy
注意：
啓動ETCD集羣同時啓動二個節點，啓動一個節點集羣是無法正常啓動的；
3、部署Flannel網絡
向 etcd 寫入集羣 Pod 網段信息
cd /k8s/etcd/ssl/
/k8s/etcd/bin/etcdctl \
--ca-file=ca.pem --cert-file=server.pem \
--key-file=server-key.pem \
--endpoints="https://172.16.8.100:2379,\
https://172.16.8.101:2379,https://172.16.8.102:2379" \
set /coreos.com/network/config '{ "Network": "172.18.0.0/16", "Backend": {"Type": "vxlan"}}'
flanneld 當前版本 (v0.10.0) 不支持 etcd v3，故使用 etcd v2 API 寫入配置 key 和網段數據；
寫入的 Pod 網段 ${CLUSTER_CIDR} 必須是 /16 段地址，必須與 kube-controller-manager 的 --cluster-cidr 參數值一致；
解壓安裝
tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/
配置Flannel
vim /k8s/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://172.16.8.100:2379,https://172.16.8.101:2379,https://172.16.8.102:2379 -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem"
創建 flanneld 的 systemd unit 文件
vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/k8s/kubernetes/cfg/flanneld
ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網網段信息寫入 /run/flannel/docker 文件，後續 docker 啓動時 使用這個文件中的環境變量配置 docker0 網橋；
flanneld 使用系統缺省路由所在的接口與其它節點通信，對於有多個網絡接口（如內網和公網）的節點，可以用 -iface 參數指定通信接口，如上面的 eth0 接口;
flanneld 運行時需要 root 權限；
配置Docker啓動指定子網段
vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
配置Docker啓動指定子網段
vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
將flanneld systemd unit 文件到所有節點
cd /k8s/
scp -r kubernetes 172.16.8.101:/k8s/
scp -r kubernetes 172.16.8.102:/k8s/
scp /k8s/kubernetes/cfg/flanneld 172.16.8.102:/k8s/kubernetes/cfg/flanneld
scp /k8s/kubernetes/cfg/flanneld 172.16.8.102:/k8s/kubernetes/cfg/flanneld
scp /usr/lib/systemd/system/docker.service 172.16.8.101:/usr/lib/systemd/system/docker.service
scp /usr/lib/systemd/system/docker.service 172.16.8.102:/usr/lib/systemd/system/docker.service
scp /usr/lib/systemd/system/flanneld.service 172.16.8.101:/usr/lib/systemd/system/flanneld.service
scp /usr/lib/systemd/system/flanneld.service 172.16.8.102:/usr/lib/systemd/system/flanneld.service
啓動服務
systemctl daemon-reload
systemctl start flanneld
systemctl enable flanneld
systemctl restart docker
查看是否生效
ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:e3:57:a4 brd ff:ff:ff:ff:ff:ff
inet 172.16.8.101/24 brd 172.16.8.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fee3:57a4/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:cf:5d:a7:af brd ff:ff:ff:ff:ff:ff
inet 172.18.25.1/24 brd 172.18.25.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 0e:bf:c5:3b:4d:59 brd ff:ff:ff:ff:ff:ff
inet 172.18.25.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::cbf:c5ff:fe3b:4d59/64 scope link
valid_lft forever preferred_lft forever
4、部署 master 節點
kubernetes master 節點運行如下組件：
kube-apiserver
kube-scheduler
kube-controller-manager
kube-scheduler 和 kube-controller-manager 可以以集羣模式運行，通過 leader 選舉產生一個工作進程，其它進程處於阻塞模式。
將二進制文件解壓拷貝到master 節點
tar -xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/
拷貝認證
cp *pem /k8s/kubernetes/ssl/
部署 kube-apiserver 組件
創建 TLS Bootstrapping Token
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
2366a641f656a0a025abb4aabda4511b
vim /k8s/kubernetes/cfg/token.csv
2366a641f656a0a025abb4aabda4511b,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
創建apiserver配置文件
vim /k8s/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://172.16.8.100:2379,https://172.16.8.101:2379,https://172.16.8.102:2379 \
--bind-address=172.16.8.100 \
--secure-port=6443 \
--advertise-address=172.16.8.100 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem"
創建 kube-apiserver systemd unit 文件
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啓動服務
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver
查看apiserver是否運行
ps -ef |grep kube-apiserver
root 76300 1 45 08:57 ? 00:00:14 /k8s/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://172.16.8.100:2379,https://172.16.8.101:2379,https://172.16.8.102:2379 --bind-address=172.16.8.100 --secure-port=6443 --advertise-address=172.16.9.51 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/k8s/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/k8s/kubernetes/ssl/server.pem --tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem --client-ca-file=/k8s/kubernetes/ssl/ca.pem --service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem --etcd-cafile=/k8s/etcd/ssl/ca.pem --etcd-certfile=/k8s/etcd/ssl/server.pem --etcd-keyfile=/k8s/etcd/ssl/server-key.pem
root 76357 4370 0 08:58 pts/1 00:00:00 grep --color=auto kube-apiserver
部署kube-scheduler
創建kube-scheduler配置文件
vim /k8s/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"
--address：在 127.0.0.1:10251 端口接收 http /metrics 請求；kube-scheduler 目前還不支持接收 https 請求；
--kubeconfig：指定 kubeconfig 文件路徑，kube-scheduler 使用它連接和驗證 kube-apiserver；
--leader-elect=true：集羣運行模式，啓用選舉功能；被選爲 leader 的節點負責處理工作，其它節點爲阻塞狀態；
創建kube-scheduler systemd unit 文件
vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啓動服務
systemctl daemon-reload
systemctl enable kube-scheduler.service
systemctl restart kube-scheduler.service
查看kube-scheduler是否運行
ps -ef |grep kube-scheduler
root 77854 1 8 09:17 ? 00:00:02 /k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
root 77901 1305 0 09:18 pts/0 00:00:00 grep --color=auto kube-scheduler
systemctl status kube-scheduler.service
● kube-scheduler.service - Kubernetes Scheduler
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; disabled; vendor preset: disabled)
Active: active (running) since 三 2018-12-05 09:17:43 CST; 29s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 77854 (kube-scheduler)
Tasks: 13
Memory: 10.9M
CGroup: /system.slice/kube-scheduler.service
└─77854 /k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
12月 05 09:17:45 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:45.642632 77854 shared_informer.go:123] caches populated
12月 05 09:17:45 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:45.743297 77854 shared_informer.go:123] caches populated
12月 05 09:17:45 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:45.844554 77854 shared_informer.go:123] caches populated
12月 05 09:17:45 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:45.945332 77854 shared_informer.go:123] caches populated
12月 05 09:17:45 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:45.945434 77854 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
12月 05 09:17:46 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:46.046385 77854 shared_informer.go:123] caches populated
12月 05 09:17:46 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:46.046427 77854 controller_utils.go:1034] Caches are synced for scheduler controller
12月 05 09:17:46 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:46.046574 77854 leaderelection.go:205] attempting to acquire leader lease kube-system/kube-scheduler...
12月 05 09:17:46 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:46.063185 77854 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
12月 05 09:17:46 qas-k8s-master01 kube-scheduler[77854]: I1205 09:17:46.164498 77854 shared_informer.go:123] caches populated
部署kube-controller-manager
創建kube-controller-manager配置文件
vim /k8s/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--root-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem"
創建kube-controller-manager systemd unit 文件
vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啓動服務
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
`
查看kube-controller-manager是否運行
systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since 三 2018-12-05 09:35:00 CST; 3s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 79191 (kube-controller)
Tasks: 8
Memory: 15.2M
CGroup: /system.slice/kube-controller-manager.service
└─79191 /k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0....
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.350599 79191 serving.go:318] Generated self-signed cert in-memory
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: W1205 09:35:01.762710 79191 authentication.go:235] No authentication-kubeconfig provided in order to lookup...on't work.
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: W1205 09:35:01.762767 79191 authentication.go:238] No authentication-kubeconfig provided in order to lookup...on't work.
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: W1205 09:35:01.762792 79191 authorization.go:146] No authorization-kubeconfig provided, so SubjectAcce***ev...on't work.
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.762827 79191 controllermanager.go:151] Version: v1.13.0
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.763446 79191 secure_serving.go:116] Serving securely on [::]:10257
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.763925 79191 deprecated_insecure_serving.go:51] Serving insecurely on 127.0.0.1:10252
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.764443 79191 leaderelection.go:205] attempting to acquire leader lease kube-system/kube-con...manager...
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.770798 79191 leaderelection.go:289] lock is held by qas-k8s-master01_fab3fbe9-f82d-11e8-9140...et expired
12月 05 09:35:01 qas-k8s-master01 kube-controller-manager[79191]: I1205 09:35:01.770817 79191 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager
Hint: Some lines were ellipsized, use -l to show in full.
ps -ef |grep kube-controller-manager
root 79191 1 10 09:35 ? 00:00:01 /k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem --cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem --root-ca-file=/k8s/kubernetes/ssl/ca.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem
root 79220 1305 0 09:35 pts/0 00:00:00 grep --color=auto kube-controller-manager
將可執行文件路/k8s/kubernetes/ 添加到 PATH 變量中
vim /etc/profile
PATH=/k8s/kubernetes/bin:$PATH:$HOME/bin
source /etc/profile
查看master集羣狀態
kubectl get cs,nodes
NAME STATUS MESSAGE ERROR
componentstatus/scheduler Healthy ok
componentstatus/etcd-2 Healthy {"health":"true"}
componentstatus/etcd-1 Healthy {"health":"true"}
componentstatus/etcd-0 Healthy {"health":"true"}
componentstatus/controller-manager Healthy ok
5、部署node 節點
kubernetes work 節點運行如下組件：
docker 前面已經部署
kubelet
kube-proxy
部署 kubelet 組件
kublet 運行在每個 worker 節點上，接收 kube-apiserver 發送的請求，管理 Pod 容器，執行交互式命令，如exec、run、logs 等;
kublet 啓動時自動向 kube-apiserver 註冊節點信息，內置的 cadvisor 統計和監控節點的資源使用情況;
爲確保安全，本文檔只開啓接收 https 請求的安全端口，對請求進行認證和授權，拒絕未授權的訪問(如apiserver、heapster)。
將kubelet 二進制文件拷貝node節點
cp kubelet kube-proxy /k8s/kubernetes/bin/
scp kubelet kube-proxy 172.16.8.101:/k8s/kubernetes/bin/
scp kubelet kube-proxy 172.16.8.102:/k8s/kubernetes/bin/
創建 kubelet bootstrap kubeconfig 文件
vim environment.sh
創建kubelet bootstrapping kubeconfig
BOOTSTRAP_TOKEN=2366a641f656a0a025abb4aabda4511b
KUBE_APISERVER="https://172.16.8.100:6443"
設置集羣參數
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
設置客戶端認證參數
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
設置上下文參數
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
設置默認上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
創建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
將bootstrap kubeconfig kube-proxy.kubeconfig 文件拷貝到所有 nodes節點
cp bootstrap.kubeconfig kube-proxy.kubeconfig /k8s/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig 172.16.8.101:/k8s/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig 172.16.8.102:/k8s/kubernetes/cfg/
創建kubelet 參數配置文件拷貝到所有 nodes節點
創建 kubelet 參數配置模板文件：
vim /k8s/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 172.16.8.100
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
創建kubelet配置文件
vim /k8s/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=172.16.8.100 \
--kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
創建kubelet systemd unit 文件
vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kubelet
ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
將kubelet-bootstrap用戶綁定到系統集羣角色
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
啓動服務
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
approve kubelet CSR 請求
可以手動或自動 approve CSR 請求。推薦使用自動的方式，因爲從 v1.8 版本開始，可以自動輪轉approve csr 後生成的證書。
手動 approve CSR 請求
查看 CSR 列表：
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 39m kubelet-bootstrap Pending
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 5m5s kubelet-bootstrap Pending
kubectl certificate approve node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
certificatesigningrequest.certificates.k8s.io/node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
kubectl certificate approve node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s
certificatesigningrequest.certificates.k8s.io/node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s approved
[
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 41m kubelet-bootstrap Approved,Issued
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 7m32s kubelet-bootstrap Approved,Issued
Requesting User：請求 CSR 的用戶，kube-apiserver 對它進行認證和授權；
Subject：請求籤名的證書信息；
證書的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kube-apiserver 的 Node 授權模式會授予該證書的相關權限；
查看集羣狀態
kubectl get nodes
NAME STATUS ROLES AGE VERSION
172.16.8.100 Ready <none> 39m v1.13.0
172.16.8.101 Ready <none> 25s v1.13.0
172.16.8.102 Ready <none> 13s v1.13.0
部署 kube-proxy 組件
kube-proxy 運行在所有 node節點上，它監聽 apiserver 中 service 和 Endpoint 的變化情況，創建路由規則來進行服務負載均衡。
創建 kube-proxy 配置文件
vim /k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=172.16.8.100 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"
bindAddress: 監聽地址；
clientConnection.kubeconfig: 連接 apiserver 的 kubeconfig 文件；
clusterCIDR: kube-proxy 根據 --cluster-cidr 判斷集羣內部和外部流量，指定 --cluster-cidr 或 --masquerade-all 選項後 kube-proxy 纔會對訪問 Service IP 的請求做 SNAT；
hostnameOverride: 參數值必須與 kubelet 的值一致，否則 kube-proxy 啓動後會找不到該 Node，從而不會創建任何 ipvs 規則；
mode: 使用 ipvs 模式；
創建kube-proxy systemd unit 文件
vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy
ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啓動服務
systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
systemctl status kube-proxy
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 三 2018-12-05 22:49:31 CST; 7s ago
Main PID: 13848 (kube-proxy)
Tasks: 0
Memory: 11.1M
CGroup: /system.slice/kube-proxy.service
‣ 13848 /k8s/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=172.16.8.100 --cluster-cidr=10.0.0.0/24 --kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubecon...
12月 05 22:49:31 qas-k8s-master01 kube-proxy[13848]: I1205 22:49:31.989376 13848 iptables.go:391] running i
集羣狀態
kubectl get node,cs
NAME STATUS ROLES AGE VERSION
node/172.16.8.100 Ready <none> 41m v1.13.0
node/172.16.8.101 Ready <none> 2m53s v1.13.0
node/172.16.8.102 Ready <none> 2m41s v1.13.0
NAME STATUS MESSAGE ERROR
componentstatus/scheduler Healthy ok
componentstatus/controller-manager Healthy ok
componentstatus/etcd-1 Healthy {"health":"true"}
componentstatus/etcd-0 Healthy {"health":"true"}
componentstatus/etcd-2 Healthy {"health":"true"}
作者：思考v
來源：CSDN
原文：https://blog.csdn.net/xiegh2014/article/details/84830880
版權聲明：本文爲博主原創文章，轉載請附上博文鏈接！