k8s1.12到1.17升級變更

1.17.2版本的部署和問題

部署環境：

• centos 7.5minimal
  • 192.168.2.137 master 密碼: xxxx
    192.168.2.138 node 密碼: xxxx
• k8s 1.17.2
• calico 3.7.5
• 集成了boc2.3 addon

部署過程的問題

1. k8s版本 < 1.13 是需要開啓–allow-privileged=true 設置爲true時，kubernetes允許在Pod中運行擁有系統特權的容器應用；當k8s版本 > 1.13 需要去掉這個參數，否則kubelet無法啓動。

[root@node-137 addon]# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
	$KUBELET_ADDRESS \
	$KUBELET_POD_INFRA_CONTAINER \
	$KUBELET_ARGS \
	$KUBE_LOGTOSTDERR \
	$KUBELET_NETWORK_ARGS \
	$KUBELET_DNS_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target
[root@node-137 addon]# 

2. k8s 摒棄了apiVersion：extensions/v1beta1。相關模塊（calico，efk，prometheus，beyondlet，beyondac，ingress）修改apiVersion如下

apiVerion: apps/v1

3. deployment和daemonset的spec.selector標籤缺失，之前1.12版本很多yaml文件沒有加該配置，新版會報錯。

error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
## 解決方法 在spec加添加selector
spec:
  selector:
    matchLabels:
      app: xxxx

4. calico BGP not established 估計是沒用發現實際的網卡

Warning Unhealthy 0s kubelet, 192.168.2.138 Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.2.1372020-01-21 06:20:51.501 INFO readiness.go 88: Number of node(s) with BGP peering established = 0

/*
調整calicao 網絡插件的網卡發現機制，修改IP_AUTODETECTION_METHOD對應的value值。官方提供的yaml文件中，ip識別策略（IPDETECTMETHOD）沒有配置，即默認爲first-found，這會導致一個網絡異常的ip作爲nodeIP被註冊，從而影響node-to-node mesh。我們可以修改成can-reach或者interface的策略，嘗試連接某一個Ready的node的IP，以此選擇出正確的IP。
*/

// calico.yaml 文件添加以下二行
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"  # ens 根據實際網卡開頭配置
 
 // 配置如下             
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"
              #或者 value: "interface=ens160"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always" 

5. helm報錯。

對於 Kubernetes v1.16.0 以上的版本，有可能會碰到 Error: error installing: the server could not find the requested resource 的錯誤。這是由於 extensions/v1beta1 已經被 apps/v1 替代。初始化命令如下修改

/usr/local/bin/helm init --service-account tiller --tiller-image deploy.bocloud/kubernetes-helm/tiller:v2.9.0 --skip-refresh --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -

6. etcd證書需要開啓雙向認證配置修改如下，server增加client auth,client增加server auth：

cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "876000h"
        },
        "profiles": {
            "server": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

7. etcd配置增加Type=notify

[root@node-216 k8s]# vi /etc/systemd/system/etcd.service 
[Unit]
Description=etcd server
After=network.target

[Service]
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd $ETCD_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

8. 修改cpu或者內存配置後：

rm -rf /var/lib/kubelet/cpu_manager_state && systemctl restart kubelet

9. ImageInspectError

請刪除節點上所有使用損壞 image 的容器，然後刪除 image，再重新 pull image

 Warning  InspectFailed  4m25s (x6 over 4m37s)  kubelet, node-217.dev  Failed to inspect image "deploy.bocloud/coreos/prometheus-config-reloader:v0.30.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
  Warning  Failed         4m25s (x6 over 4m37s)  kubelet, node-217.dev  Error: ImageInspectError

10. ingress-controller報錯(刪掉pod和鏡像後apply重啓)

didn't have free ports for the requested pod ports...

2.3升級的組件

組件	BOC-2.2	BOC-2.3
Kubernetes	1.12.6	1.17.2
Docker	18.09.2	19.03.5
CoreDNS	kubedns	1.6.6
Prometheus	2.4.3	2.11.0
Node-Exporter	0.16.0	0.18.1
OpenVswitch	2.10.1	2.11.2
beyondac	1.0.5	1.0.7
beyondlet	0.3.6	0.3.7