Kubernetes-1.16部署之三 Traefik 2.0

一、創建Traefik CRD資源

traefik v2.0 版本後，開始使用CRD（Custom Resource Definition）來完成路由配置

# mkdir -p /opt/kubernetes/traefik/yaml
# cd /opt/kubernetes/traefik/yaml

# vi crd.yaml
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

創建Rraefik CRD資源

# kubectl create -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/ingressroutes.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/ingressroutetcps.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/middlewares.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/tlsoptions.traefik.containo.us created

二、創建Traefik RBAC權限

# vi rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

創建Traefik RBAC資源

# kubectl create -f rbac.yaml
serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

三、創建Traefik配置文件

用DaemonSet方式部署，便於在多服務器間擴展

# vi traefik.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      restartPolicy: Always
      tolerations:
      - operator: "Exists"
      containers:
      - image: traefik:v2.0.7
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        ports:
        - name: web
          containerPort: 80
          hostPort: 80
        - name: websecure
          containerPort: 443
          hostPort: 443
        - name: mysql
          containerPort: 3306
          hostPort: 3306
        - name: redis
          containerPort: 6379
          hostPort: 6379
        - name: admin
          containerPort: 8080
          hostPort: 9999
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --entrypoints.web.Address=:80
        - --entrypoints.websecure.Address=:443
        - --entrypoints.mysql.Address=:3306
        - --entrypoints.redis.Address=:6379
        - --providers.kubernetescrd
        - --api
        - --api.dashboard=true
        - --api.insecure=true
        - --metrics.prometheus=true
        - --tracing.zipkin=true
        - --accesslog
        - --accesslog.filepath=/var/log/access.log
      nodeSelector:
        edgenode: "true"

---

kind: Service
apiVersion: v1
metadata:
  name: traefik
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - name: admin
      port: 9999
      protocol: TCP

創建Traefik資源

# kubectl create -f traefik.yaml
daemonset.apps/traefik created
service/traefik created

四、設置節點Label標籤

由於是使用Kubernetes DeamonSet這種方式部署traefik，所以需要提前給節點設置label，這樣當程序部署時pod會自動調度到設置label的點上

# kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE     VERSION   LABELS
192.168.168.3  Ready    <none>   4d11h   v1.16.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.168.3,kubernetes.io/os=linux
192.168.168.4  Ready    <none>   4d11h   v1.16.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.168.4,kubernetes.io/os=linux

# kubectl label nodes 192.168.168.3 edgenode=true
node/192.168.168.3 labeled

注：如想刪除label標籤使用如下命令

# kubectl label nodes 192.168.168.3 edgenode-

五、配置Traefik路由規則

想讓外部訪問Kubernetes內部服務，需要配置路由規則，這裏配置Traefik Dashboard的路由規則，使外部能夠訪問Traefik Dashboard

# vi IngressRoute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui
  namespace: kube-system
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefik.k8s.local`)
    kind: Rule
    services:
    - name: traefik
      port: 9999

注:Host(` `)中的內容也可爲自定義域名，如配置爲traefik.

創建Traefik Dashboard https協議路由規則對象

# kubectl create -f IngressRoute.yaml
ingressroute.traefik.containo.us/traefik-webui created

注：卸載Traefik時先卸載IngressRoute.yaml再卸載其它資源

在管理機hosts裏配置映射192.168.168.3 traefik.k8s.local(或搭建內網DNS服務器)，然後在瀏覽器中輸入http://traefik.k8s.local:9999

六、配置多邊緣節點高可用

1）在work-node01/02上安裝keepalived｛安裝過程此文略過｝

設置一個VIP IP指向自定義域名traefik.k8s.local，本文示例爲192.168.168.100，這樣集羣外部就可以通過service的DNS映射名稱來訪問服務。

2）設置節點Label標籤

# kubectl label nodes 192.168.168.3 edgenode=true
# kubectl label nodes 192.168.168.4 edgenode=true

3）查看DaemonSet啓動情況

# kubectl -n kube-system get ds
NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
traefik                  2         2         2       2            2           edgenode=true                 16m

4）配置keepalived

# vi $KEEPALIVED_HOME/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from k8s_admin@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 6666
    }
    virtual_ipaddress {
        192.168.168.100
    }
}

virtual_server 192.168.168.100 9999{
    delay_loop 6
    lb_algo loadbalance
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 0
    protocol TCP

    real_server 192.168.168.3 9999{
        weight 1
        TCP_CHECK {
        connect_timeout 3
        }
    }
    real_server 192.168.168.4 9999{
        weight 1
        TCP_CHECK {
        connect_timeout 3
        }
    }
}

注：real_server的IP和端口即traefik供外網訪問的IP和端口

在管理機hosts裏配置映射192.168.168.100 traefik.k8s.local，然後在在瀏覽器中輸入http://traefik.k8s.local