【OSS】——阿里雲OSS作爲k8s存儲(pv/pvc)實踐

一、前言

    關於k8s的存儲知識，大家可以參考博文應用存儲和持久化數據卷：核心知識和Kubernetes 存儲架構及插件使用，我這就不科普了。阿里雲官網上的文檔都是基於阿里雲自己的k8s的，找了半天資料也沒有文章好好介紹一下自己通過kubeadm搭建的k8s集羣該怎麼用阿里雲的oss作爲pv/pvc使用，今天這篇文章就是我踩坑之後寫下來作爲記錄的。

說明：

• OSS 數據卷是使用 OSSFS 文件進行掛載的 FUSE 文件系統，適合於讀文件場景。例如：讀配置文件、視頻、圖片文件等場景。
• OSSFS 不擅長於寫文件的應用場景。

二、準備工作

1、k8s集羣，我是通過kubeadm搭建的三節點集羣。搭建步驟參見我的另外一篇博文:搭建k8s多節點集羣

NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node1   Ready    master   51d   v1.16.2   10.53.5.94    <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4
node2   Ready    <none>   51d   v1.16.2   10.53.6.185   <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4
node3   Ready    <none>   51d   v1.16.2   10.53.7.37    <none>        Ubuntu 16.04.1 LTS   4.4.0-127-generic   docker://19.3.4

2、阿里雲oss賬號，並創建bucket。這個沒什麼好說的，參見阿里雲OSS官方文檔

3、在每臺機器上安裝阿里雲ossfs軟件。這個軟件必須要安裝，因爲pv/pvc如果想要用阿里雲的oss的話，這是必須的軟件。安裝步驟參見阿里雲oss安裝文檔，我的機器是三節點ubuntu 16.04,下邊我就寫一下ubuntu 16.04該怎麼安裝ossfs。

# 下載安裝包
$ wget http://gosspublic.alicdn.com/ossfs/ossfs_1.80.6_ubuntu16.04_amd64.deb

#安裝
$ sudo apt-get update
$ sudo apt-get install gdebi-core
#sudo gdebi <your_ossfs_package>
$ sudo gdebi ossfs_1.80.6_ubuntu16.04_amd64.deb

三、yaml文件準備

1.rbac.yaml

# This YAML file contains all RBAC objects that are necessary to run external
# CSI provisioner.
#
# In production, each CSI driver deployment has to be customized:
# - to avoid conflicts, use non-default namespace and different names
#   for non-namespaced entities like the ClusterRole
# - decide whether the deployment replicates the external CSI
#   provisioner, in which case leadership election must be enabled;
#   this influences the RBAC setup, see below

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  # replace with the same namespace name with plugin
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: alicloud-csi-plugin
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: alicloud-csi-plugin
subjects:
  - kind: ServiceAccount
    name: admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: alicloud-csi-plugin
  apiGroup: rbac.authorization.k8s.io

2.oss-plugin.yaml

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: ossplugin.csi.alibabacloud.com
spec:
  attachRequired: false
---
# This YAML defines all API objects to create RBAC roles for csi node plugin.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-ossplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-ossplugin
  template:
    metadata:
      labels:
        app: csi-ossplugin
    spec:
      tolerations:
      - operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      hostPID: true
      containers:
      - name: driver-registrar
        image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.1.0
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "rm -rf /registration/ossplugin.csi.alibabacloud.com /registration/ossplugin.csi.alibabacloud.com-reg.sock"]
        args:
        - "--v=5"
        - "--csi-address=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
        - "--kubelet-registration-path=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
        env:
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        volumeMounts:
        - name: kubelet-dir
          mountPath: /var/lib/kubelet/
        - name: registration-dir
          mountPath: /registration

      - name: csi-ossplugin
        securityContext:
          privileged: true
          capabilities:
            add: ["SYS_ADMIN"]
          allowPrivilegeEscalation: true
        image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8.32-c77e277b-aliyun
        imagePullPolicy: "Always"
        args:
        - "--endpoint=$(CSI_ENDPOINT)"
        - "--v=5"
        - "--driver=ossplugin.csi.alibabacloud.com"
        - "--nodeid=$(KUBE_NODE_NAME)"
        env:
        - name: CSI_ENDPOINT
          value: unix://var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: kubelet-dir
          mountPath: /var/lib/kubelet/
          mountPropagation: "Bidirectional"
        - name: etc
          mountPath: /host/etc
        - mountPath: /var/log/
          name: host-log
        - mountPath: /host/usr/
          name: flexvolumedir
      volumes:
      - name: kubelet-dir
        hostPath:
          path: /var/lib/kubelet/
          type: Directory
      - name: registration-dir
        hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: DirectoryOrCreate
      - name: etc
        hostPath:
          path: /etc
      - name: flexvolumedir
        hostPath:
          path: /usr/
      - name: host-log
        hostPath:
          path: /var/log/
  updateStrategy:
    type: RollingUpdate

3.pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: oss-csi-pv
  labels:
    alicloud-pvname: oss-csi-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    # set volumeHandle same value pv name
    volumeHandle: oss-csi-pv
    volumeAttributes:
      bucket: "*****" #重要
      url: "******" #重要
      otherOpts: "-o max_stat_cache_size=0 -o allow_other"
      akId: "****" #重要
      akSecret: "*******" #重要
      path: "/"

說明：

• bucket：目前只支持掛載Bucket，不支持掛載Bucket下面的子目錄或文件。
• url：OSS endpoint，掛載OSS的接入域名，掛載節點和bucket相同region時，可使用內網地址。
• akId：用戶的access id值。
• akSecret：用戶的access secret值。
• otherOpts：掛載OSS時支持定製化參數輸入，格式爲：-o *** -o ***。

 

 4.pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: oss-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      alicloud-pvname: oss-csi-pv

5. deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-oss
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: oss-pvc
            mountPath: "/data"
      volumes:
        - name: oss-pvc
          persistentVolumeClaim:
            claimName: oss-pvc

四、部署

至此，部署階段就比較簡單了。按照上邊準備的文件順序知行yaml文件：

#創建rbac權限
$ kubectl create -f ./rbac.yaml 
serviceaccount/admin created
clusterrole.rbac.authorization.k8s.io/alicloud-csi-plugin created
clusterrolebinding.rbac.authorization.k8s.io/alicloud-csi-plugin created


#創建oss-plugin
$ kubectl create -f ./oss-plugin.yaml

#檢查創建情況
$ kubectl get pod -n kube-system | grep csi-oss
kube-system             csi-ossplugin-9jdhw                                  2/2     Running             0          55m
kube-system             csi-ossplugin-f7n5f                                  2/2     Running             0          55m
kube-system             csi-ossplugin-vgkcp                                  2/2     Running             0          55m

#查驗CSIDriver安裝情況
$ kubectl get CSIDriver
NAME                             CREATED AT
ossplugin.csi.alibabacloud.com   2020-06-23T14:48:18Z

#創建pv
$ kubectl create -f ./pv.yaml

#創建pvc
$ kubectl create -f ./pvc.yaml

#檢驗一下阿里雲oss是否可以成功掛載到k8s集羣中做pv使用
$ kubectl create -f ./deploy.yaml

五、驗證

$ kubectl get pod
NAME                              READY   STATUS              RESTARTS   AGE
deployment-oss-795894886d-lhpsx   1/1     Running             0          11h

#pod成功後通過kubectl exec 進入到pod中，你能看到你賬號下bucket裏邊的所有文件。樣例如下：
$ kubectl exec -it deployment-oss-795894886d-lhpsx -- sh
$ ls
bin  boot  data  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  selinux  srv  sys	tmp  usr  var
$ cd data	
$ ls
osstest

六、參考文章

1、K8S有狀態服務-OSS存儲使用最佳實踐；

2、阿里雲oss CSI安裝步驟

3、阿里雲oss官方文檔