OpenStack虛擬機部署K8S集羣，使用Cinder提供PV報錯：mount: special device does not exist

1、出錯信息

kubelet掛載pv卷失敗，從出錯信息來看失敗原因是設備不存在

# kubectl describe pods testpvcpod-cinder
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-18372.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not exist
  Warning  FailedMount  6m6s                 kubelet, k8s-n1  Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[default-token-lrww8 testpv-cinder]: timed out waiting for the condition
  Warning  FailedMount  116s (x4 over 6m1s)  kubelet, k8s-n1  (combined from similar events): MountVolume.SetUp failed for volume "pvc-203a9760-85e9-4f58-91cd-081e9be96946" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-21619.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not exist
  Warning  FailedMount  93s  kubelet, k8s-n1  Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[testpv-cinder default-token-lrww8]: timed out waiting for the condition

2、帶Cinder PV pod部署流程

2.1、K8S集羣信息

在OpenStack集羣中部署了4個虛擬機，分別用於部署K8S Master和Node節點

# kubectl get nodes
NAME     STATUS   ROLES    AGE    VERSION
k8s-m1   Ready    master   112s   v1.17.0
k8s-m2   Ready    master   87s    v1.17.0
k8s-m3   Ready    master   21s    v1.17.0
k8s-n1   Ready    <none>   13s    v1.17.0

2.2、創建Cinder Storage Class

# cat cinder-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/cinder
reclaimPolicy: Delete
volumeBindingMode: Immediate

# kubectl apply -f cinder-sc.yaml
storageclass.storage.k8s.io/standard created

# kubectl get sc
NAME       PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard   kubernetes.io/cinder   Delete          Immediate           false                  5s

2.3、創建帶動態Cinder PV的POD

# cat busybox-cinder-dym.yaml
apiVersion: v1
kind: Pod
metadata:
  name: testpvcpod-cinder
spec:
  containers:
  - name: busybox-pvc-cinder
    image: busybox
    command:
      - sh
      - -c
      - "while true; do date > /home/loop; sleep 10m; done"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: testpv-cinder
      mountPath: /home
  volumes:
  - name: testpv-cinder
    persistentVolumeClaim:
      claimName: cinder-pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cinder-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 1Gi

# kubectl apply -f busybox-cinder-dym.yaml
pod/testpvcpod-cinder created
persistentvolumeclaim/cinder-pvc created

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
nfs-pv                                     1Gi        RWX            Recycle          Bound    default/nfs-pvc      slow                    26h
pvc-ce961d70-ea7f-4f23-9cf7-7de782951136   1Gi        RWO            Delete           Bound    default/cinder-pvc   standard                7m46s

# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cinder-pvc   Bound    pvc-ce961d70-ea7f-4f23-9cf7-7de782951136   1Gi        RWO            standard       7m49s
nfs-pvc      Bound    nfs-pv                                     1Gi        RWX            slow           26h

PV、PVC和POD都已經創建好，但是POD處於ContainerCreating狀態

# kubectl get pods
NAME                READY   STATUS              RESTARTS   AGE
testpvcpod          1/1     Running             0          25h
testpvcpod-cinder   0/1     ContainerCreating   0          7m45s

沒有running的原因就是因爲本文開頭的問題

3、解決問題

3.1、查看Cinder PV卷是否掛載到節點上

k8s-n1節點上的/dev/vdb就是這個PV

[root@k8s-n1 ~]# fdisk -l
Disk /dev/vda: 54.8 GB, 54760833024 bytes, 106954752 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00072170

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     2099199     1048576   83  Linux
/dev/vda2         2099200   104857599    51379200   8e  Linux LVM

Disk /dev/mapper/centos-root: 47.2 GB, 47240445952 bytes, 92266496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/vdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

3.2、PVC對應的目錄不存在

[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
ls: cannot access /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946: No such file or directory

3.3、手工創建目錄

[root@k8s-n1 ~]# mkdir -p /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
total 0

3.4、手工掛載卷

[root@k8s-n1 ~]# mkfs.xfs /dev/vdb
meta-data=/dev/vdb               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@k8s-n1 ~]# mount /dev/vdb /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# df
/dev/vdb                                    1038336   32992   1005344   4% /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946

3.5、POD處於running狀態

# kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
testpvcpod          1/1     Running   0          42h
testpvcpod-cinder   1/1     Running   0          9m54s

查看卷已經掛載到pod上

# kubectl exec -it testpvcpod-cinder df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay               46110724   5801528  40309196  13% /
tmpfs                    65536         0     65536   0% /dev
tmpfs                  4004356         0   4004356   0% /sys/fs/cgroup
/dev/vdb               1038336     32996   1005340   3% /home

4、問題分析

因爲本環境用的是K8S原生的Cinder驅動，沒有使用CSI，而原生的Cinder驅動在1.17.0版本及後續版本將逐漸廢棄，轉爲CSI。所以這塊代碼可能存在bug。即k8s master雖然調用OpenStack接口爲pod創建並掛載了卷，但是kubelet不知道卷信息，無法做自動mount操作。
後續有空分析下這塊代碼，並驗證下CSI Cinder的使用