在Kubernetes中使用Sateful Set部署Redis

面寫過過幾篇關於在Kubernetes上運行有狀態服務相關的博文:

• Kubernetes資源對象之Persistent Volumes
• 在Kubernetes Pod中使用Ceph RBD Volume
• 在Kubernetes上部署有狀態服務

最近需要在我們的一個Kubernetes集羣上部署Redis，因此重新整理，寫一下如何在Kubernetes上使用Sateful Set部署Redis。

1.需求和環境

我們的需求是需要部署三節點的Redis主從複製，並部署三個節點的Redis Sentinel實現Redis的高可用。

環境信息如下：

• Kubernetes 1.6.7集羣
• Ceph 11.2.0集羣

Kubernetes的官方examples中已經給出了一個在k8s集羣上部署Redis的例子Reliable, Scalable Redis on Kubernetes， 就是基於Redis主從複製+Sentinel實現的，但是這個例子是以無狀態服務形式部署的，如果整個k8s集羣重啓了，Redis的狀態就會丟失，因此不能用於生產環境。 但我們可以參考這個例子，以Satefult Set的形式部署。

我們的線上環境主要使用Ceph的塊存儲RBD作爲Kubernetes的存儲卷，這裏可以將Redis服務的狀態保存在Ceph RBD中。

關於Kubernetes和Ceph的部署可以參考我之前寫過的幾篇博文，這裏不再展開：

• 使用kubeadm安裝Kubernetes 1.6
• Kubernetes 1.6 高可用集羣部署
• 使用kubeadm安裝Kubernetes 1.7
• Ceph Kraken 11.2.0部署記錄

2.Storage Classes和Dynamic Storage Provision

Kubernetes 1.6開始Storage Classes和Dynamic Storage Provision已經是穩定可用的了。 StorageClass是Dynamic Storage Provision的基礎，k8s的管理員可以定義底層存儲平臺抽象。 用戶通過在PVC(Persistent Volume Claim)中通過名字引用StorageClass，PV(Persistent Volume)將使用StorageClass來動態創建，這樣就節省了集羣管理員手動創建PV的時間。

2.1 在Ceph中創建存儲池Pool

我們需要先在Ceph中創建一個k8s集羣專用的Ceph Pool，在創建之前我們先看一下當前Ceph集羣中的存儲池：

ceph osd lspools
0 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,

一個Ceph集羣可以有多個pool，pool是邏輯上的存儲池。不同的pool可以有不一樣的數據處理方式，例如replica size, placement groups, crush rules,snapshot等等。 可以看到因爲我們這個環境還是用Ceph的RGW作爲我們的對象存儲，因此除了默認的名稱爲rbd的pool外，還有很多rgw的pool。

下面創建一個專門給k8s集羣專用的pool kube:

ceph osd pool create kube 128
pool 'kube' created

ceph osd lspools
0 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,12 kube,

• 當前這個ceph集羣只有3個osd，所以設置pg_num爲128，可參考PLACEMENT GROUPS

2.2 配置k8s Node節點訪問Ceph

爲了讓Kubernetes的Node可以調用rbd，如果Ceph集羣和Kubernetes集羣不是在相同的機器上，還需要在Kubernetes的Node上安裝ceph-common:

yum install -y ceph-common

接下來在Kubernetes上創建ceph-secret，這個Secret將用於Kubernetes集羣的StorageClass上。

我們先查看一下ceph集羣上的所有用戶列表：

ceph auth list

這個命令會列出針對Ceph的每種類型的進程已經創建的不同權限的用戶，同時也會列出client.admin用戶，這個是Ceph集羣的管理員用戶。

接下來我們創建一個client.kube用戶：

ceph auth get-or-create client.kube
[client.kube]
 key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==

創建好的client.kube用戶用戶還沒有任何權限，下面給其授權：

ceph auth caps client.kube mon 'allow r' osd 'allow rwx pool=kube'
updated caps for client.kube

查看用戶和權限信息：

ceph auth get client.kube
exported keyring for client.kube
[client.kube]
 key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==
 caps mon = "allow r"
 caps osd = "allow rwx pool=kube"

因爲Kubernetes的Secret需要Base64編碼，下面將這個keyring轉換成Base64編碼：

ceph auth get-key client.kube | base64
QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

接下來創建Secret，ceph-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
 name: ceph-secret
 namespace: kube-system
type: kubernetes.io/rbd
data:
 key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

kubectl create -f ceph-secret.yaml
secret "ceph-secret" created

2.3 在k8s集羣創建StorageClass

首先檢查我們的集羣中是否有默認的StorageClass：

kubectl get storageclass
No resources found.

我們這裏使用的k8s集羣是使用ansible部署的Kubernetes 1.6 高可用集羣，可以看出我們部署的這個集羣並沒有創建默認的StorageClass。

我們現在集羣中創建默認的Storage Class， storege.yaml文件如下：

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: default
 annotations:
 storageclass.kubernetes.io/is-default-class: "true"
 labels:
 kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/rbd
parameters:
 monitors: 192.168.61.3:6789,192.168.61.4:6789,192.168.61.5:6789
 adminId: kube
 adminSecretName: ceph-secret
 adminSecretNamespace: kube-system
 pool: kube
 userId: kube
 userSecretName: ceph-secret-user

• annotations中storageclass.kubernetes.io/is-default-class: “true”表示這個StorageClass是集羣默認的StorageClass
• provisioner: kubernetes.io/rbd表示這個StorageClass的類型時Ceph RBD
• parameters配置了這個StorageClass使用的Ceph集羣以及RBD的相關參數
• monitors是逗號分隔的Ceph Mon節點地址
• adminId指定Ceph client 的ID需要具有能在配置的Ceph RBD Pool中創建鏡像的權限。默認值爲admin
• adminSecret：adminId的Secret Name,該Secret的type必須是”kubernetes.io/rbd”，該參數是必須的
• adminSecretNamespace： adminSecret的namespace，默認爲”default”
• pool： Ceph RBD Pool，默認爲”rbd”
• userId: Ceph client Id，用來映射RBD鏡像。
• userSecretName: userId在映射RBD鏡像時所需要的Secret的名稱。該Secret要求必須出現在和PVC相同的namespace內，並且type必須是”kubernetes.io/rbd”。該參數是必須的

創建這個默認的StorageClass：

kubectl create -f storage.yaml
storageclass "default" created

kubectl get storageclass
NAME TYPE
default (default) kubernetes.io/rbd

• (default)表示這個名稱爲default的StorageClass是k8s集羣默認的StorageClass

3.構建Redis的Docker鏡像

參考Reliable, Scalable Redis on Kubernetes中的Redis鏡像，我們的Redis的Dockerfile定製如下：

FROM harbor.frognew.com/rg/alpine-glibc:0.1

RUN apk add --no-cache redis sed bash

COPY redis-master.conf /redis-master/redis.conf
COPY redis-slave.conf /redis-slave/redis.conf
COPY run.sh /run.sh
RUN chmod u+x /run.sh
CMD [ "/run.sh" ]

ENTRYPOINT [ "bash", "-c" ]

• alpine-glibc:0.1是我們的基礎鏡像，在alpine:3.6的基礎上增加了glibc，並將時區設置爲Asia/Shanghai

參考Reliable, Scalable Redis on Kubernetes中的run.sh做如下定製，原來的run.sh不支持對redis設置密碼，加上從環境變量$REDIS_PASS讀取redis密碼：

#!/bin/bash # Copyright 2014 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. function launchmaster() { if [[ ! -e /redis-master-data ]]; then
 echo "Redis master data doesn't exist, data won't be persistent!"
 mkdir /redis-master-data
 fi
 sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-master/redis.conf
 redis-server /redis-master/redis.conf --protected-mode no } function launchsentinel() { while true; do
 master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1) if [[ -n ${master} ]]; then
 master="${master//\"}" else
 master=${REDIS_MASTER_SERVICE_HOST} fi

 redis-cli -a $REDIS_PASS -h ${master} INFO
 if [[ "$?" == "0" ]]; then break fi
 echo "Connecting to master failed. Waiting..."
 sleep 10 done

 sentinel_conf=sentinel.conf

 echo "sentinel monitor mymaster ${master} 6379 2" > ${sentinel_conf}
 echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${sentinel_conf}
 echo "sentinel down-after-milliseconds mymaster 60000" >> ${sentinel_conf}
 echo "sentinel failover-timeout mymaster 180000" >> ${sentinel_conf}
 echo "sentinel parallel-syncs mymaster 1" >> ${sentinel_conf}
 echo "bind 0.0.0.0" >> ${sentinel_conf}

 redis-sentinel ${sentinel_conf} --protected-mode no } function launchslave() { while true; do
 master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1) if [[ -n ${master} ]]; then
 master="${master//\"}" else
 echo "Failed to find master."
 sleep 60 exit 1 fi 
 redis-cli -a $REDIS_PASS -h ${master} INFO
 if [[ "$?" == "0" ]]; then break fi
 echo "Connecting to master failed. Waiting..."
 sleep 10 done
 sed -i "s/%master-ip%/${master}/" /redis-slave/redis.conf
 sed -i "s/%master-port%/6379/" /redis-slave/redis.conf
 sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-slave/redis.conf
 redis-server /redis-slave/redis.conf --protected-mode no } if [[ "${MASTER}" == "true" ]]; then
 launchmaster
 exit 0 fi if [[ "${SENTINEL}" == "true" ]]; then
 launchsentinel
 exit 0 fi

launchslave

• 這個腳本根據環境變量MASTER, SENTINEL來判斷是啓動不同類型的redis進程，如果MASTER爲true，則啓動redis master，否則如果SENTINEL爲true則啓動redis sentinel，否則啓動redis salve
• 從環境變量REDIS_PASS中讀取並設置redis的密碼

redis-master.conf的配置文件內容如下：

daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 60
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /redis-master-data
slave-serve-stale-data yes
rename-command FLUSHALL ""
rename-command FLUSHDB ""
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass %redis-pass%
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64 set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

redis-slave.conf配置文件內容如下：

daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 60
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir "/data"
slaveof %master-ip% %master-port% 
masterauth %redis-pass%
slave-serve-stale-data yes
rename-command FLUSHALL ""
rename-command FLUSHDB ""
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass %redis-pass%
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64 set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

構建redis鏡像並推送到我們的私有倉庫：

docker build -t harbor.frognew.com/rg/redis:1.0 .
docker push harbor.frognew.com/rg/rg/redis

4.在Kubernetes集羣上部署Redis

假設我們的redis要部署在devops這個namespace下，先在這個namespace下創建ceph-secret-user這個Secret：

apiVersion: v1
kind: Secret
metadata:
 name: ceph-secret-user
 namespace: devops
type: kubernetes.io/rbd
data:
 key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

kubectl crate -f ceph-secret-user.yaml

4.1 redis-master.statefulset.yaml

redis-master.statefulset.yaml是redis master的Service和StatefulSet。

apiVersion: v1
kind: Service
metadata:
 name: redis-master
 namespace: devops
 labels:
 name: redis-master
spec:
 ports: - port: 6379
 selector:
 redis-master: "true" ---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
 name: redis-master
 namespace: devops
 labels:
 name: redis-master
spec:
 serviceName: redis-master
 replicas: 1 template:
 metadata:
 labels:
 app: redis-master
 redis-master: "true"
 spec:
 terminationGracePeriodSeconds: 10
 containers: - name: redis
 image: harbor.frognew.com/rg/redis:1.0
 imagePullPolicy: IfNotPresent
 ports: - containerPort: 6379
 env: - name: MASTER
 value: "true" - name: REDIS_PASS
 valueFrom:
 secretKeyRef:
 name: devopssecret
 key: redisAuthPass
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "512Mi"
 cpu: "500m"
 volumeMounts: - name: redis-master-volume
 mountPath: /data
 imagePullSecrets: - name: regsecret
 volumeClaimTemplates: - metadata:
 name: redis-master-volume
 spec:
 accessModes: [ "ReadWriteOnce" ]
 resources:
 requests:
 storage: 5Gi

• 通過設置環境變量MASTER爲true，表明以master形式啓動redis，而環境變量REDIS_PASS從devopssecret這個Secret中獲取的值，這裏略過devopssecret這個Secret的內容
• volumeClaimTemplates中定義了PVC，因爲沒有給定storageClassName，所以將使用我們前面創建的默認的StorageClass，會根據PVC動態創建StatefulSet中Pod所需的PV

4.2 redis-sentinel.statefulset.yaml

redis-sentinel.statefulset.yaml定義了redis-sentinel的Service和StatefulSet：

apiVersion: v1
kind: Service
metadata:
 name: redis-sentinel
 namespace: devops
 labels:
 name: redis-sentinel
spec:
 ports: - port: 26379
 targetPort: 26379
 selector:
 redis-sentinel: "true" ---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
 name: redis-sentinel
 namespace: devops
spec:
 serviceName: redis-sentinel
 replicas: 3 template:
 metadata:
 labels:
 redis-sentinel: "true"
 spec:
 terminationGracePeriodSeconds: 10
 containers: - name: redis-sentinel
 image: harbor.frognew.com/rg/redis:1.0
 imagePullPolicy: IfNotPresent
 ports: - containerPort: 26379
 name: redis-sentinel
 env: - name: SENTINEL
 value: "true" - name: REDIS_PASS
 valueFrom:
 secretKeyRef:
 name: devopssecret
 key: redisAuthPass
 imagePullSecrets: - name: regsecret

• sentinel的啓動邏輯可以查看Docker鏡像中的run.sh中launchsentinel()的邏輯

4.5 redis.statefulset.yaml

redis.statefulset.yaml定義了redis slave的Service和SatefulSet：

apiVersion: v1
kind: Service
metadata:
 name: redis
 namespace: devops
 labels:
 app: redis
spec:
 ports: - port: 6379
 clusterIP: None
 selector:
 app: redis
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
 name: redis
 namespace: devops
 labels:
 name: redis
spec:
 serviceName: redis
 replicas: 2 template:
 metadata:
 labels:
 app: redis
 spec:
 terminationGracePeriodSeconds: 10
 containers: - name: redis
 image: harbor.frognew.com/rg/redis:1.0
 imagePullPolicy: IfNotPresent
 ports: - containerPort: 6379
 env: - name: REDIS_PASS
 valueFrom:
 secretKeyRef:
 name: devopssecret
 key: redisAuthPass
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "512Mi"
 cpu: "500m"
 volumeMounts: - name: redis-volume
 mountPath: /data
 imagePullSecrets: - name: regsecret
 volumeClaimTemplates: - metadata:
 name: redis-volume
 spec:
 accessModes: [ "ReadWriteOnce" ]
 resources:
 requests:
 storage: 5Gi

• volumeClaimTemplates中定義了PVC，因爲沒有給定storageClassName，所以將使用我們前面創建的默認的StorageClass，會根據PVC動態創建StatefulSet中Pod所需的PV。

4.4 以StatefulSet的形式部署Redis

下面實際操作一遍基於StatefulSet的Redis的部署。

先創建redis-master的Service和StatefulSet：

kubectl create -f redis-master.statefulset.yaml
service "redis-master" created
statefulset "redis-master" created

確保這redis master Pod處於running狀態：

kubectl get pods -l redis-master="true" -n devops
NAME READY STATUS RESTARTS AGE
redis-master-0 1/1 Running 0 48s

kubectl get svc -l name="redis-master" -n devops
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-master 10.104.132.220 <none> 6379/TCP 1m

下面創建redis-sentinel的Service和StatefulSet：

kubectl create -f redis-sentinel.statefulset.yaml
service "redis-sentinel" created
statefulset "redis-sentinel" created

kubectl get svc -l name="redis-sentinel" -n devops -o wide
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
redis-sentinel 10.97.4.9 <none> 26379/TCP 16s redis-sentinel=true

查看StatefulSet確保DESIRED和CURRENT的數量是相同的。

kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis-master 1 1 3m
redis-sentinel 3 3 42s

查看sentinel Pod：

kubectl get pod -l redis-sentinel="true" -n devops
NAME READY STATUS RESTARTS AGE
redis-sentinel-0 1/1 Running 0 1m
redis-sentinel-1 1/1 Running 0 1m
redis-sentinel-2 1/1 Running 0 1m

下面創建redis slave的Service和StatefulSet：

kubectl create -f redis.statefulset.yaml
service "redis" created
statefulset "redis" created

注意上面的過程中，在創建redis master和slave的stateful set時可能需要一定的時間，因爲涉及到PVC, PV, rbd image的創建，耐心等待。

因爲redis-master這個StatefulSet的副本數爲1，redis slave這個SatefulSet中的副本數爲2，所以我們可以看到集羣中創建了3個PVC，並創建了3個PV:

kubectl get pvc -n devops
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
redis-master-volume-redis-master-0 Bound pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175 5Gi RWO default 10m
redis-volume-redis-0 Bound pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO default 7m
redis-volume-redis-1 Bound pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO default 7m


kubectl get pv -n devops
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-volume-redis-0 default 10m
pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-volume-redis-1 default 7m
pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-master-volume-redis-master-0 default 7m

一定要確認STATUS的狀態爲Bound，如果不是可以通過kubectl describe pvc <pvc-name> -n <namespace>查看具體的事件。實際上rbd image的創建是由controller-manager調用rbd命令完成的，所以如果有問題也可以看一下controller-manager的日誌。

下面的decribe pv命令詳細打印出了這個PV已經使用Ceph RBD Image:

kubectl describe pv pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 -n devops
Name: pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 Labels: <none> Annotations: pv.kubernetes.io/bound-by-controller=yes
 pv.kubernetes.io/provisioned-by=kubernetes.io/rbd
StorageClass: default Status: Bound Claim: devops/redis-volume-redis-0 Reclaim Policy: Delete Access Modes: RWO
Capacity: 5Gi Message: Source: Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
 CephMonitors: [192.168.61.3:6789 192.168.61.4:6879 192.168.61.5:6789]
 RBDImage: kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
 FSType:
 RBDPool: kube
 RadosUser: kube
 Keyring: /etc/ceph/keyring
 SecretRef: &{ceph-secret-user}
 ReadOnly: false
Events: <none>

另外可以在Ceph集羣中查看創建的rbd image：

rbd list kube
kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
kubernetes-dynamic-pvc-8b1be6fc-7a7b-11e7-ac3c-1866da8c2fcd
kubernetes-dynamic-pvc-cce6429c-7a7a-11e7-ac3c-1866da8c2fcd

rbd info -p kube --image kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
rbd image 'kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd':
 size 5120 MB in 1280 objects
 order 22 (4096 kB objects)
 block_name_prefix: rb.0.3e17b.238e1f29
 format: 1

kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis 2 2 5m
redis-master 1 1 10m
redis-sentinel 3 3 7m

我們重點來看一下redis statefulset和redis service：

kubectl get svc -l app="redis" -n devops
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis None <none> 6379/TCP 7m


kubectl get pod -l app="redis" -n devops
NAME READY STATUS RESTARTS AGE
redis-0 1/1 Running 0 7m
redis-1 1/1 Running 0 7m

注意redis service的CLUSTER-IP爲None，這是由有狀態服務的特徵決定的。 有狀態服務具有以下特徵：

• 要求有穩定的網絡身份，即唯一不變的hostname，並保存在DNS中。hostname是由statefulset的名字後邊跟隨”-序號”組成，這裏是redis-1, redis-2。 同時每個Pod的網絡身份也是通過Service定義被創建出來了，根據Service的定義，通過ClusterIp:None指定，該Service將在DNS生成一條沒有ClusterIP的記錄。
• 要求有持久穩定的存儲，通過PVC和PV提供。這裏使用了Kubernetes的通過Dynamic Storage Provision特性，PV使用StorageClass來動態創建。
• redis-0,redis-1這兩個是reddis的slave節點。

最後我們來看一下k8s集羣中redis節點：

kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis 2 2 8m
redis-master 1 1 12m
redis-sentinel 3 3 10m

我們以StatefulSet的形式部署了1個master, 2個slave, 3個sentinel。當其中master節點發生故障時，sentinel會從剩餘redis節點中選舉新的master並切換。 3個redis節點的數據都是保存在ceph rbd中。

本文轉自中文社區-在Kubernetes中使用Sateful Set部署Redis