一、前言
本次案例是基於Statefulset部署RabbitMQ集羣,同時基於Storage Class(存儲類)來作爲集羣數據的持久化後端,因此在此之前已經部署好了NSF作爲後端存儲
1、創建服務命名空間Namespace
#kubectl create ns rabbitmq
2、創建rbac權限
Service Account是kubernetes Pod中的程序用於訪問Kubernetes API的Account(賬號),它作爲Pod中的程序提供訪問Kubernetes API的身份標示,下面是我們創建Rabbitmq Pod的ServiceAccount,並針對Kubernetes的enpoint資源授權,創建相關的role和rolebinding
#vim rabbitmq-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: rmq-cluster
namespace: rabbitmq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rmq-cluster
namespace: rabbitmq
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rmq-cluster
namespace: rabbitmq
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rmq-cluster
subjects:
- kind: ServiceAccount
name: rmq-cluster
namespace: rabbitmq
#kubectl create -f rabbitmq-rbac.yaml
#kubectl get -f rabbitmq-rbac.yaml
3、創建訪問rabbitmqm集羣的service
#vim rabbitmq-service.yaml
kind: Service
apiVersion: v1
metadata:
labels:
app: rmq-cluster
name: rmq-cluster
namespace: rabbitmq
spec:
clusterIP: None
ports:
- name: amqp
port: 5672
targetPort: 5672
selector:
app: rmq-cluster
---
kind: Service
apiVersion: v1
metadata:
labels:
app: rmq-cluster
type: LoadBalancer
name: rmq-cluster-balancer
namespace: rabbitmq
spec:
ports:
- name: http
port: 15672
protocol: TCP
targetPort: 15672
- name: amqp
port: 5672
protocol: TCP
targetPort: 5672
selector:
app: rmq-cluster
type: NodePort
#kubectl create -f rabbitmq-service.yaml
#kubectl get -f rabbitmq-service.yaml
4、創建rabbitmq secret
#vim rabbitmq-secret.yaml
kind: Secret
apiVersion: v1
metadata:
name: rmq-cluster-secret
namespace: rabbitmq
stringData:
cookie: ERLANG_COOKIE
password: RABBITMQ_PASS
url: amqp://RABBITMQ_USER:RABBITMQ_PASS@rmq-cluster-balancer
username: RABBITMQ_USER
type: Opaque
# kubectl create -f rabbitmq-secret.yaml
#kubectl get -f rabbitmq-secret.yaml
5、創建rabbitmq configmap資源
#vim rabbitmq-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: rmq-cluster-config
namespace: rabbitmq
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
rabbitmq.conf: |
loopback_users.guest = false
default_user = RABBITMQ_USER
default_pass = RABBITMQ_PASS
## Clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
#################################################
# public-service is rabbitmq-cluster's namespace#
#################################################
cluster_formation.k8s.hostname_suffix = .rmq-cluster.rabbitmq.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
## queue master locator
queue_master_locator=min-masters
#kubectl create -f rabbitmq-configmap.yaml
#kubectl get -f rabbitmq-configmap.yaml
6、通過Statefulset類型創建rabbitmq集羣
#vim rabbitmq-cluster-sts.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
labels:
app: rmq-cluster
name: rmq-cluster
namespace: rabbitmq
spec:
replicas: 3
selector:
matchLabels:
app: rmq-cluster
serviceName: rmq-cluster
template:
metadata:
labels:
app: rmq-cluster
spec:
containers:
- args:
- -c
- cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh
rabbitmq-server
command:
- sh
env:
- name: RABBITMQ_DEFAULT_USER
valueFrom:
secretKeyRef:
key: username
name: rmq-cluster-secret
- name: RABBITMQ_DEFAULT_PASS
valueFrom:
secretKeyRef:
key: password
name: rmq-cluster-secret
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
key: cookie
name: rmq-cluster-secret
- name: K8S_SERVICE_NAME
value: rmq-cluster
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: RABBITMQ_USE_LONGNAME
value: "true"
- name: RABBITMQ_NODENAME
value: rabbit@$(POD_NAME).rmq-cluster.$(POD_NAMESPACE).svc.cluster.local
- name: RABBITMQ_CONFIG_FILE
value: /var/lib/rabbitmq/rabbitmq.conf
image: registry.cn-beijing.aliyuncs.com/dotbalo/rabbitmq:3.7-management
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 30
timeoutSeconds: 10
name: rabbitmq
ports:
- containerPort: 15672
name: http
protocol: TCP
- containerPort: 5672
name: amqp
protocol: TCP
readinessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 10
timeoutSeconds: 10
volumeMounts:
- mountPath: /etc/rabbitmq
name: config-volume
readOnly: false
- mountPath: /var/lib/rabbitmq
name: rabbitmq-storage
readOnly: false
serviceAccountName: rmq-cluster
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: rabbitmq.conf
path: rabbitmq.conf
- key: enabled_plugins
path: enabled_plugins
name: rmq-cluster-config
name: config-volume
volumeClaimTemplates:
- metadata:
name: rabbitmq-storage
spec:
accessModes:
- ReadWriteMany
storageClassName: "nfs-storage"
resources:
requests:
storage: 4Gi
#kubectl create -f rabbitmq-cluster-sts.yaml
二、驗證集羣
進入其中一個pod副本查看集羣狀態是否異常
[root@k8s-master01 rabbitmq]# kubectl exec -it rmq-cluster-0 -n rabbitmq bash
通過service地址+端口訪問rabbitmq web界面或者創建ingress實現域名訪問1