雲原生應用Etcd監控

一、在現有的K8s集羣ETCD節點上進行測試

測試訪問Etcd Metrics接口

[root@k8s-master01 ~]#for ((i=131;i<134;i++));do curl -s --cert /etc/kubernetes/pki/etcd/etcd.pem --key /etc/kubernetes/pki/etcd/etcd-key.pem https://192.168.126.$i:2379/metrics -k  | tail -1;done
promhttp_metric_handler_requests_total{code="503"} 0
promhttp_metric_handler_requests_total{code="503"} 0
promhttp_metric_handler_requests_total{code="503"} 0

查看etcd配置文件中的證書文件位置

[root@k8s-master01 ~]# egrep "key-file|cert-file" /etc/etcd/etcd.config.yml
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'

二、創建Etcd Service

首先需要配置Etcd的Service和End point

# [root@k8s-master01 prometheus]# vim etcd-svc.yaml

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
subsets:
- addresses:
  - ip: 192.168.126.131
  - ip: 192.168.126.132
  - ip: 192.168.126.133
  ports:
  - name: https-metrics
    port: 2379 # etcd 端口
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
  type: ClusterIP

三、創建關於etcd service、endpoints資源

[root@k8s-master01 prometheus]# kubectl apply -f etcd-svc.yaml
endpoints/etcd-prom created
service/etcd-prom created
[root@k8s-master01 prometheus]# kubectl get -f etcd-svc.yaml
NAME                  ENDPOINTS                                                        AGE
endpoints/etcd-prom   192.168.126.131:2379,192.168.126.132:2379,192.168.126.133:2379   5s

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/etcd-prom   ClusterIP   192.168.68.182   <none>        2379/TCP   5s

[root@k8s-master01 prometheus]# curl -s --cert /etc/kubernetes/pki/etcd/etcd.pem --key /etc/kubernetes/pki/etcd/etcd-key.pem https://192.168.68.182:2379/metrics -k | tail -1 #指定etcd service的cluster IP測試訪問
promhttp_metric_handler_requests_total{code="503"} 0

 

四、將證書掛載至Prometheus容器（由於Prometheus是operator部署的，所以只需要修改prometheus資源即可）

[root@k8s-master01 prometheus]# kubectl get deploy -n monitoring
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
blackbox-exporter     1/1     1            1           10d
grafana               1/1     1            1           10d
kube-state-metrics    1/1     1            1           10d
prometheus-adapter    2/2     2            2           10d
prometheus-operator   1/1     1            1           10d
[root@k8s-master01 prometheus]# #kubectl edit prometheus-operator -n monitoring

 保存退出之後Prometheus的Pod會自動重啓，最後驗證查看證書是否掛載（任意一個Prometheus的Pod均可驗證）

[root@k8s-master01 prometheus]# kubectl get pod -n monitoring -l app=prometheus
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   2/2     Running   1          39m
prometheus-k8s-1   2/2     Running   1          39m
[root@k8s-master01 prometheus]# kubectl exec prometheus-k8s-0  -n monitoring -c prometheus -- ls /etc/prometheus/secrets/etcd-ssl
etcd-ca.pem
etcd-key.pem
etcd.pem

五、創建Etcd ServiceMonitor

#vim etcd-servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd
  namespace: monitoring
  labels:
    app: etcd
spec:
  jobLabel: k8s-app
  endpoints:
    - interval: 30s
      port: https-metrics #這個port對應Service.spec.ports.name
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/secrets/etcd-ssl/etcd-ca.pem #證書路徑
        certFile: /etc/prometheus/secrets/etcd-ssl/etcd.pem
        keyFile: /etc/prometheus/secrets/etcd-ssl/etcd-key.pem
        insecureSkipVerify: true # 關閉證書校驗
  selector:
    matchLabels:
      app: etcd-prom # 跟 svc 的 lables 保持一致
  namespaceSelector:
    matchNames:
    - kube-system

六、Grafana配置

我們用戶可以登陸到grafana.com/grafana/dashboards官網監控模版平臺，可根據自己喜好選擇熱度最高的模版即可

登陸到Grafana平臺，添加Etcd的監控模版

依次點擊“+”號--->Inport,之後輸入Etcd 的Grafana Dashboard地址:

https://grafana.com/grafana/dashboards/3070 

選擇Load，然歐選擇Prometheus數據源，點擊Inport導入即可

 

最後就可以看到ETCD監控的狀態了