Kubernetes實錄(18) Kubernetes監控方案-使用prometheus實現全棧監控(4)

Kubernetes實錄系列記錄文檔完整目錄參考: Kubernetes實錄-目錄

相關記錄鏈接地址 ：

• 第15篇：Kubernetes監控方案-使用prometheus實現全棧監控(1)-監控場景與架構
• 第16篇：Kubernetes監控方案-使用prometheus實現全棧監控(2)-Kubernetes集成部署各種metrics客戶端服務
• 第17篇：Kubernetes監控方案-使用prometheus實現全棧監控(3)-prometheus部署並配置獲取各種metrics
• 第18篇：Kubernetes監控方案-使用prometheus實現全棧監控(4)-監控可視化工具grafana配置

前面的幾篇記錄（15/16/17）記錄了kubernetes使用prometheus作爲監控方案的場景，架構，各種exporter配置以及prometheus的集成部署配置。本篇記錄可視化工具Grafana在kubernetes監控方案中的集成。
備註：grafana的介紹可以參考grafana官方文檔

一、kubernetes集成部署grafana

1.1 grafana定義文件

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:5.0.4
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
          protocol: TCP
        volumeMounts:
        - mountPath: /var
          name: grafana-storage
        env:
        - name: INFLUXDB_HOST
          value: monitoring-influxdb
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
          # The following env variables are required to make Grafana accessible via
          # the kubernetes api-server proxy. On production clusters, we recommend
          # removing these env variables, setup auth for grafana, and expose the grafana
          # service using a LoadBalancer or a public IP.
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: Admin
        - name: GF_SERVER_ROOT_URL
          # If you're only using the API Server proxy, set this value instead:
          # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
          value: /
      volumes:
      - name: grafana-storage
        emptyDir: {}
      tolerations:
      - key: node-role.kubernetes.io/master
        value: ""
        effect: NoSchedule
      nodeSelector:
        node-role.kubernetes.io/master: ""
---
apiVersion: v1
kind: Service
metadata:
  name: monitoring-grafana
  namespace: kube-system
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/tcp-probe: 'true'
    prometheus.io/tcp-probe-port: '80'
  labels:
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    #kubernetes.io/cluster-service: 'true'
    #kubernetes.io/name: monitoring-grafana
spec:
  # type: NodePort
  # In a production setup, we recommend accessing Grafana through an external Loadbalancer
  # or through a public IP.
  # type: LoadBalancer
  # You could also use NodePort to expose the service at a randomly-generated port
  # type: NodePort
  #type: ClusterIP
  selector:
    k8s-app: grafana
  ports:
    - name: grafana
      port: 80
      targetPort: 3000


---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  rules:
  - host: grafana.ejuops.com
    http:
      paths:
      - path: /
        backend:
          serviceName: monitoring-grafana
          servicePort: grafana

1.2 部署grafana

反向代理使用traefik，域名使用grafana.ejuops.com(非真域名，需要配置hosts訪問)

kubectl apply -f grafana-deployment.yaml 
	deployment.extensions/monitoring-grafana created
	service/monitoring-grafana created
	ingress.extensions/monitoring-grafana created

kubectl get -f grafana-deployment.yaml 
	NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
	deployment.extensions/monitoring-grafana   1/1     1            1           5m
	
	NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
	service/monitoring-grafana   ClusterIP   10.99.65.209   <none>        80/TCP    5m
	
	NAME                                    HOSTS                 ADDRESS   PORTS   AGE
	ingress.extensions/monitoring-grafana   grafana.ejuops.com             80      6d

1.3 訪問grafana

配置hosts文件（假域名訪問grafana.ejuops.com）

C:\Windows\System32\drivers\etc\hosts
10.99.12.201 grafana.ejuops.com

二、grafana配置

2.1 配置數據源爲前面章節配置好的prometheus服務

參數項	值	類型	備註
name	kubernetes-prometheus	字符串
Type	Prometheus	下拉選擇項
URL	http://prometheus.ejuops.com	訪問地址
Access	direct	下拉選擇項

2.2 配置dashboard

grafana的dashboard可以根據需求自定義，也可以導入（import）官方已經配置好的模板，直接輸入需要import的模板號。
更多模板可以去grafana模板市場去查 https://grafana.com/dashboards?search=kubernetes

模板號	描述	備註
315,1621	cadvisor採集的各種指標的圖表	二選一
1860	node-exporter採集的各種主機相關的指標的圖表
6417	kube-state-metrics採集的各種k8s資源對象的狀態的圖表
4859,4865	blackbox-exporter採集的服務的http狀態指標的圖表	二選一
8588	Kubernetes Deployment Statefulset Daemonset metrics
5345	blackbox-exporter採集的服務的網絡狀態指標的圖表

• 第一步： 選擇New dashboard，import dashboard
  
• 第二步，根據模板號查詢，導入模板，例如315
  
• 第三步，顯示效果（模板號315）
  
  其他模板的顯示這裏就不展示了。