Kubernetes实录(18) Kubernetes监控方案-使用prometheus实现全栈监控(4)

Kubernetes实录系列记录文档完整目录参考: Kubernetes实录-目录

相关记录链接地址 ：

• 第15篇：Kubernetes监控方案-使用prometheus实现全栈监控(1)-监控场景与架构
• 第16篇：Kubernetes监控方案-使用prometheus实现全栈监控(2)-Kubernetes集成部署各种metrics客户端服务
• 第17篇：Kubernetes监控方案-使用prometheus实现全栈监控(3)-prometheus部署并配置获取各种metrics
• 第18篇：Kubernetes监控方案-使用prometheus实现全栈监控(4)-监控可视化工具grafana配置

前面的几篇记录（15/16/17）记录了kubernetes使用prometheus作为监控方案的场景，架构，各种exporter配置以及prometheus的集成部署配置。本篇记录可视化工具Grafana在kubernetes监控方案中的集成。
备注：grafana的介绍可以参考grafana官方文档

一、kubernetes集成部署grafana

1.1 grafana定义文件

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:5.0.4
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
          protocol: TCP
        volumeMounts:
        - mountPath: /var
          name: grafana-storage
        env:
        - name: INFLUXDB_HOST
          value: monitoring-influxdb
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
          # The following env variables are required to make Grafana accessible via
          # the kubernetes api-server proxy. On production clusters, we recommend
          # removing these env variables, setup auth for grafana, and expose the grafana
          # service using a LoadBalancer or a public IP.
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: Admin
        - name: GF_SERVER_ROOT_URL
          # If you're only using the API Server proxy, set this value instead:
          # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
          value: /
      volumes:
      - name: grafana-storage
        emptyDir: {}
      tolerations:
      - key: node-role.kubernetes.io/master
        value: ""
        effect: NoSchedule
      nodeSelector:
        node-role.kubernetes.io/master: ""
---
apiVersion: v1
kind: Service
metadata:
  name: monitoring-grafana
  namespace: kube-system
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/tcp-probe: 'true'
    prometheus.io/tcp-probe-port: '80'
  labels:
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    #kubernetes.io/cluster-service: 'true'
    #kubernetes.io/name: monitoring-grafana
spec:
  # type: NodePort
  # In a production setup, we recommend accessing Grafana through an external Loadbalancer
  # or through a public IP.
  # type: LoadBalancer
  # You could also use NodePort to expose the service at a randomly-generated port
  # type: NodePort
  #type: ClusterIP
  selector:
    k8s-app: grafana
  ports:
    - name: grafana
      port: 80
      targetPort: 3000


---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  rules:
  - host: grafana.ejuops.com
    http:
      paths:
      - path: /
        backend:
          serviceName: monitoring-grafana
          servicePort: grafana

1.2 部署grafana

反向代理使用traefik，域名使用grafana.ejuops.com(非真域名，需要配置hosts访问)

kubectl apply -f grafana-deployment.yaml 
	deployment.extensions/monitoring-grafana created
	service/monitoring-grafana created
	ingress.extensions/monitoring-grafana created

kubectl get -f grafana-deployment.yaml 
	NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
	deployment.extensions/monitoring-grafana   1/1     1            1           5m
	
	NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
	service/monitoring-grafana   ClusterIP   10.99.65.209   <none>        80/TCP    5m
	
	NAME                                    HOSTS                 ADDRESS   PORTS   AGE
	ingress.extensions/monitoring-grafana   grafana.ejuops.com             80      6d

1.3 访问grafana

配置hosts文件（假域名访问grafana.ejuops.com）

C:\Windows\System32\drivers\etc\hosts
10.99.12.201 grafana.ejuops.com

二、grafana配置

2.1 配置数据源为前面章节配置好的prometheus服务

参数项	值	类型	备注
name	kubernetes-prometheus	字符串
Type	Prometheus	下拉选择项
URL	http://prometheus.ejuops.com	访问地址
Access	direct	下拉选择项

2.2 配置dashboard

grafana的dashboard可以根据需求自定义，也可以导入（import）官方已经配置好的模板，直接输入需要import的模板号。
更多模板可以去grafana模板市场去查 https://grafana.com/dashboards?search=kubernetes

模板号	描述	备注
315,1621	cadvisor采集的各种指标的图表	二选一
1860	node-exporter采集的各种主机相关的指标的图表
6417	kube-state-metrics采集的各种k8s资源对象的状态的图表
4859,4865	blackbox-exporter采集的服务的http状态指标的图表	二选一
8588	Kubernetes Deployment Statefulset Daemonset metrics
5345	blackbox-exporter采集的服务的网络状态指标的图表

• 第一步： 选择New dashboard，import dashboard
  
• 第二步，根据模板号查询，导入模板，例如315
  
• 第三步，显示效果（模板号315）
  
  其他模板的显示这里就不展示了。