Kube-prometheus監控jmx指標

一、程序包說明：

1)       監控代理程序包：AppMonitor.jar

2)       項目監控依賴包：AppMonitor-1.0.0.jar

3)       監控插件包：jmx_prometheus_javaagent-0.13.0.jar

 

二、部署說明：

1)       新建共享目錄，用於存放監控代理程序包、監控插件包、以及配置文件

---
startDelaySeconds: 0
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:8012/jmxrmi
lowercaseOutputName: false
lowercaseOutputLabelNames: false
ssl: false
username: jmx
password: jmx
rules:
- pattern: ".*"

 

2)       新建K8S持久化資源(PV-PVC)，用於共享上述目錄到各個容器化項目中

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jmxprometheus
spec:
  capacity:
    storage: 100M
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /data/nfsprovisioner/jmxprometheus/
    server: 10.20.30.108
    
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jmxprometheus
  namespace: default
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: "nfs"
  resources:
    requests:
      storage: 100M

 

3)       修改Deployment資源配置

a．新增標籤jmx=prometheus

b．新增監控啓動項…

c．新增監控端口jmx

d．新增共享目錄jmxprometheus

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appservice
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: appservice
      
  template:
    metadata:
      labels:
        app: appservice
        jmx: prometheus
      annotations:
        version: "20200722"
    spec:
      containers:
      - name: appservice
        image: hub.com/test/appservice:latest
        imagePullPolicy: Always
        env:
        - name: JAVA_OPTS
          value: >
            -Dprops=/root/appservice.properties -Dlogpath=/root/logs
            -javaagent:jmxprometheus/AppMonitor.jar
            -javaagent:jmxprometheus/jmx_prometheus_javaagent-0.13.0.jar=8013:jmxprometheus/namenode.yaml
   -Dcom.sun.management.jmxremote
            -Djava.rmi.server.hostname=localhost
            -Dcom.sun.management.jmxremote.port=8012
            -Dcom.sun.management.jmxremote.rmi.port=8012
            -Dcom.sun.management.jmxremote.authenticate=false
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.access.file=jmx
            -Dcom.sun.management.jmxremote.password.file=jmx
            -noverify
        ports:
        - name: dubbo
          containerPort: 20005
        - name: rest
          containerPort: 8113
        - name: jmx
          containerPort: 8013
        livenessProbe:
          initialDelaySeconds: 60
          timeoutSeconds: 10
          tcpSocket:
            port: 20005
        resources:
          limits:
            memory: 1Gi
        volumeMounts:
        - name: config
          mountPath: /root/appservice.properties
          subPath: appservice.properties- name: log
          mountPath: /root/logs
        - name: html
          mountPath: /root/html
        - name: jmxprometheus
          mountPath: /root/jmxprometheus
          
      volumes:
      - name: config
        configMap:
          name: appservice
      - name: log
        hostPath:
          path: /home/logs/appservice
          type: DirectoryOrCreate
      - name: jmxprometheus
        persistentVolumeClaim:
          claimName: jmxprometheus

 

4)       新增監控Service資源，用於動態發現應用PODIP

apiVersion: v1
kind: Service
metadata:
  name: jmxprometheus
  namespace: default
  labels:
    jmx: prometheus
spec:
  selector:
    jmx: prometheus
  ports: 
  - name: jmx
    port: 8013

 

 

5)       新增prometheus監控配置

- job_name: jmxprometheus
  honor_labels: false       # 將衝突標籤重命名爲exported_<original-label>
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  scrape_interval: 30s      # 抓取間隔
  relabel_configs:
  - action: keep            # 收集匹配regex的源標籤(service標籤爲jmx=prometheus)
    source_labels:
    - __meta_kubernetes_service_label_jmx
    regex: prometheus
  - action: keep            # 收集匹配regex的源標籤(port的name=jmx)
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: jmx
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:      # 替換標籤namespace、service、pod
    - __meta_kubernetes_namespace
    target_label: namespace    
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: jmx

 

三、監控展示：

注意：非集羣機器無法直接訪問PODIP，也就無法直接查看監控數據頁面。