Celery: grafana + promethus +celery_explore

celery 監控

• flower(web ui and realtime)
• envents(終端)
• grafana + promethus + explore

官方監控指南 前兩種基於官方教程，參考連接即可.

本文解釋第三種方式:

• explore: 採集celery指標
• promethus: 收集數據
• grafana: 報表統計

1.docker-compose 管理服務

• docker-compose.yml

version: '2'

services:
  # exporter
  cadvisor:
   image: google/cadvisor:latest
   container_name: cadvisor
   restart: unless-stopped
   ports:
     - '8080:8080'
   network_mode: host
   volumes:
     - /:/rootfs:ro
     - /var/run:/var/run:rw
     - /sys:/sys:ro
     - /var/lib/docker/:/var/lib/docker:ro

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    user: '0'
    network_mode: host
    ports:
      - '9100:9100'
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.textfile.directory=/node_exporter/prom'
    volumes:
      - /:/rootfs
      - /proc:/host/proc
      - /sys:/host/sys
      - ~/docker/prometheus_exporter/prom:/node_exporter/prom

  # redis exporter
  redis-exporter:
    image: oliver006/redis_exporter
    container_name: redis-exporter
    hostname: redis-exporter
    restart: always
    network_mode: host
    ports:
      - "9121:9121"
    command:
      - "--redis.addr=redis://localhost:6379"

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    network_mode: host
    ports:
      - '9090:9090'
    user: '0'
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus/data'
      - '--storage.tsdb.retention=90d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/conf/prometheus.yml:/etc/prometheus/prometheus.yml
      - ~/docker/prometheus/data:/prometheus/data
    depends_on:
      #- cadvisor
      - node-exporter

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    user: '0'
    network_mode: host
    ports:
      - '3000:3000'
    depends_on:
      - prometheus
    volumes:
      - ~/docker/grafana:/var/lib/grafana
      - ./grafana/logs:/var/log/grafana
      - ./grafana/conf/grafana.ini:/etc/grafana/grafana.ini
    environment:
      - GF_PATHS_CONFIG=/etc/grafana/grafana.ini

1.celery採集器

celery採集器

啓動採集器

(venv) ➜ git:(master) ✗ pip install celery-prometheus-exporter
(venv) ➜ git:(master) ✗ celery-prometheus-exporter --broker redis://0.0.0.0:6379/2 --addr 0.0.0.0:8888  --enable-events --verbose
[2021-03-08 14:50:46,793] root:INFO: Starting HTTPD on 0.0.0.0:8888
[2021-03-08 14:50:47,859] kombu.mixins:INFO: Connected to redis://0.0.0.0:6379/2

2.prometheus 增加採集任務

• celery 採集服務的ip:port

prometheus/conf/prometheus.yml 加入celery-exporter採集任務

  - job_name: 'celery-exporter'
    scrape_interval: 5s
    static_configs:
    - targets: ['ip:8888']

重啓prometheus後, 查看是否採集成功

docker-compose restart prometheus

3.grafana 報表

報表模板