大數據組件監控體系構建

大數據監控體系構建

基於開源xxx_exporter+promethues+grafana的構建監控系統

1.概況

其中，exporter一般是使用來採集各種組件運行時的指標數據；
promethues構建指標時序數據庫；
grafana構建指標顯示面板；

目前已有各種docker容器方便的構建各種監控體系；

2.詳細構建過程

其中我的promethues的配置如下：

#全局配置信息：
global:
  scrape_interval:     15s #默認抓取間隔1m, 15秒向目標抓取一次數據
  evaluation_interval: 15s 

#個性化的抓取配置信息
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['192.168.32.70:9090']
        labels:
          instance:  192.168.32.70
 
#監控物理主機：cpu、memory、disk等 
  - job_name: node_exporter
    static_configs:
      - targets:
        - '192.168.32.62:9100'
        - '192.168.32.63:9100'
        - '192.168.32.64:9100'
        - '192.168.32.65:9100'
        - '192.168.32.66:9100'
        - '192.168.32.67:9100'
        - '192.168.32.68:9100'
        - '192.168.32.69:9100'
        - '192.168.32.70:9100'
        - '192.168.32.71:9100'
        - '192.168.32.72:9100'
        - '192.168.32.73:9100'
        - '192.168.32.101:9100'
        - '192.168.32.106:9100'
        - '192.168.32.111:9100'
        - '192.168.32.116:9100'
        - '192.168.32.121:9100'
        - '192.168.32.126:9100'
        - '192.168.32.141:9100'
        - '192.168.32.146:9100'


#  - job_name: 'redis'
#    static_configs:
#     - targets: ['192.168.32.70:9121']
#       labels:
#         instance: redis-poc

  - job_name: 'mongo'
    static_configs:
     - targets: ['192.168.32.70:9104']
       labels:
         instance: mongo3


  - job_name: 'postgres'
    static_configs:
     - targets: ['192.168.32.70:9187']
       labels:
         instance: 'node70'
         platform: 'dcos-node70'

  - job_name: 'kafka'
    static_configs:
     - targets: ['192.168.32.70:9308']
       labels:
         instance: kafka-67


  - job_name: 'redis_exporter_poc'
    static_configs:
      - targets:
        - redis://192.168.32.141:16055
        - redis://192.168.32.68:16052
        - redis://192.168.32.68:16053
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.32.70:9121


  - job_name: 'elasticsearch'
    scrape_interval: 15s
    metrics_path: "/_prometheus/metrics"
    static_configs:
    - targets: ['192.168.32.62:9200']

---

3.構建之後的效果

grafana指標展示界面

2.在線下載已知的控制面板

監控虛擬機物理節點資源

kafka相關監控體系

其他的類似，還有redis、postgresql、mysql、MongoDB等相關的監控體系；

甚至還有微服務相關的監控體系

時間關係，大概展示一些圖片，其實dashboard面板還是挺美觀的

---