一、Prometheus与Grafana
Prometheus是由SoundCloud开发的开源监控报警系统和时序列数据库(TSDB)。Prometheus使用Go语言开发,是Google BorgMon监控系统的开源版本。2016年由Google发起Linux基金会旗下的原生云基金会(Cloud Native Computing Foundation), 将Prometheus纳入其下第二大开源项目。Prometheus目前在开源社区相当活跃。Prometheus和Heapster(Heapster是K8S的一个子项目,用于获取集群的性能数据。)相比功能更完善、更全面。Prometheus性能也足够支撑上万台规模的集群。其架构图如下:
Grafana是一个跨平台的开源的度量分析和可视化工具,可以通过将采集的数据查询然后可视化的展示,并及时通知。它主要有以下六大特点:
- 1、展示方式:快速灵活的客户端图表,面板插件有许多不同方式的可视化指标和日志,官方库中具有丰富的仪表盘插件,比如热图、折线图、图表等多种展示方式;
- 2、数据源:Graphite,InfluxDB,OpenTSDB,Prometheus,Elasticsearch,CloudWatch和KairosDB等;
- 3、通知提醒:以可视方式定义最重要指标的警报规则,Grafana将不断计算并发送通知,在数据达到阈值时通过Slack、PagerDuty等获得通知;
- 4、混合展示:在同一图表中混合使用不同的数据源,可以基于每个查询指定数据源,甚至自定义数据源;
- 5、注释:使用来自不同数据源的丰富事件注释图表,将鼠标悬停在事件上会显示完整的事件元数据和标记;
- 6、过滤器:Ad-hoc过滤器允许动态创建新的键/值过滤器,这些过滤器会自动应用于使用该数据源的所有查询。
二、服务端prometheus安装
1、下载
wget https://github.com/prometheus/prometheus/releases/download/v2.19.2/prometheus-2.19.2.linux-amd64.tar.gz
tar zxvf prometheus-2.19.2.linux-amd64.tar.gz
mv prometheus-2.19.2.linux-amd64 /usr/local/prometheus
2、创建用户
groupadd prometheus
useradd -g prometheus -m -d /var/lib/prometheus -s /sbin/nologin prometheus
chown prometheus.prometheus -R /usr/local/prometheus
3、创建Systemd服务
cat > /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=prometheus
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
4、启动Prometheus
systemctl daemon-reload
systemctl start prometheus
systemctl status prometheus
systemctl enable prometheus
5、访问WEB界面
访问http://10.207.10.248:9090
三、节点端node_exporter安装
1、下载
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar zxvf node_exporter-1.0.1.linux-amd64.tar.gz
mv node_exporter-1.0.1.linux-amd64 /usr/local/node_exporter
2、创建用户
groupadd prometheus
useradd -g prometheus -m -d /var/lib/prometheus -s /sbin/nologin prometheus
chown prometheus.prometheus -R /usr/local/node_exporter
3、创建Systemd服务
cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=node_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
4、启动node_exporter
systemctl daemon-reload
systemctl start node_exporter
systemctl status node_exporter
systemctl enable node_exporter
5、验证启动成功
curl 127.0.0.1:9100
curl 127.0.0.1:9100/metrics
四、服务端配置
prometheus的服务端通过pull向各个node_exporter节点端抓取信息,需要在各个node上安装exporter。可以利用 Prometheus 的 static_configs 来拉取 node_exporter 的数据。
编辑prometheus.yml文件,添加内容:
- job_name: 'node'
static_configs:
- targets: ['XXX.XXX.XXX.XXX:9100']
完整配置如下:
[root@localhost prometheus]# cat prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'server248'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['172.17.10.248:9090']
- job_name: 'node247'
static_configs:
- targets: ['172.17.10.247:9100']
- job_name: 'node246'
static_configs:
- targets: ['172.17.10.246:9100']
- job_name: 'node244'
static_configs:
- targets: ['172.17.10.244:9100']
重启prometheus,然后在Prometheus页面中的Targets中就能看到新加入的node:
systemctl restart prometheus
然后再次查询web界面如下:
访问http://10.207.10.248:9090
五、集成各服务监控
1、 nginx
https://github.com/knyar/nginx-lua-prometheus
2、 php
https://github.com/bakins/php-fpm-exporter
3、 mysql
https://github.com/prometheus/mysqld_exporter
(1)下载
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz
tar zxvf mysqld_exporter-0.12.1.linux-amd64.tar.gz
mv mysqld_exporter-0.12.1.linux-amd64 /usr/local/mysqld_exporter
(2)修改所属
chown prometheus.prometheus -R /usr/local/mysqld_exporter
(3)创建Systemd服务
cat > /etc/systemd/system/mysqld_exporter.service <<EOF
[Unit]
Description=mysqld_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
(4)创建密码填写内容.my.cnf文件
vim .my.cnf
[client]
host=127.0.0.1
port=3306
user=root
password=root
(5)启动mysqld_exporter
systemctl daemon-reload
systemctl start mysqld_exporter
systemctl status mysqld_exporter
systemctl enable mysqld_exporter
4、 zookeeper
https://github.com/jiankunking/zookeeper_exporter
5、 kafka
https://github.com/danielqsj/kafka_exporter
6、 redis
https://github.com/oliver006/redis_exporter
7、 postgresql
https://github.com/wrouesnel/postgres_exporter
8、 springboot
https://blog.csdn.net/qq_33257527/article/details/88294016
更多expoter参见官网:https://prometheus.io/docs/instrumenting/exporters/#exporters-and-integrations
六、Grafana安装
官方地址:https://grafana.com/grafana/download?platform=linux
1、下载
wget https://dl.grafana.com/oss/release/grafana-7.0.5-1.x86_64.rpm
2、安装
yum install -y grafana-7.0.5-1.x86_64.rpm
3、配置
配置文件位于/etc/grafana/grafana.ini,这里暂时保持默认配置即可
4、启动
systemctl enable grafana-server
systemctl start grafana-server
5、访问
http://10.207.10.248:3000
用户名:admin
密码:admin