前言：
1、線上業務使用了幾十上百臺haproxy服務，需要針對這些服務做監控(包含haproxy端口對應的TCP連接數監控，haproxy服務監控等)。
2、普通的在zabbix_agentd.conf裏面配置"UserParameter"的方式太繁瑣，這裏使用建表(收集數據,彙總數據)的方式。
3、將數據彙總到表裏面，通過第三方腳本對數據做一個處理，拿到處理結果並監控。
一、創建數據表(收集haproxy服務的信息)
1.1、在zabbix_DB上創建表：t_3party_server
create table t_3party_server(
app varchar(64) NOT NULL,
server varchar(64) NOT NULL,
division varchar(64) NOT NULL,
node varchar(64) NOT NULL,
port varchar(64) NOT NULL,
remark int(10) unsigned NOT NULL,
PRIMARY KEY (app,server,division,node,port)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
註釋：
app #這裏寫的是項目名稱(MOBA項目)
server #服務名稱(haproxy服務)
division #由於起了多個haproxy實例，名稱也不一樣，所以這裏做了區分
node #haproxy服務IP地址
port #haproxy服務端口,多個實例端口不同
remark #備註作用(由於線上某些haproxy服務安裝路徑不一致，所以此處用來區分路徑，也可以區分其他)
PRIMARY KEY (app,server,division,node,port) #添加主鍵
1.2、創建完畢後效果
mysql> desc t_3party_server;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| app | varchar(64) | NO | PRI | NULL | |
| server | varchar(64) | NO | PRI | NULL | |
| division | varchar(64) | NO | PRI | NULL | |
| node | varchar(64) | NO | PRI | NULL | |
| port | varchar(64) | NO | PRI | NULL | |
| remark | text | NO | | NULL | |
+----------+-------------+------+-----+---------+-------+
6 rows in set (0.00 sec)
1.3、然後往數據表裏面插入數據：
1.3.1、在Navicat上添加
insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-login','10.117.16.242','30021','0');
insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-match','10.117.16.242','30032','0');
insert into t_3party_server(app,server,division,node,port,status) values('MOBA','haproxy','haproxy-test','10.117.16.242','30035','0');
1.3.2、在MySQL數據庫客戶端上添加：
INSERT INTO `zabbix`.`t_3party_server` (`app`, `server`, `division`, `node`, `port`, `remark`) VALUES ('MOBA', 'haproxy', 'haproxy-test', '10.117.16.242', '30035', '0');
1.4、最終數據生成圖
二、針對生成的數據表做監控
2.1、監控腳本：放在zabbix_server下 "/install_path/zabbix_server/share/zabbix/externalscripts"
#!/usr/bin/php
<?php
if ($_SERVER["argc"] != 2) {
echo "Usage: " . $_SERVER["argv"][0] . " host\n";
return;
}
$host = $_SERVER["argv"][1];
$mysqlconn = new mysqli("10.116.207.188", "moba", "moba2016", "zabbix", 3306);
if (mysqli_connect_errno()) {
echo "fail to connect mysql: " . mysqli_connect_error() . "\n";
return;
}
$result = $mysqlconn->query("SELECT * FROM t_3party_server WHERE node=\"" . $mysqlconn->real_escape_string($host) . "\"");
$hostvalue = array(
"data" => array()
);
while ($row = $result->fetch_assoc()) {
$app = $row["app"];
$server = $row["server"];
$division = $row["division"];
$node = $row["node"];
$port = $row["port"];
$remark = $row["remark"];
if ($remark == "/usr/local")
{
$hostvalue["data"][] = array(
"{#APP}" => $app,
"{#SERVER}" => $server,
"{#DIVISION}" => $division,
"{#PORT}" => $port,
"{#PATH}" => "/usr/local/" . $division . "/sbin",
);
continue;
}
$server = substr($server, 0, 15);
$hostvalue["data"][] = array(
"{#APP}" => $app,
"{#SERVER}" => $server,
"{#DIVISION}" => $division,
"{#PORT}" => $port,
"{#PATH}" => "/data/app/" . $division . "/sbin",
#"{#MFW_PATH}" => "/data/app/" . $division . "/" . ($division ? $division . "/" : "") . $server,
);
}
echo json_encode($hostvalue);
echo "\n";
2.1.1、腳本生成的數據：
./mfw_discovery_haproxy.php 10.117.16.222
{"data":[{"{#APP}":"MOBA","{#SERVER}":"haproxy","{#DIVISION}":"haproxy-login","{#PORT}":"30021","{#PATH}":"\/data\/app\/haproxy-login\/sbin"}]}
2.1.2、轉換爲Json格式如下：
{
"data":[
{
"{#APP}":"MOBA",
"{#SERVER}":"haproxy",
"{#DIVISION}":"haproxy-login",
"{#PORT}":"30021",
"{#PATH}":"/data/app/haproxy-login/sbin"
}
]
}
2.2、在haproxy服務機器上配置
1、配置：zabbix_agentd.conf
vim /data/app/zabbix_agent/etc/zabbix_agentd.conf
LogRemoteCommands=1
LogFileSize=1
DebugLevel=3
#Haproxy #用來監控TCP連接數
UserParameter=haproxy.tcp.[*],/usr/sbin/ss -ant sport eq :"$1" | awk 'NR>1{++s[$$1]} END {for(k in s){print k,s[k]}}' |grep -w "ESTAB" |awk '{print $$2}'
2、配置：haproxy服務守護腳本 #此處可不配(根據業務選擇)
啓動腳本：
#!/bin/bash
old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy
if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
echo -e "\033[32m $path/$server was running!.... \033[0m"
ps -ef|grep "$path/$server"|grep -v "grep"
else
echo -e "\033[32m start $path/$server .... \033[0m"
$path/$server -f $path/conf/haproxy.cfg -p /tmp/$RANDOM-haproxy.pid -n 20000 -N 20000
if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
echo -e "\033[32m start $path/$server ok .... \033[0m"
ps -ef|grep "$path/$server"|grep -v "grep"
else
echo -e "\033[31m start $path/$server faild .... \033[0m"
fi
fi
關閉腳本：
#!/bin/bash
old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy
if [ "$(ps -ef|grep "$path/$server"|grep -v "grep" | awk '{print $2}')" == "" ];then
echo -e "\033[32m can not find $path/$server process .... \033[0m"
else
killall $path/$server 2>/dev/null
until [ -z "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
do
sleep 1
done
ps -ef|grep "$path/$server"|grep -v "grep"
echo -e "\033[31m stop $path/$server ok .... \033[0m"
fi
ps -ef|grep "$path/$server"|grep -v "grep"
killall $path/$server 2>/dev/null
重載腳本：
#!/bin/bash
old_path=`pwd`
cd `dirname $0`
path=`pwd`
server=sbin/haproxy
hpid=`ps -ef|grep "$path/$server"|grep -v "grep" |awk '{print $ 12}'`
if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
echo -e "\033[32m $path/$server is running!.... \033[0m"
$path/$server -f $path/conf/haproxy.cfg -p $hpid -n 20000 -N 20000 -sf $(cat $hpid)
ps -ef|grep "$path/$server"|grep -v "grep"
else
echo -e "\033[31m $path/$server was not running.... \033[0m"
echo -e "\033[32m start $path/$server process.... \033[0m"
$path/$server -f $path/conf/haproxy.cfg -p /tmp/$RANDOM-haproxy.pid -n 20000 -N 20000
if [ -n "`ps -ef|grep "$path/$server"|grep -v "grep"`" ]
then
echo -e "\033[32m start $path/$server ok .... \033[0m"
ps -ef|grep "$path/$server"|grep -v "grep"
else
echo -e "\033[31m start $path/$server faild .... \033[0m"
fi
fi
守護進程腳本：
#!/bin/sh
#chkconfig: 2345 59 81
#description: The Haproxy service daemon script.
if [ -d /usr/local/haproxy/ ];then
SCRIPT=`/bin/find /usr/local/*haproxy* -name 'start.sh'`
else
SCRIPT=`/bin/find /data/app/*haproxy* -name 'start.sh'`
fi
PROC_NAME=haproxy
PID=0
function proc_num()
{
NUM=`ps -ef | grep ${PROC_NAME} | grep -v grep |wc -l`
return $NUM
}
function proc_id()
{
PID=`ps -ef | grep ${PROC_NAME} | grep -v grep | awk '{print $2}'`
return $PID
}
function shell(){
echo "${SCRIPT}" | /bin/sed -n '1p' | xargs /bin/sh >>/dev/null
sleep 0.3
echo "${SCRIPT}" | /bin/sed -n '2p' | xargs /bin/sh >>/dev/null
sleep 0.3
echo "${SCRIPT}" | /bin/sed -n '3p' | xargs /bin/sh >>/dev/null
}
function main(){
proc_num
#proc_id
#if [ ${NUM} -eq 0 ] || [ ${PID} -eq 0 ];then
if [ ${NUM} -eq 0 ];then
shell
fi
}
main
創建定時任務：
*/2 * * * * /data/tools/script/daemon_Haproxy.sh > /dev/null 2>&1
2.3、在zabbix上添加監控
2.3.1、創建模板-->自動發現規則
定義宏變量：宏的值爲json導出後的值
2.3.2、添加item和trigger
Item1：檢查haproxy進程是否存在
Key：proc.num["{#SERVER}",,,"{#PATH}"]
Trigger1：對應Item1
{Template moba server:proc.num["{#SERVER}",,,"{#PATH}"].last()} < 1
Item2：檢查haproxy端口對應的TCP連接數
Key：haproxy.tcp.[{#PORT}] 這個key需要在haproxy機器上的zabbix_agent.conf手動配置UserParameter (前面有介紹)
Trigger2：對應Item2
{Template moba server:haproxy.tcp.[{#PORT}].last()}>40000
三、之後新增haproxy服務添加到監控的流程如下
1、在zabbixDB上的t_3party_server表裏添加數據：
INSERT INTO `zabbix`.`t_3party_server` (`app`, `server`, `division`, `node`, `port`, `remark`) VALUES ('MOBA', 'haproxy', 'haproxy', '10.111.152.15', '30032', '0');
2、在zabbix上 Host groups-->Haproxy_Group 組內添加IP #自定義(可不要)
3、在機器上添加配置參數：
vim /data/app/zabbix_agent/etc/zabbix_agentd.conf
LogRemoteCommands=1
LogFileSize=1
DebugLevel=3
#Haproxy
UserParameter=haproxy.tcp.[*],/usr/sbin/ss -ant sport eq :"$1" | awk 'NR>1{++s[$$1]} END {for(k in s){print k,s[k]}}' |grep -w "ESTAB" |awk '{print $$2}'
4、在機器上添加守護腳本：
/data/tools/script/daemon_Haproxy.sh
5、添加定時任務
*/2 * * * * /data/tools/script/daemon_Haproxy.sh > /dev/null 2>&1
6、把守護進程腳本放到 /data/app/zabbix_agent/share/zabbix/externalscripts/ 這個目錄內： #這一步主要是給zabbix上執行遠程命令使用的【Configuration--》Actions--》WeChat--》Operations--》--》2 Run remote commands on host groups: Haproxy_Group】 #自定義(可不要)
cp -ap /data/tools/script/daemon_Haproxy.sh /data/app/zabbix_agent/share/zabbix/externalscripts/
7、配置完畢後重啓 zabbix_agentd
pkill zabbix_agentd
/data/app/zabbix_agent/sbin/zabbix_agentd
8、到zabbix上監控haproxy服務是否被監控