環境配置：
zabbix服務版本：3.4.1
mysql當前版本：5.1.7
mysql新版本：5.7.26
zabbix服務器硬件配置：8核32G100G
mysql服務器當前硬件配置：4核16G100G
mysql新服務器硬件配置：4核32G100G
zabbix的監控主機數量有四百多臺，監控項數量有三萬多個，目前已經運行了快兩年時間，隨着多樣化的監控項和告警需求不斷的接入，發現前端頁面無響應，或者響應慢，zabbix繪圖中經常出現斷圖，一些item沒有數據，告警短信時延也逐步加劇，zabbix性能是越來越低，因此有必要對zabbix的整體性能做一次大的調整。
性能優化第一步：zabbix自身性能優化
zabbix內部組件進行監控，這是zabbix監控的第一步
配置-主機 找到zabbix服務器主機，添加Template APP Zabbix Server模板保存即可
需要關注的幾個內部性能監控圖表
Zabbix inernal process busy 監控圖各個監控項對應配置文件的參數和默認值如下，當某個進程負載過高時調整相應的值：
Zabbix busy timer processes,in % StartTimers=1 Zabbix busy eacalators processes,in % StartEscalators=1 Zabbix busy housekeeper processes,in % HousekeepingFrequency=1 , MaxHousekeeperDelete=5000 Zabbix busy alerter processes,in % StartAlerters=3 Zabbix busy configuration syncer processes,in % Zabbix busy history syncer processes,in % Zabbix busy self-monitoring processes,in % Zabbix busy task manager processes,in % Zabbix busy ipmi manager processes,in % Zabbix busy alter manager processes,in % Zabbix busy preprocessing manager processes,in % Zabbix busy preprocessing worker processes,in %
Zabbix data gathering process busy % 監控圖各個監控項對應配置文件的參數和默認值如下，當某個進程負載過高時調整相應的值：
Zabbix busy trapper processes, in % StartTrappers=5 Zabbix busy poller processes, in % StartPollers=5 Zabbix busy ipmi poller processes, in % StartIPMIPollers=0 Zabbix busy discoverer processes, in % StartDiscoverers=1 Zabbix busy icmp pinger processes, in % StartPingers=1 Zabbix busy http poller processes, in % StartHTTPPollers=1 Zabbix busy proxy poller processes, in % StartProxyPollers=1 Zabbix busy unreachable poller processes, in % StartPollersUnreachable=1 Zabbix busy java poller processes, in % StartJavaPollers=0 Zabbix busy snmp trapper processes, in % StartSNMPTrapper=0 Zabbix busy vmware collector processes, in % StartVMwareCollectors=0
Zabbix cache usage, % free 監控圖各個監控項對應配置文件的參數和默認值如下，當某個進程負載過高時調整相應的值：
Zabbix trend write cache, % free TrendCacheSize=4M Zabbix configuration cache, % free CacheSize=8M Zabbix text write cache, % free HistoryTextCacheSize=16M Zabbix history write cache, % free HistoryCacheSize=8M Zabbix value cache, % free ValueCacheSize=8M Zabbix vmware cache, % free VMwareCacheSize=8M
目前設備和監控規模的服務端配置參數
ListenPort=21004 LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=50 SocketDir=/tmp/zabbix DBName=zabbix DBUser=zabbix StartPollers=200 StartPollersUnreachable=20 StartTrappers=120 StartDiscoverers=10 StartHTTPPollers=20 StartAlerters=50 HousekeepingFrequency=12 MaxHousekeeperDelete=1000000 CacheSize=1024M StartDBSyncers=8 HistoryCacheSize=2048M HistoryIndexCacheSize=2048M TrendCacheSize=2048M ValueCacheSize=16G Timeout=10 AlertScriptsPath=/usr/local/zabbix-3.4.1/alertscripts LogSlowQueries=3000 LogFile=/var/log/zabbix/zabbix_server.log
監控項間隔檢查
select delay, count(*), concat(round(count(*) / (select count(*) from items where status=0) * 100,2),'%') as percent from items where status=0 group by delay order by 2 desc;
MYSQL數據庫優化
優化主要是兩個方面
1、MYSQL參數優化
[client] default-character-set = utf8 socket=/var/lib/mysql/mysql.sock [mysqld] max_connections = 2000 open_files_limit = 30000 table_open_cache = 20000 explicit_defaults_for_timestamp = true transaction-isolation = READ-COMMITTED max_allowed_packet = 128M innodb_buffer_pool_size = 20G innodb_log_file_size = 512M innodb_file_per_table = 1 innodb_log_buffer_size=4M innodb_thread_concurrency=64 innodb_flush_log_at_trx_commit=0 innodb_flush_method=O_DIRECT lower_case_table_names = 1 # collation-server = gbk_bin # # character-set-server = gbk default-storage-engine = INNODB join_buffer_size = 512M sort_buffer_size = 20M read_rnd_buffer_size = 20M log_timestamps = system collation-server = utf8_bin character-set-server = utf8 validate_password_policy = 0 validate_password_number_count = 0 validate_password_length = 4 validate_password_special_char_count = 0 default_password_lifetime = 0 innodb_read_io_threads = 16 innodb_write_io_threads = 16 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock skip-host-cache skip-name-resolve # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # Recommended in standard MySQL setup #sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES #sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION' sql_mode='STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION' #[mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
2、關閉zabbixhousekeeper清理歷史和趨勢記錄數據功能，zabbix幾個大表時間分區(history,history_log,history_str,history_text,history_unit,trends,trends_unit),
關閉housekeeper禁止自動定期清除歷史和趨勢記錄數據，轉而使用數據庫分區的方式配合存儲過程來定期清除歷史和趨勢記錄數據。
停止zabbix server進程，清空歷史和趨勢數據。我這裏是直接更換了一臺高版本mysql服務器，然後使用navicat同步除這七張表的zabbix所有數據表數據，再同步的這七張表的數據結構，然後在新數據庫上進行這七張數據表分區
[zabbix]> truncate table history; [zabbix]> truncate table history_log; [zabbix]> truncate table history_str; [zabbix]> truncate table history_text; [zabbix]> truncate table history_unit; [zabbix]> truncate table trends; [zabbix]> truncate table trends_unit; [zabbix]> optimize table history; [zabbix]> optimize table history_log; [zabbix]> optimize table history_str; [zabbix]> optimize table history_text; [zabbix]> optimize table history_unit; [zabbix]> optimize table trends; [zabbix]> optimize table trends_unit;
將官方的四個分散代碼拷貝至一個文件保存爲sql，導入數據庫；
https://www.zabbix.org/wiki/Docs/howto/mysql_partition#All_done
cat /root/zabbix-partition.sql
四個存儲過程如下：
DELIMITER $$
CREATE PROCEDURE `partition_create`(SCHEMANAME varchar(64), TABLENAME varchar(64), PARTITIONNAME varchar(64), CLOCK int)
BEGIN
/*
SCHEMANAME = The DB schema in which to make changes
TABLENAME = The table with partitions to potentially delete
PARTITIONNAME = The name of the partition to create
*/
/*
Verify that the partition does not already exist
*/
DECLARE RETROWS INT;
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_description >= CLOCK;
IF RETROWS = 0 THEN
/*
1. Print a message indicating that a partition was created.
2. Create the SQL to create the partition.
3. Execute the SQL from #2.
*/
SELECT CONCAT( "partition_create(", SCHEMANAME, ",", TABLENAME, ",", PARTITIONNAME, ",", CLOCK, ")" ) AS msg;
SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME, ' ADD PARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' );
PREPARE STMT FROM @sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_drop`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), DELETE_BELOW_PARTITION_DATE BIGINT)
BEGIN
/*
SCHEMANAME = The DB schema in which to make changes
TABLENAME = The table with partitions to potentially delete
DELETE_BELOW_PARTITION_DATE = Delete any partitions with names that are dates older than this one (yyyy-mm-dd)
*/
DECLARE done INT DEFAULT FALSE;
DECLARE drop_part_name VARCHAR(16);
/*
Get a list of all the partitions that are older than the date
in DELETE_BELOW_PARTITION_DATE. All partitions are prefixed with
a "p", so use SUBSTRING TO get rid of that character.
*/
DECLARE myCursor CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND CAST(SUBSTRING(partition_name FROM 2) AS UNSIGNED) < DELETE_BELOW_PARTITION_DATE;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
/*
Create the basics for when we need to drop the partition. Also, create
@drop_partitions to hold a comma-delimited list of all partitions that
should be deleted.
*/
SET @alter_header = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " DROP PARTITION ");
SET @drop_partitions = "";
/*
Start looping through all the partitions that are too old.
*/
OPEN myCursor;
read_loop: LOOP
FETCH myCursor INTO drop_part_name;
IF done THEN
LEAVE read_loop;
END IF;
SET @drop_partitions = IF(@drop_partitions = "", drop_part_name, CONCAT(@drop_partitions, ",", drop_part_name));
END LOOP;
IF @drop_partitions != "" THEN
/*
1. Build the SQL to drop all the necessary partitions.
2. Run the SQL to drop the partitions.
3. Print out the table partitions that were deleted.
*/
SET @full_sql = CONCAT(@alter_header, @drop_partitions, ";");
PREPARE STMT FROM @full_sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, @drop_partitions AS `partitions_deleted`;
ELSE
/*
No partitions are being deleted, so print out "N/A" (Not applicable) to indicate
that no changes were made.
*/
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, "N/A" AS `partitions_deleted`;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32), KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT)
BEGIN
DECLARE OLDER_THAN_PARTITION_DATE VARCHAR(16);
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE OLD_PARTITION_NAME VARCHAR(16);
DECLARE LESS_THAN_TIMESTAMP INT;
DECLARE CUR_TIME INT;
CALL partition_verify(SCHEMA_NAME, TABLE_NAME, HOURLY_INTERVAL);
SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00'));
SET @__interval = 1;
create_loop: LOOP
IF @__interval > CREATE_NEXT_INTERVALS THEN
LEAVE create_loop;
END IF;
SET LESS_THAN_TIMESTAMP = CUR_TIME + (HOURLY_INTERVAL * @__interval * 3600);
SET PARTITION_NAME = FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL * (@__interval - 1) * 3600, 'p%Y%m%d%H00');
IF(PARTITION_NAME != OLD_PARTITION_NAME) THEN
CALL partition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP);
END IF;
SET @__interval=@__interval+1;
SET OLD_PARTITION_NAME = PARTITION_NAME;
END LOOP;
SET OLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVAL KEEP_DATA_DAYS DAY), '%Y%m%d0000');
CALL partition_drop(SCHEMA_NAME, TABLE_NAME, OLDER_THAN_PARTITION_DATE);
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_verify`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), HOURLYINTERVAL INT(11))
BEGIN
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE RETROWS INT(11);
DECLARE FUTURE_TIMESTAMP TIMESTAMP;
/*
* Check if any partitions exist for the given SCHEMANAME.TABLENAME.
*/
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_name IS NULL;
/*
* If partitions do not exist, go ahead and partition the table
*/
IF RETROWS = 1 THEN
/*
* Take the current date at 00:00:00 and add HOURLYINTERVAL to it. This is the timestamp below which we will store values.
* We begin partitioning based on the beginning of a day. This is because we don't want to generate a random partition
* that won't necessarily fall in line with the desired partition naming (ie: if the hour interval is 24 hours, we could
* end up creating a partition now named "p201403270600" when all other partitions will be like "p201403280000").
*/
SET FUTURE_TIMESTAMP = TIMESTAMPADD(HOUR, HOURLYINTERVAL, CONCAT(CURDATE(), " ", '00:00:00'));
SET PARTITION_NAME = DATE_FORMAT(CURDATE(), 'p%Y%m%d%H00');
-- Create the partitioning query
SET @__PARTITION_SQL = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " PARTITION BY RANGE(`clock`)");
SET @__PARTITION_SQL = CONCAT(@__PARTITION_SQL, "(PARTITION ", PARTITION_NAME, " VALUES LESS THAN (", UNIX_TIMESTAMP(FUTURE_TIMESTAMP), "));");
-- Run the partitioning query
PREPARE STMT FROM @__PARTITION_SQL;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;[root@Zabbix-Server ~]# mysql -u zabbix -p zabbix Enter password: Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 48790 Server version: 5.5.52-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [zabbix]> use zabbix; Database changed MariaDB [zabbix]> source /root/zabbix-partition.sql; Query OK, 0 rows affected (0.04 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec)
分區維護存儲過程SQL
DELIMITER $$ CREATE PROCEDURE `partition_maintenance_all`(SCHEMA_NAME VARCHAR(32)) BEGIN CALL partition_maintenance(SCHEMA_NAME, 'history', 28, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'history_log', 28, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'history_str', 28, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'history_text', 28, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'history_uint', 28, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'trends', 730, 24, 14); CALL partition_maintenance(SCHEMA_NAME, 'trends_uint', 730, 24, 14); END$$ DELIMITER ;
以上代碼部分的含義爲(庫名,表名,保存多少天的數據,每隔多久生成一個分區,本次生成多少分區）
調用分區維護存儲過程SQL，將歷史和趨勢數據表做成表分區
mysql> source /root/partition_maintenance_all.sql;
Query OK, 0 rows affected (0.00 sec)
mysql> CALL partition_maintenance_all('zabbix');爲了讓時間分區不斷的創建出來，使用crontab任務調用分區維護存儲過程SQL來生成新的表分區
[root@HXQ-WLOMC-APP03 ~]# crontab -l
## zabbix partition stored procedures
0 3 * * 7 /usr/bin/mysql -uzabbix -p'SHipnet!23$' -e "use zabbix;" -e "CALL partition_maintenance_all('zabbix');"
[root@HXQ-WLOMC-APP03 ~]#參考鏈接