MHA(Master High Availability)目前在MySQL高可用方面是一個相對成熟的解決方案,它由日本DeNA公司的youshimaton(現就職於Facebook公司)開發,是一套優秀的作爲MySQL高可用性環境下故障切換和主從提升的高可用軟件。在MySQL故障切換過程中,MHA能做到在0~30秒之內自動完成數據庫的故障切換操作,並且在進行故障切換的過程中,MHA能在最大程度上保證數據的一致性,以達到真正意義上的高可用。
該軟件由兩部分組成:MHA Manager(管理節點)和MHA Node(數據節點)。
MHA Manager可以單獨部署在一臺獨立的機器上管理多個master-slave集羣,也可以部署在一臺slave節點上。
MHA Node運行在每臺MySQL服務器上,MHA Manager會定時探測集羣中的master節點,當master出現故障時,它可以自動將最新數據的slave提升爲新的master,然後將所有其他的slave重新指向新的master。整個故障轉移過程對應用程序完全透明。
在MHA自動故障切換過程中,MHA試圖從宕機的主服務器上保存二進制日誌,最大程度的保證數據的不丟失,但這並不總是可行的。例如,如果主服務器硬件故障或無法通過ssh訪問,MHA沒法保存二進制日誌,只進行故障轉移而丟失了最新的數據。使用MySQL 5.5的半同步複製,可以大大降低數據丟失的風險。MHA可以與半同步複製結合起來。如果只有一個slave已經收到了最新的二進制日誌,MHA可以將最新的二進制日誌應用於其他所有的slave服務器上,因此可以保證所有節點的數據一致性。
目前MHA主要支持一主多從的架構。要搭建MHA,要求一個複製集羣中必須最少有三臺數據庫服務器,一主二從,即一臺充當master,一臺充當備用master,另外一臺充當從庫。因爲至少需要三臺服務器,出於機器成本的考慮,淘寶也在該基礎上進行了改造,目前淘寶TMHA已經支持一主一從。
一、MHA工作原理、功能
1、MHA工作原理總結爲以下幾條:
- 從宕機崩潰的master保存二進制日誌事件(binlog events);
- 識別含有最新更新的slave;
- 應用差異的中繼日誌(relay log)到其他slave;
- 應用從master保存的二進制日誌事件(binlog events);
- 提升一個slave爲新master;
- 使用其他的slave連接新的master進行復制。
2、MHA提供瞭如下功能:
(1)master自動監控,故障轉移一體化(Automated master monitoring and failover)
(2)MHA可以在一個複製組中監控master的狀態,如果掛了,就可以自動的做failover。
(3)MHA通過所有slave的差異relay-log來保證數據的一致性。
(4)MHA在做故障轉移,日誌補償這些動作的時候,通常只需要10~30秒。
(5)通常情況下,MHA會選擇最新的slave作爲new master,但是你也可以指定哪些是候選maser,那麼新master選舉的時候,就從這些host裏面挑。
(6)導致複製環境中斷的一致性問題,在MHA中是不會發生的,請放心使用。
在MHA自動故障切換過程中,MHA試圖從宕機的主服務器上保存二進制日誌,最大程度的保證數據的不丟失,但這並不總是可行的。例如,如果主服務器硬件故障或無法通過ssh訪問,MHA沒法保存二進制日誌,只進行故障轉移而丟失了最新的數據。使用MySQL 5.5及以上版本的半同步複製,可以大大降低數據丟失的風險。MHA可以與半同步複製結合起來。如果只有一個slave已經收到了最新的二進制日誌,MHA可以將最新的二進制日誌應用於其他所有的slave服務器上,因此可以保證所有節點的數據一致性。
(7)手工-交互式master故障轉移(Interactive manually initiated Master Failover)
MHA可以配置成手工-交互式方式進行故障轉移,不支持監控master的狀態。
(8)非交互式master故障轉移 (Non-interactive master failover)
非交互式,自動的故障轉移,不提供監控master狀態功能,監控可以交給其他組件做(如:Pacemaker heartbeat)。
(9)在線master切換 (Online switching master to a different host)
二、MHA組件介紹
MHA組件介紹MHA軟件由兩部分組成,Manager工具包和Node工具包,具體的說明如下。
Manager工具包主要包括以下幾個工具:
(1)masterha_check_ssh #檢查MHA的SSH配置狀況;
(2)masterha_check_repl #檢查MySQL複製狀況;
(3)masterha_manger #啓動MHA;
(4)masterha_check_status #檢測當前MHA運行狀態;
(5)masterha_master_monitor #檢測master是否宕機;
(6)masterha_master_switch #控制故障轉移(自動或者手動);
(7)masterha_conf_host #添加或刪除配置的server信息;
Node工具包(這些工具通常由MHA Manager的腳本觸發,無需人爲操作)主要包括以下幾個工具:
(1)save_binary_logs #保存和複製master的二進制日誌;
(2)apply_diff_relay_logs #識別差異的中繼日誌事件並將其差異的事件應用於其他的slave;
(3)purge_relay_logs #清除中繼日誌(不會阻塞SQL線程);
注意: 爲了儘可能的減少主庫硬件損壞宕機造成的數據丟失,因此在配置MHA的同時建議配置成MySQL半同步複製
環境:
- server1:172.25.70.1
- server2:172.25.70.2
- server3:172.25.70.3
三、搭建基於MySQL的MHA
1.關閉server1-3的mysqld服務;修改server1、server2、server3的msyqld配置文件
- server1-3:
systemctl stop mysqld
- server1:
vim /etc/my.cnf
=================
29 server_id=1
30 gtid_mode=ON
31 enforce_gtid_consistency=ON
32 log_slave_updates=ON
33 log_bin=binlog
- server2:
vim /etc/my.cnf
===================
29 server-id=2
30 gtid_mode=ON
31 enforce_gtid_consistency=ON
32 log_slave_updates=ON
33 log_bin=binlog
- server3:
vim /etc/my.cnf
====================
29 server-id=3
30 gtid_mode=ON
31 enforce_gtid_consistency=ON
32 log_slave_updates=ON
33 log_bin=binlog
2.初始化server3的mysqld服務
[root@server3 ~]# cd /var/lib/mysql
[root@server3 mysql]# ls
auto.cnf binlog.index ib_buffer_pool performance_schema sys
binlog.000001 ca-key.pem ibdata1 private_key.pem
binlog.000002 ca.pem ib_logfile0 public_key.pem
binlog.000003 client-cert.pem ib_logfile1 server-cert.pem
binlog.000004 client-key.pem mysql server-key.pem
[root@server3 mysql]# rm -fr *
[root@server3 mysql]# systemctl start mysqld
[root@server3 mysql]# cat /var/log/mysqld.log | grep password
==============================================================================
2019-05-03T00:04:13.360192Z 1 [Note] A temporary password is generated for root@localhost: (kkw=D?1,fu2
2019-05-03T01:11:12.145588Z 0 [Note] Shutting down plugin 'validate_password'
2019-05-03T01:11:13.674624Z 0 [Note] Shutting down plugin 'sha256_password'
2019-05-03T01:11:13.674629Z 0 [Note] Shutting down plugin 'mysql_native_passwor'
2019-05-03T03:01:23.405961Z 0 [Note] Shutting down plugin 'validate_password'
2019-05-03T03:01:24.718329Z 0 [Note] Shutting down plugin 'sha256_password'
2019-05-03T03:01:24.718331Z 0 [Note] Shutting down plugin 'mysql_native_passwor'
2019-05-03T03:08:43.045997Z 1 [Note] A temporary password is generated for root@localhost: MkT>18/5CK=)
==============================================================================
[root@server3 mysql]# mysql -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.24-log
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> alter user root@localhost identified by 'Mahao+123';
Query OK, 0 rows affected (0.01 sec)
mysql> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
3.爲server3設置複製用戶,並開啓server3的slave線程
mysql> change master to master_host='172.25.13.1',master_user='rep',master_password='Mahao+123',master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.04 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.25.13.1
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: binlog.000006
Read_Master_Log_Pos: 526
Relay_Log_File: server3-relay-bin.000008
Relay_Log_Pos: 485
Relay_Master_Log_File: binlog.000006
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
#測試
4.主庫寫入信息
- server1:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.01 sec)
mysql> drop database test
-> ;
Query OK, 1 row affected (0.02 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql> CREATE DATABASE taylor;
Query OK, 1 row affected (0.00 sec)
mysql> use taylor;
Database changed
mysql> create table usertb (
-> username varchar(10) not null,
-> password varchar(15) not null);
Query OK, 0 rows affected (0.02 sec)
mysql> desc usertb;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| username | varchar(10) | NO | | NULL | |
| password | varchar(15) | NO | | NULL | |
+----------+-------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
mysql> insert into usertb values ('user1','123');
Query OK, 1 row affected (0.00 sec)
mysql> select * from usertb;
+----------+----------+
| username | password |
+----------+----------+
| user1 | 123 |
+----------+----------+
1 row in set (0.00 sec)
5.從庫查看
- server3:
[root@server3 mysql]# mysql -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.7.24-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| taylor |
+--------------------+
5 rows in set (0.00 sec)
mysql> use taylor
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from usertb;
+----------+----------+
| username | password |
+----------+----------+
| user1 | 123 |
+----------+----------+
1 row in set (0.00 sec)
- server1:
mysql> SHOW GLOBAL VARIABLES LIKE '%gtid%';
+----------------------------------+------------------------------------------------------------------------------------+
| Variable_name | Value |
+----------------------------------+------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| gtid_executed | 742e05c2-6cad-11e9-a66b-52540075b0d5:1-8,
f32c0a71-6c6f-11e9-8a9f-52540075b0d5:1-7 |
| gtid_executed_compression_period | 1000 |
| gtid_mode | ON |
| gtid_owned | |
| gtid_purged | |
| session_track_gtids | OFF |
+----------------------------------+------------------------------------------------------------------------------------+
8 rows in set (0.00 sec)
mysqldump --all-databases --single-transaction --triggers --routines --events --host=127.0.0.1 --port=3306 --user=root --password=Mahao+123 > Mahao.sql
- server4:
安裝MHA
[root@server4 ~]# ls
MHA-7
[root@server4 ~]# cd
.config/ .local/ MHA-7/ .ssh/
[root@server4 ~]# cd MHA-7/
[root@server4 MHA-7]# ls
mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
mha4mysql-manager-0.58.tar.gz
mha4mysql-node-0.58-0.el7.centos.noarch.rpm
perl-Config-Tiny-2.14-7.el7.noarch.rpm
perl-Email-Date-Format-1.002-15.el7.noarch.rpm
perl-Log-Dispatch-2.41-1.el7.1.noarch.rpm
perl-Mail-Sender-0.8.23-1.el7.noarch.rpm
perl-Mail-Sendmail-0.79-21.el7.noarch.rpm
perl-MIME-Lite-3.030-1.el7.noarch.rpm
perl-MIME-Types-1.38-2.el7.noarch.rpm
perl-Parallel-ForkManager-1.18-2.el7.noarch.rpm
1.先安裝依賴
[root@server4 MHA-7]# yum install -y perl*
2.安裝MHA
[root@server4 MHA-7]# yum install -y mha4mysql-manager-0.58-0.el7.centos.noarch.rpm mha4mysql-node-0.58-0.el7.centos.noarch.rpm
3.生成ssh驗證密鑰
[root@server4 MHA-7]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
ea:fd:44:c4:e1:41:20:10:d4:d7:b5:d4:ba:f5:08:69 root@server4
The key's randomart image is:
+--[ RSA 2048]----+
| .++. .++.o. |
| ...o.+ .. |
| . + .o |
| . E . |
| S .. + o |
| . . . . .|
| . . |
| . . . |
| . ... |
+-----------------+
4.發送公鑰到server1—server3
ssh-copy-id server1
ssh-copy-id server2
ssh-copy-id server3
#驗證
ssh server1
ssh server2
ssh server3
5.給server1-3發送mha4mysql-node-0.58-0.el7.centos.noarch.rpm,並安裝
[root@server1 ~]# yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm
[root@server2 ~]# yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm
[root@server3 ~]# yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm
6.server4創建目錄,並編寫配置文件
[root@server4 MHA-7]# mkdir -p /etc/masterha
[root@server4 MHA-7]# cd /etc/masterha/
[root@server4 masterha]# ls
[root@server4 masterha]#
[root@server4 masterha]# vim app1.cnf
=============================================
[server default]
manager_workdir=/etc/masterha
manager_log=/var/log/masterha.log
master_binlog_dir=/etc/masterha
#master_ip_failover_script= /usr/local/bin/master_ip_failover
#master_ip_online_change_script= /usr/local/bin/master_ip_online_change
password=Mahao+123
user=root
ping_interval=1
remote_workdir=/tmp
repl_password=Mahao+123
repl_user=repl
#report_script=/usr/local/send_report
#secondary_check_script= /usr/local/bin/masterha_secondary_check -s server03 -s server02
#shutdown_script=""
ssh_user=root
[server1]
hostname=172.25.13.1
port=3306
[server2]
hostname=172.25.13.2
port=3306
candidate_master=1
check_repl_delay=0
[server3]
hostname=172.25.13.3
port=3306
no_master=1
7.ssh驗證
[root@server4 masterha]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Fri May 3 14:27:19 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri May 3 14:27:19 2019 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri May 3 14:27:19 2019 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri May 3 14:27:19 2019 - [info] Starting SSH connection tests..
Fri May 3 14:27:19 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln63]
Fri May 3 14:27:19 2019 - [debug] Connecting via SSH from [email protected](172.25.13.1:22) to [email protected](172.25.13.2:22)..
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Fri May 3 14:27:19 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln111] SSH connection from [email protected](172.25.13.1:22) to [email protected](172.25.13.2:22) failed!
Fri May 3 14:27:20 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln63]
Fri May 3 14:27:19 2019 - [debug] Connecting via SSH from [email protected](172.25.13.2:22) to [email protected](172.25.13.1:22)..
Warning: Permanently added '172.25.13.1' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Fri May 3 14:27:19 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln111] SSH connection from [email protected](172.25.13.2:22) to [email protected](172.25.13.1:22) failed!
Fri May 3 14:27:20 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln63]
Fri May 3 14:27:20 2019 - [debug] Connecting via SSH from [email protected](172.25.13.3:22) to [email protected](172.25.13.1:22)..
Warning: Permanently added '172.25.13.1' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Fri May 3 14:27:20 2019 - [error][/usr/share/perl5/vendor_perl/MHA/SSHCheck.pm, ln111] SSH connection from [email protected](172.25.13.3:22) to [email protected](172.25.13.1:22) failed!
SSH Configuration Check Failed!
at /usr/bin/masterha_check_ssh line 44.
8.給server1-3發送/root/.ssh/目錄
[root@server4 masterha]# scp -r ~/.ssh server1:
known_hosts 100% 718 0.7KB/s 00:00
authorized_keys 100% 831 0.8KB/s 00:00
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 394 0.4KB/s 00:00
[root@server4 masterha]# scp -r ~/.ssh server2:
known_hosts 100% 718 0.7KB/s 00:00
authorized_keys 100% 831 0.8KB/s 00:00
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 394 0.4KB/s 00:00
[root@server4 masterha]# scp -r ~/.ssh server3:
known_hosts 100% 718 0.7KB/s 00:00
authorized_keys 100% 831 0.8KB/s 00:00
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub
=====================================================================================
#自己特殊設定的解決方式
[root@server4 masterha]# cd ~/.ssh/
[root@server4 .ssh]# ls
authorized_keys id_rsa id_rsa.pub known_hosts
[root@server4 .ssh]# scp id_rsa id_rsa.pub known_hosts server1:~/.ssh/
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 394 0.4KB/s 00:00
known_hosts 100% 718 0.7KB/s 00:00
[root@server4 .ssh]# scp id_rsa id_rsa.pub known_hosts server2:~/.ssh/
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 394 0.4KB/s 00:00
known_hosts 100% 718 0.7KB/s 00:00
[root@server4 .ssh]# scp id_rsa id_rsa.pub known_hosts server3:~/.ssh/
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 394 0.4KB/s 00:00
known_hosts 100% 718 0.7KB/s 00:00
[root@server4 .ssh]#
======================================================================================
再次檢測ssh
[root@server4 .ssh]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Fri May 3 14:40:44 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri May 3 14:40:44 2019 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri May 3 14:40:44 2019 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri May 3 14:40:44 2019 - [info] Starting SSH connection tests..
Fri May 3 14:40:44 2019 - [debug]
Fri May 3 14:40:44 2019 - [debug] Connecting via SSH from [email protected](172.25.13.1:22) to [email protected](172.25.13.2:22)..
Fri May 3 14:40:44 2019 - [debug] ok.
Fri May 3 14:40:44 2019 - [debug] Connecting via SSH from [email protected](172.25.13.1:22) to [email protected](172.25.13.3:22)..
Fri May 3 14:40:44 2019 - [debug] ok.
Fri May 3 14:40:45 2019 - [debug]
Fri May 3 14:40:44 2019 - [debug] Connecting via SSH from [email protected](172.25.13.2:22) to [email protected](172.25.13.1:22)..
Fri May 3 14:40:44 2019 - [debug] ok.
Fri May 3 14:40:44 2019 - [debug] Connecting via SSH from [email protected](172.25.13.2:22) to [email protected](172.25.13.3:22)..
Fri May 3 14:40:44 2019 - [debug] ok.
Fri May 3 14:40:45 2019 - [debug]
Fri May 3 14:40:45 2019 - [debug] Connecting via SSH from [email protected](172.25.13.3:22) to [email protected](172.25.13.1:22)..
Fri May 3 14:40:45 2019 - [debug] ok.
Fri May 3 14:40:45 2019 - [debug] Connecting via SSH from [email protected](172.25.13.3:22) to [email protected](172.25.13.2:22)..
Fri May 3 14:40:45 2019 - [debug] ok.
Fri May 3 14:40:45 2019 - [info] All SSH connection tests passed successfully.
[root@server4 .ssh]#
檢測repl
[root@server4 .ssh]# masterha_check_repl --conf=/etc/masterha/app1.cnf
==========================================================================
Fri May 3 14:43:04 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri May 3 14:43:04 2019 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri May 3 14:43:04 2019 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri May 3 14:43:04 2019 - [info] MHA::MasterMonitor version 0.58.
Fri May 3 14:43:05 2019 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln188] There is no alive server. We can't do failover
Fri May 3 14:43:05 2019 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 329.
Fri May 3 14:43:05 2019 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Fri May 3 14:43:05 2019 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
#有報錯
server1提供授權
- server1:
[root@server1 ~]# systemctl start mysqld
[root@server1 ~]# mysql -p
=============================================================
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.24-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all on *.* to root@'%' identified by 'Mahao+123';
Query OK, 0 rows affected, 1 warning (0.00 sec)
#再次檢測