1，簡介
1.1mha簡介
MHA，即MasterHigh Availability Manager and Tools for MySQL，是日本的一位MySQL專家採用Perl語言編寫的一個腳本管理工具，該工具僅適用於MySQLReplication（二層）環境，目的在於維持Master主庫的高可用性。
MHA(Master High Availability)是自動的master故障轉移和Slave提升的軟件包.它是基於標準的MySQL複製(異步/半同步).
MHA有兩部分組成:MHA Manager(管理節點)和MHA Node(數據節點).
MHA Manager可以單獨部署在一臺獨立機器上管理多個master-slave集羣,也可以部署在一臺slave上.MHA Manager探測集羣的node節點,當發現master出現故障的時候,它可以自動將具有最新數據的slave提升爲新的master,然後將所有其它的slave導向新的master上.整個故障轉移過程對應用程序是透明的。
MHA node運行在每臺MySQL服務器上(master/slave/manager),它通過監控具備解析和清理logs功能的腳本來加快故障轉移的。
1.2，mha特點
1. 10-30s實現master failover（9-12s可以檢測到主機故障，7-10s可以關閉主機避免SB，在用很短的時間應用差異日誌）
2. 部署簡單，無需對現有M-S結構做任何改動（至少3臺，保證切換後仍保持M-S結構）
3. 支持手動在線切換（主機硬件維護），downtime幾乎很短0.5-2s
4. 保證故障切換後多從庫數據的一致性
5. 完全自動化的failover及快速複製架構恢復方案（一主多從）
6. 恢復過程包括：選擇新主庫、確認從庫間relaylog差異、新主庫應用必要語句、其他從庫同步差異語句、重新建立複製連接
2，工作原理
相較於其它HA軟件，MHA的目的在於維持MySQL Replication中Master庫的高可用性，其最大特點是可以修復多個Slave之間的差異日誌，最終使所有Slave保持數據一致，然後從中選擇一個充當新的Master，並將其它Slave指向它。
-從宕機崩潰的master保存二進制日誌事件(binlogevents)。
-識別含有最新更新的slave。
-應用差異的中繼日誌(relay log)到其它slave。
-應用從master保存的二進制日誌事件(binlogevents)。
-提升一個slave爲新master。
-使其它的slave連接新的master進行復制。
3，mha工具包
(1)、 Manager工具:
- masterha_check_ssh : 檢查MHA的SSH配置。
- masterha_check_repl : 檢查MySQL複製。
- masterha_manager : 啓動MHA。
- masterha_check_status : 檢測當前MHA運行狀態。
- masterha_master_monitor : 監測master是否宕機。
- masterha_master_switch : 控制故障轉移(自動或手動)。
- masterha_conf_host : 添加或刪除配置的server信息。
(2)、 Node工具(這些工具通常由MHAManager的腳本觸發,無需人手操作)。
- save_binary_logs : 保存和複製master的二進制日誌。
- apply_diff_relay_logs : 識別差異的中繼日誌事件並應用於其它slave。
- filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用這個工具)。
- purge_relay_logs : 清除中繼日誌(不會阻塞SQL線程)。
4，主機部署
Ip地址 | mha狀況 | Mysql狀況 |
192.168.52.129 | Node | Master |
192.168.52.130 | Node | Slave |
192.168.52.131 | Manager | Slave |
5，生成ssh無密鑰證書
5.1 先在192.168.52.129上面生成密鑰
ssh-keygen -t dsa -P '' -f id_dsa
Id_dsa.pub爲公鑰，id_dsa爲私鑰，緊接着將公鑰文件複製成authorized_keys文件，這個步驟是必須的，過程如下：
cat id_dsa.pub >> authorized_keys
5.2 在192.168.52.130上面生產密鑰
ssh-keygen -t dsa -P '' -f id_dsa
cat id_dsa.pub >> authorized_keys
5.3 在192.168.52.131上面生產密鑰
ssh-keygen -t dsa -P '' -f id_dsa
cat id_dsa.pub >> authorized_keys
5.4 構造3個通用的authorized_keys
在192.168.52.129上面操作：
cd /root/.ssh #copy130和131上面的密鑰過來
scp 192.168.52.130:/root/.ssh/id_dsa.pub./id_dsa.pub.130
scp 192.168.52.131:/root/.ssh/id_dsa.pub./id_dsa.pub.131
cat id_dsa.pub.130 >> authorized_keys
cat id_dsa.pub.131 >> authorized_keys
查看生成的通用密鑰
[root@data01 .ssh]# cat authorized_keys
ssh-dssAAAAB3NzaC1kc3MAAACBAKe9oTz+hQ3vAUE+x7s2EIT6RSrlPoD2VHuSsDo+ZmAyv6+DD6/eVhVXrCyqzjQPJa6UI0PYjlPMk2r2wqdvC/YqQaLhFuQmsBQwrVA2xNHwhB3ic+Om44GVoiZFM7+bGAtfhQ9DLK2+sjfaa/oQfuDvTJ2SI/f0oG3wDGmokgdLAAAAFQC/O4R1yX1FxW7+dCKHhV+LQHWWHQAAAIADR5yqZGZCx8VB8Q6kAYg3cuUCCo3gF/hA5bHY+2xO5CcGAD1mq/l5v55QPUlGAH7btdmfip1tiaQ+V3N+Ektf2psM0uap/LKvbV2jQYKc2UZrpfF6R0lG+x9rpkxWMce1TJ4yANGOasjNjUV6Lg0RVDxLiMT4Ja4+edQVduYt2AAAAIBPNfJlyglZ5CwsI+v753tD8WT4GaH8ssNLpIKrH9qJU6OuT9MmniKE1RqZr+e5MCvkUAAXHFPq0DhHQlPKWqhIpVlu0E8Zsn9a5tv728JpIYz1umB5VLo2J5fhpobefD3AhjEHSyaubJgQG4Gu+jdwsR0H21xLNx0VoP8YPbYkAQ==root@data01
ssh-dssAAAAB3NzaC1kc3MAAACBAPtU+mTL9an88U1wfSwxdlHBg9n8eB9l218sXGLHwJyxNJN0pq4iPCLMfRxhM6T30HxUnyhghxBF2XvkAkMEjZ+IuCoA0mwRi1CcYSMh72SXqfRdgE2GpRBZDRRhlCIE5feNswWZdC7fIDmgNHtK5CFmJLcl+9Bkb3kfGqu8JOxbAAAAFQDo2YRMd5ZsfBRvPZcCYWcOsuy2oQAAAIEA4pGH2w7luN9JhJ0V6sDUUySg4R488yloKR/k4yD33qPXTejCiDUKUbanMLs6obQqxpbVvlhRT6cyo/le7PO6H8IzRHjFy65EPL0omn7J06PmHBUMqCn4jXo27EGXlRLavnonUf3tFeaVo7GxXerj71NdBKkQX7e/bgzD4d5v0PMAAACBAIhx1X50hlYzqPEZEDXTJmntLRXsMB20DvngvUcQLlgLozwfaNdJAQuYSGqklXYTquSmsyaTNJsxj8EkKG4XbM/7hHtNbV8KuAMJGT4Dm7jEdiKClyle2tIvVtonYaL41KeZVdn6Lk4lRYIFxpDoQHKXXr+XEFhLjoniT8urPISlroot@data02
ssh-dss AAAAB3NzaC1kc3MAAACBAJtC3j4Gq+zR7adyKFco/1hELblR65Af+Cgi81mfL+sJAFuOqPcFvDANhymBQ9ltH1N2/eDq1CrD0U9tRMxSwBvgiWZW9brkMpl5ix6oJSCBHdyqL6iuREk7CZ3V/y7P2V+TDCc+am5onMWDG1Af9o6CeA7CP92CHaUoltcXd7L7AAAAFQCqpeVqjbEs/lC/J1utfWrpGDxt8QAAAIB1aeB6C3S9t0dU3bEZs06DaooO46foBFMOI7u0w7uEWvj48UQF7W5Y++vjsiARxr6clk2g2T70n0fJmZCtMMiHqD6vyy8OOy8PzdUcQVAUW2GZQ8hn7M1U2GOz2KPo6uUbPiKkXilEfh9YRsyZyxMdmC4raPjPea8sj6favK8RbgAAAIAima6hWfBFsFcE1oh02xk7ydavHGJbHAlxeH9g9glE0FPmzPgWMFkYQnLsyV2z+ouMPFmERUPYzg1k/8Dhoxjp9j4JB6bIcPNtKdlS660NcFLxRtFAhrnzvLOTzXYzeIuZOlE0WcjeQGNpx8JHAef/3WzpHnVmnhbmlkBrZ8X/OQ==root@oraclem1
[root@data01 .ssh]#
看到authorized_keys文件裏面有3行記錄，分別代表了訪問data01（192.168.52.129），oraclem1（192.168.52.131），data02（192.168.52.130）的公用密鑰。然後把這個authorized_keys公鑰文件copy到oraclem1（192.168.52.131）和data02（192.168.52.130）上面同一個目錄下。Scp命令如下：
scp authorized_keys192.168.52.130:/root/.ssh/
scp authorized_keys192.168.52.131:/root/.ssh/
5.5 開始check驗證如下：
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr 7 02:40:40 2015 from data01
[root@data02 ~]# ssh 192.168.52.131
Last login: Tue Apr 7 02:40:56 2015 from 192.168.52.131
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr 7 02:41:11 2015 from 192.168.52.131
[root@data02 ~]# ssh 192.168.52.129
Last login: Tue Apr 7 02:40:01 2015 from 192.168.52.131
[root@data01 ~]# ssh 192.168.52.131
Last login: Tue Apr 7 02:41:18 2015 from 192.168.52.130
[root@oraclem1 ~]# ssh 192.168.52.129
Last login: Tue Apr 7 02:41:26 2015 from data02
[root@data01 ~]# ssh 192.168.52.131
Last login: Tue Apr 7 02:41:31 2015 from 192.168.52.129
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr 7 02:41:21 2015 from 192.168.52.131
[root@data02 ~]# ssh 192.168.52.129
Last login: Tue Apr 7 02:41:42 2015 from 192.168.52.131
[root@data01 ~]
OK，3臺服務器已經能實現兩兩互相ssh通了，不需要輸入密碼即可。
PS：如果不能實現任何兩臺主機互相之間可以無密碼登錄，後面的環節可能會有問題。
5.6 實現主機名hostname登錄
在3臺服務器上，編輯/etc/hosts,追加入以下內容保存退出
192.168.52.129 data01
192.168.52.130 data02
192.168.52.131 oraclem1
驗證主機名登錄：
[root@data02 ~]# ssh oraclem1
The authenticity of host 'oraclem1(192.168.52.131)' can't be established.
RSA key fingerprint ise5:f0:ae:e3:14:35:2f:09:1f:88:dd:31:c3:1a:e1:73.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added 'oraclem1' (RSA)to the list of known hosts.
Last login: Tue Apr 7 02:51:59 2015 from data01
[root@oraclem1 ~]# ssh data02
Last login: Tue Apr 7 02:52:10 2015 from data01
[root@data02 ~]# ssh oraclem1
Last login: Tue Apr 7 02:52:18 2015 from data02
[root@oraclem1 ~]# vim /etc/hosts
[root@oraclem1 ~]# ssh data01
Last login: Tue Apr 7 02:52:06 2015 from data02
[root@data01 ~]# ssh data02
Last login: Tue Apr 7 02:52:21 2015 from oraclem1
[root@data02 ~]# ssh data01
Last login: Tue Apr 7 02:55:13 2015 from oraclem1
[root@data01 ~]#
OK，看到可以實現通過主機名來無密碼登錄了。
6，準備好mysql主從環境
具體搭建過程，參考以前的blog：http://blog.csdn.net/mchdba/article/details/44734597
架構如下，一主二從的架構
Master主庫à192.168.52.129，slave從庫à192.168.52.130
Master 主庫à192.168.52.129，slave從庫à192.168.52.131
創建用戶mha管理的賬號，在所有mysql服務器上都需要執行：
GRANT SUPER,RELOAD,REPLICATIONCLIENT,SELECT ON *.* TO manager@'192.168.52.%' IDENTIFIED BY 'manager_1234';
GRANT CREATE,INSERT,UPDATE,DELETE,DROP ON*.* TO manager@'192.168.52.%';
創建主從賬號，在所有mysql服務器上都需要執行：
GRANT RELOAD, SUPER, REPLICATION SLAVE ON*.* TO 'repl'@'192.168.52.%' IDENTIFIED BY 'repl_1234';
7，開始安裝mha
mha包括manager節點和data節點，data節點包括原有的MySQL複製結構中的主機，至少3臺，即1主2從，當masterfailover後，還能保證主從結構；只需安裝node包。manager server：運行監控腳本，負責monitoring 和 auto-failover；需要安裝node包和manager包。
爲了節省機器，可以從現有複製架構中選一臺“閒置”從庫作爲manager server，比如：某臺從庫不對外提供讀的服務，只是作爲候選主庫，或是專門用於備份。
7.1，在數據節點上安裝mha
首先安裝yum -y install perl-DBD-MySQL
tar -zxvpf mha4mysql-node-0.56.tar.gz
perl Makefile.PL
make && make install
7.2 在管理節點上安裝mha
首先安裝perl的mysql包：
yum install -y perl-DBD-MySQL
yum install -y perl-Config-Tiny
yum install -y perl-Log-Dispatch
yum install -y perl-Parallel-ForkManager
yum install -y perl-Config-IniFiles
一些安裝失敗，可以直接下載rpm包安裝：
wgetftp://ftp.muug.mb.ca/mirror/centos/5.10/os/x86_64/CentOS/perl-5.8.8-41.el5.x86_64.rpm
wgetftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/compat-db43-4.3.29-15.el6.x86_64.rpm
wgethttp://downloads.naulinux.ru/pub/NauLinux/6x/i386/sites/School/RPMS/perl-Log-Dispatch-2.27-1.el6.noarch.rpm
wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Parallel-ForkManager-0.7.9-1.el6.noarch.rpm
wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Mail-Sender-0.8.16-3.el6.noarch.rpm wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Mail-Sendmail-0.79-12.el6.noarch.rpm
wget http://mirror.centos.org/centos/6/os/x86_64/Packages/perl-Time-HiRes-1.9721-136.el6.x86_64.rpm
下載完後，一個個rpm安裝好
如果最後還是安裝不好，可以嘗試一下perl CPAN的方式：
perl -MCPAN -e shell
cpan[1]> install Log::Dispatch
然後通過perlMakefile.PL檢查mha的perl安裝環境，如下所示：
[root@oraclem1 mha4mysql-manager-0.56]#perl Makefile.PL
*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.609)
- DBD::mysql ...loaded. (4.013)
- Time::HiRes ...loaded. (1.9726)
- Config::Tiny ...loaded. (2.12)
- Log::Dispatch ...loaded. (2.44)
- Parallel::ForkManager ...loaded. (0.7.9)
- MHA::NodeConst ...loaded. (0.56)
*** Module::AutoInstall configurationfinished.
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::manager
Writing MYMETA.yml and MYMETA.json
[root@oraclem1 mha4mysql-manager-0.56]#
然後解壓縮安裝：
tar -xvf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56
perl Makefile.PL
make && make install
7.3 編輯管理節點配置
在管理節點192.168.52.129上面
[root@data01 mha4mysql-manager-0.56]# vim/etc/masterha/app1.cnf
[server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/manager.log
ssh_user=root #ssh免密鑰登錄的帳號名
repl_user=repl #mysql複製帳號，用來在主從機之間同步二進制日誌等
repl_password=repl_1234
ping_interval=1 #ping間隔，用來檢測master是否正常
[server1]
hostname=192.168.52.129
candidate_master=1 #master機宕掉後,優先啓用這臺作爲新master
master_binlog_dir=/home/data/mysql/binlog/
[server2]
hostname=192.168.52.130
#candidate_master=1
master_binlog_dir=/home/data/mysql/binlog/
[server3]
hostname=192.168.52.131
#candidate_master=1
master_binlog_dir=/home/data/mysql/binlog/
#[server4]
#hostname=host4
#no_master=1
7.4 利用mha工具檢測ssh
然後check ssh成功，採用命令：masterha_check_ssh --conf=/etc/masterha/app1.cnf，如下所示：
1 | [root@data01 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf |
1 | Tue Apr 7 02:56:12 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. |
1 | Tue Apr 7 02:56:12 2015 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. |
1 | Tue Apr 7 02:56:12 2015 - [info] Reading server configuration from /etc/masterha/app1.cnf.. |
1 | Tue Apr 7 02:56:12 2015 - [info] Starting SSH connection tests.. |
1 | Tue Apr 7 02:56:15 2015 - [debug] |
1 | Tue Apr 7 02:56:12 2015 - [debug] Connecting via SSH from [email protected](192.168.52.129:22) to [email protected](192.168.52.130:22).. |
1 | Warning: Permanently added '192.168.52.129' (RSA) to the list of known hosts. |
1 | Tue Apr 7 02:56:13 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:13 2015 - [debug] Connecting via SSH from [email protected](192.168.52.129:22) to [email protected](192.168.52.131:22).. |
1 | Tue Apr 7 02:56:15 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:15 2015 - [debug] |
1 | Tue Apr 7 02:56:13 2015 - [debug] Connecting via SSH from [email protected](192.168.52.130:22) to [email protected](192.168.52.129:22).. |
1 | Tue Apr 7 02:56:14 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:14 2015 - [debug] Connecting via SSH from [email protected](192.168.52.130:22) to [email protected](192.168.52.131:22).. |
1 | Tue Apr 7 02:56:15 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:16 2015 - [debug] |
1 | Tue Apr 7 02:56:13 2015 - [debug] Connecting via SSH from [email protected](192.168.52.131:22) to [email protected](192.168.52.129:22).. |
1 | Tue Apr 7 02:56:15 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:15 2015 - [debug] Connecting via SSH from [email protected](192.168.52.131:22) to [email protected](192.168.52.130:22).. |
1 | Tue Apr 7 02:56:16 2015 - [debug] ok. |
1 | Tue Apr 7 02:56:16 2015 - [info] All SSH connection tests passed successfully. |
1 | [root@data01 ~]# |
7.5 使用mha工具check檢查repl環境
檢測命令爲：masterha_check_repl--conf=/etc/masterha/app1.cnf，檢測結果如下：
1 | [root@oraclem1 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf |
1 | Fri Apr 10 01:02:18 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. |
1 | Fri Apr 10 01:02:18 2015 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. |
1 | Fri Apr 10 01:02:18 2015 - [info] Reading server configuration from /etc/masterha/app1.cnf.. |
1 | Fri Apr 10 01:02:18 2015 - [info] MHA::MasterMonitor version 0.56. |
1 | Fri Apr 10 01:02:18 2015 - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.52.129(192.168.52.129:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] Master configurations are as below: |
1 | Master 192.168.52.130(192.168.52.130:3306), replicating from 192.168.52.129(192.168.52.129:3306), read-only |
1 | Master 192.168.52.129(192.168.52.129:3306), replicating from 192.168.52.130(192.168.52.130:3306) |
1
1 | Fri Apr 10 01:02:18 2015 - [info] GTID failover mode = 0 |
1 | Fri Apr 10 01:02:18 2015 - [info] Dead Servers: |
1 | Fri Apr 10 01:02:18 2015 - [info] Alive Servers: |
1 | Fri Apr 10 01:02:18 2015 - [info] 192.168.52.129(192.168.52.129:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] 192.168.52.130(192.168.52.130:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] 192.168.52.131(192.168.52.131:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] Alive Slaves: |
1 | Fri Apr 10 01:02:18 2015 - [info] 192.168.52.130(192.168.52.130:3306) Version=5.6.12-log (oldest major version between slaves) log-bin:enabled |
1 | Fri Apr 10 01:02:18 2015 - [info] Replicating from 192.168.52.129(192.168.52.129:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] 192.168.52.131(192.168.52.131:3306) Version=5.6.12-log (oldest major version between slaves) log-bin:enabled |
1 | Fri Apr 10 01:02:18 2015 - [info] Replicating from 192.168.52.129(192.168.52.129:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] Current Alive Master: 192.168.52.129(192.168.52.129:3306) |
1 | Fri Apr 10 01:02:18 2015 - [info] Checking slave configurations.. |
1 | Fri Apr 10 01:02:18 2015 - [info] Checking replication filtering settings.. |
1 | Fri Apr 10 01:02:18 2015 - [info] binlog_do_db= user_db, binlog_ignore_db= information_schema,mysql,performance_schema,test |
1 | Fri Apr 10 01:02:18 2015 - [info] Replication filtering check ok. |
1 | Fri Apr 10 01:02:18 2015 - [info] GTID (with auto-pos) is not supported |
1 | Fri Apr 10 01:02:18 2015 - [info] Starting SSH connection tests.. |
1 | Fri Apr 10 01:02:20 2015 - [info] All SSH connection tests passed successfully. |
1 | Fri Apr 10 01:02:20 2015 - [info] Checking MHA Node version.. |
1 | Fri Apr 10 01:02:23 2015 - [info] Version check ok. |
1 | Fri Apr 10 01:02:23 2015 - [info] Checking SSH publickey authentication settings on the current master.. |
1 | Fri Apr 10 01:02:23 2015 - [info] HealthCheck: SSH to 192.168.52.129 is reachable. |
1 | Fri Apr 10 01:02:23 2015 - [info] Master MHA Node version is 0.56. |
1 | Fri Apr 10 01:02:23 2015 - [info] Checking recovery script configurations on 192.168.52.129(192.168.52.129:3306).. |
1 | Fri Apr 10 01:02:23 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/data/mysql/binlog/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000183 |
1 | Fri Apr 10 01:02:23 2015 - [info] Connecting to [email protected](192.168.52.129:22).. |
1 | Creating /var/tmp if not exists.. ok. |
1 | Checking output directory is accessible or not.. |
1 | ok. |
1 | Binlog found at /home/data/mysql/binlog/, up to mysql-bin.000183 |
1 | Fri Apr 10 01:02:23 2015 - [info] Binlog setting check done. |
1 | Fri Apr 10 01:02:23 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. |
1 | Fri Apr 10 01:02:23 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.52.130 --slave_ip=192.168.52.130 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.12-log --manager_version=0.56 --relay_dir=/home/data/mysql/data --current_relay_log=mysqld-relay-bin.000011 --slave_pass=xxx |
1 | Fri Apr 10 01:02:23 2015 - [info] Connecting to [email protected](192.168.52.130:22).. |
1 | Checking slave recovery environment settings.. |
1 | Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.000013 |
1 | Temporary relay log file is /home/data/mysql/data/mysqld-relay-bin.000013 |
1 | Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. |
1 | done. |
1 | Testing mysqlbinlog output.. done. |
1 | Cleaning up test file(s).. done. |
1 | Fri Apr 10 01:02:24 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.52.131 --slave_ip=192.168.52.131 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.12-log --manager_version=0.56 --relay_log_info=/home/data/mysql/data/relay-log.info --relay_dir=/home/data/mysql/data/ --slave_pass=xxx |
1 | Fri Apr 10 01:02:24 2015 - [info] Connecting to [email protected](192.168.52.131:22).. |
1 | Checking slave recovery environment settings.. |
1 | Opening /home/data/mysql/data/relay-log.info ... ok. |
1 | Relay log found at /home/data/mysql/data, up to mysql-relay-bin.000023 |
1 | Temporary relay log file is /home/data/mysql/data/mysql-relay-bin.000023 |
1 | Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. |
1 | done. |
1 | Testing mysqlbinlog output.. done. |
1 | Cleaning up test file(s).. done. |
1 | Fri Apr 10 01:02:26 2015 - [info] Slaves settings check done. |
1 | Fri Apr 10 01:02:26 2015 - [info] |
1 | 192.168.52.129(192.168.52.129:3306) (current master) |
1 | +--192.168.52.130(192.168.52.130:3306) |
1 | +--192.168.52.131(192.168.52.131:3306) |
1
1 | Fri Apr 10 01:02:26 2015 - [info] Checking replication health on 192.168.52.130.. |
1 | Fri Apr 10 01:02:26 2015 - [info] ok. |
1 | Fri Apr 10 01:02:26 2015 - [info] Checking replication health on 192.168.52.131.. |
1 | Fri Apr 10 01:02:26 2015 - [info] ok. |
1 | Fri Apr 10 01:02:26 2015 - [warning] master_ip_failover_script is not defined. |
1 | Fri Apr 10 01:02:26 2015 - [warning] shutdown_script is not defined. |
1 | Fri Apr 10 01:02:26 2015 - [info] Got exit code 0 (Not master dead). |
1
1 | MySQL Replication Health is OK. |
1 | [root@oraclem1 ~]# |
8，管理mha操作
8.1 啓動manager
1 | nohup masterha_manager --conf=/etc/masterha/app1.cnf < /dev/null >/logs/mha/app1/manager.log 2>&1 & |
執行後臺情形，如下所示：
1 | [root@oraclem1 mha4mysql-manager-0.56]# nohup masterha_manager --conf=/etc/masterha/app1.cnf /logs/mha/app1/manager.log 2>&1 & |
1 | [1] 8973 |
1 | [root@oraclem1 mha4mysql-manager-0.56]# |
1 | [root@oraclem1 mha4mysql-manager-0.56]# tail -f /logs/mha/app1/manager.log |
1 | Fri Apr 10 02:46:43 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. |
1 | Fri Apr 10 02:46:43 2015 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. |
1 | Fri Apr 10 02:46:43 2015 - [info] Reading server configuration from /etc/masterha/app1.cnf.. |
8.2 使用masterha_check_status檢測下
[root@oraclem1 ~]# masterha_check_status--conf=/etc/masterha/app1.cnf
app1 (pid:8973) is running(0:PING_OK),master:192.168.52.129
[root@oraclem1 ~]#
手動操作：
8.3 停止manager
命令：masterha_stop --conf=/etc/masterha/app1.cnf
[root@oraclem1 mha4mysql-manager-0.56]#masterha_stop --conf=/etc/masterha/app1.cnf
Stopped app1 successfully.
[1]+ Exit 1 nohupmasterha_manager --conf=/etc/masterha/app1.cnf < /dev/null >/logs/mha/app1/manager.log 2>&1
[root@oraclem1 mha4mysql-manager-0.56]#
8.4 master死機自動切換測試
在mysql的master庫52.129上，執行如下命令：echo c> /proc/sysrq-trigger
後果是：然後會看到master庫變成了52.130，而52.131從庫也被迫去連接新的主庫52.130了。
Manager自動完成了切換操作。
9報錯記錄總結
報錯記錄1：
[root@data01 ~]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Tue Apr 7 22:31:06 2015 - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Tue Apr 7 22:31:07 2015 - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Tue Apr 7 22:31:07 2015 - [info] Reading server configuration from/etc/masterha/app1.cnf..
Tue Apr 7 22:31:07 2015 - [info] MHA::MasterMonitor version 0.56.
Tue Apr 7 22:31:07 2015 - [error][/usr/local/share/perl5/MHA/Server.pm,ln303] Getting relay log directory orcurrent relay logfile from replication table failed on192.168.52.130(192.168.52.130:3306)!
Tue Apr 7 22:31:07 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/ServerManager.pmline 315
Tue Apr 7 22:31:07 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Tue Apr 7 22:31:07 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
[root@data01 ~]#
解決辦法：在192.168.52.130上面，vim /etc/my.cnf，在裏面添加
relay-log=/home/data/mysql/binlog/mysql-relay-bin
然後重啓mysql，再去重新設置slave連接。
STOP SLAVE;
RESET SLAVE;
CHANGE MASTER TOMASTER_HOST='192.168.52.129',MASTER_USER='repl',MASTER_PASSWORD='repl_1234',MASTER_LOG_FILE='mysql-bin.000178',MASTER_LOG_POS=459;
START SLAVE;
Ok，搞定了。
報錯記錄2：
[root@data01 perl]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Thu Apr 9 00:54:32 2015 - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Thu Apr 9 00:54:32 2015 - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Thu Apr 9 00:54:32 2015 - [info] Reading server configuration from/etc/masterha/app1.cnf..
Thu Apr 9 00:54:32 2015 - [info] MHA::MasterMonitor version 0.56.
Thu Apr 9 00:54:32 2015 - [error][/usr/local/share/perl5/MHA/Server.pm,ln306] Getting relay log directory orcurrent relay logfile from replication table failed on 192.168.52.130(192.168.52.130:3306)!
Thu Apr 9 00:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at/usr/local/share/perl5/MHA/ServerManager.pm line 315
Thu Apr 9 00:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr 9 00:54:32 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
[root@data01 perl]#
解決方法：
/etc/masterha/app1.cnf文件裏面的參數配置，user和repl_user都是mysql賬號，需要創建好，這裏是只創建了repl_user而沒有創建好user賬號：
user=manager
password=manager_1234
repl_user=repl
repl_password=repl_1234
在mysql節點上，建立允許manager 訪問數據庫的“ manager manager ”賬戶，主要用於SHOW SLAVESTATUS,RESET SLAVE; 所以需要執行如下命令：
GRANT SUPER,RELOAD,REPLICATIONCLIENT,SELECT ON *.* TO manager@'192.168.52.%' IDENTIFIED BY 'manager_1234';
錯誤記錄3：
[root@oraclem1 ~]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Thu Apr 9 23:09:05 2015 - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Thu Apr 9 23:09:05 2015 - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Thu Apr 9 23:09:05 2015 - [info] Reading server configuration from/etc/masterha/app1.cnf..
Thu Apr 9 23:09:05 2015 - [info] MHA::MasterMonitor version 0.56.
Thu Apr 9 23:09:05 2015 - [error][/usr/local/share/perl5/MHA/ServerManager.pm,ln781] Multi-master configuration is detected, but two or more masters areeither writable (read-only is not set) or dead! Check configurations fordetails. Master configurations are as below:
Master 192.168.52.130(192.168.52.130:3306),replicating from 192.168.52.129(192.168.52.129:3306)
Master 192.168.52.129(192.168.52.129:3306),replicating from 192.168.52.130(192.168.52.130:3306)
Thu Apr 9 23:09:05 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at/usr/local/share/perl5/MHA/MasterMonitor.pm line 326
Thu Apr 9 23:09:05 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr 9 23:09:05 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
[root@oraclem1 ~]#
解決辦法：
mysql> set global read_only=1;
Query OK, 0 rows affected (0.00 sec)
mysql>
報錯記錄4：
Thu Apr 9 23:54:32 2015 - [info] Checking SSH publickey authentication andchecking recovery script configurations on all alive slave servers..
Thu Apr 9 23:54:32 2015 - [info] Executing command : apply_diff_relay_logs --command=test--slave_user='manager' --slave_host=192.168.52.130 --slave_ip=192.168.52.130--slave_port=3306 --workdir=/var/tmp --target_version=5.6.12-log--manager_version=0.56 --relay_dir=/home/data/mysql/data--current_relay_log=mysqld-relay-bin.000011 --slave_pass=xxx
Thu Apr 9 23:54:32 2015 - [info] Connecting to [email protected](192.168.52.130:22)..
Can't exec "mysqlbinlog": No suchfile or directory at /usr/local/share/perl5/MHA/BinlogManager.pm line 106.
mysqlbinlog version command failed with rc1:0, please verify PATH, LD_LIBRARY_PATH, and client options
at/usr/local/bin/apply_diff_relay_logs line 493
Thu Apr 9 23:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln205] Slaves settings check failed!
Thu Apr 9 23:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln413] Slave configuration failed.
Thu Apr 9 23:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/bin/masterha_check_repl line 48
Thu Apr 9 23:54:32 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr 9 23:54:32 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
[root@oraclem1 ~]#
解決辦法：
[root@data02 ~]# type mysqlbinlog
mysqlbinlog is/usr/local/mysql/bin/mysqlbinlog
[root@data02 ~]#
[root@data02 ~]# ln -s/usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
報錯記錄5：
Thu Apr 9 23:57:24 2015 - [info] Connecting to [email protected](192.168.52.130:22)..
Checking slave recovery environment settings..
Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.000013
Temporary relay log file is /home/data/mysql/data/mysqld-relay-bin.000013
Testing mysql connection and privileges..sh: mysql: command not found
mysql command failed with rc 127:0!
at/usr/local/bin/apply_diff_relay_logs line 375
main::check()called at /usr/local/bin/apply_diff_relay_logs line 497
eval{...} called at /usr/local/bin/apply_diff_relay_logs line 475
main::main()called at /usr/local/bin/apply_diff_relay_logs line 120
Thu Apr 9 23:57:24 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln205] Slaves settings check failed!
Thu Apr 9 23:57:24 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln413] Slave configuration failed.
Thu Apr 9 23:57:24 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/bin/masterha_check_repl line 48
Thu Apr 9 23:57:24 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr 9 23:57:24 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
解決辦法：
ln -s /usr/local/mysql/bin/mysql/usr/bin/mysql
報錯記錄6：
Fri Apr 10 00:58:36 2015 - [info] Executing command : apply_diff_relay_logs--command=test --slave_user='manager' --slave_host=192.168.52.130--slave_ip=192.168.52.130 --slave_port=3306 --workdir=/var/tmp--target_version=5.6.12-log --manager_version=0.56--relay_dir=/home/data/mysql/data--current_relay_log=mysqld-relay-bin.000011 --slave_pass=xxx
Fri Apr 10 00:58:36 2015 - [info] Connecting to [email protected](192.168.52.130:22)..
Checking slave recovery environment settings..
Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.000013
Temporary relay log file is/home/data/mysql/data/mysqld-relay-bin.000013
Testing mysql connection and privileges..Warning: Using a password onthe command line interface can be insecure.
ERROR 1142 (42000) at line 1: CREATEcommand denied to user 'manager'@'192.168.52.130' for table'apply_diff_relay_logs_test'
mysql command failed with rc 1:0!
at/usr/local/bin/apply_diff_relay_logs line 375
main::check()called at /usr/local/bin/apply_diff_relay_logs line 497
eval{...} called at /usr/local/bin/apply_diff_relay_logs line 475
main::main()called at /usr/local/bin/apply_diff_relay_logs line 120
Fri Apr 10 00:58:37 2015 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln205] Slaves settingscheck failed!
Fri Apr 10 00:58:37 2015 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln413] Slave configurationfailed.
Fri Apr 10 00:58:37 2015 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48
Fri Apr 10 00:58:37 2015 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened onmonitoring servers.
Fri Apr 10 00:58:37 2015 - [info] Got exitcode 1 (Not master dead).
MySQL Replication Health is NOT OK!
解決辦法：
執行如下授權語句sql：
GRANT CREATE,INSERT,UPDATE,DELETE,DROP ON*.* TO manager@'192.168.52.%';