一、mysql架构描述
主库master：192.168.66.202 port:3306
从库slave1:192.168.66.203 port:3306
从库slave2:192.168.66.204 port:3306
VIP: 192.168.66.235
mariadb 10.1.18
centos 6.6
一主二从+MHA
二、问题描述
使用MHA的复制检测脚本执行时，报如下提示：有一个从库IO线程没有运行，具体如下:
..........................................................................
Wed Feb 22 17:28:00 2017 - [info] Checking replication health on 192.168.66.203..
Wed Feb 22 17:28:01 2017 - [info] ok.
Wed Feb 22 17:28:01 2017 - [info] Checking replication health on 192.168.66.204..
Wed Feb 22 17:28:01 2017 - [error][/usr/local/share/perl5/MHA/Server.pm, ln485] Slave IO thread is not running on 192.168.66.204(192.168.66.204:3306)
Wed Feb 22 17:28:01 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln1526] failed!
Wed Feb 22 17:28:01 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at
/usr/local/share/perl5/MHA/MasterMonitor.pm line 417
Wed Feb 22 17:28:01 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.
Wed Feb 22 17:28:01 2017 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
三、原因分析
在slave1上查看进程和slave状态：
pager cat | egrep -i 'system user|Exec_Master_Log_Pos|Seconds_Behind_Master|Read_Master_Log_Pos|Master_Log_File|Relay_Master_Log_File|Slave_IO_Running|
Slave_SQL_Running|Master_Host|Master_User|Master_Port'
MariaDB [(none)]> show processlist; show slave status\G
| 2732923 | system user | | NULL | Connect | 16365 | Queueing master event to the relay log | NULL |
0.000 |
| 2732924 | system user | | NULL | Connect | 0 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL |
0.000 |
3 rows in set (0.00 sec)
Master_Host: 192.168.66.202
Master_User: rep
Master_Port: 3306
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 2206
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 2206
Seconds_Behind_Master: 0
1 row in set (1.04 sec)
在slave2上查看进程和slave状态：
pager cat | egrep -i 'system user|Exec_Master_Log_Pos|Seconds_Behind_Master|Read_Master_Log_Pos|Master_Log_File|Relay_Master_Log_File|Slave_IO_Running|
Slave_SQL_Running|Master_Host|Master_User|Master_Port'
MariaDB [(none)]> show processlist; show slave status\G
| 2734714 | system user | | NULL | Connect | 16413 | Queueing master event to the relay log | NULL |
0.000 |
| 2734715 | system user | | NULL | Connect | 0 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL |
0.000 |
3 rows in set (0.00 sec)
Master_Host: 192.168.66.202
Master_User: rep
Master_Port: 3306
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 2206
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 2206
Seconds_Behind_Master: 16415
1 row in set (1.05 sec)
上面看到从库的IO线程有在运行,且Master_Log_File=Relay_Master_Log_File和Read_Master_Log_Pos=Exec_Master_Log_Pos ,为什么会提示：192.168.66.204的IO线程没有运行呢？
上面还发现IO thread长时间一直处于:Queueing master event to the relay log,非常奇怪。
在master上查看slave数量：
MariaDB [(none)]> show slave hosts;
+-----------+------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+------+------+-----------+
| 2 | | 3306 | 1 |
+-----------+------+------+-----------+
1 row in set (0.00 sec)
正常情况会显示两个从库，而上面查到master上只有一个从库，其server id为2,在slave1从库上查看对应server_id为2：
MariaDB [(none)]> show global variables like 'server_id';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id | 2 |
+---------------+-------+
1 row in set (0.00 sec)
而第二个从库slave2查到servier_id也为2：
MariaDB [(none)]> show global variables like 'server_id';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id | 2 |
+---------------+-------+
1 row in set (0.00 sec)
我们再看error log:
slave1的错误日志：
2017-02-22 22:45:13 47275402185472 [Note] Slave: received end packet from server, apparent master shutdown:
2017-02-22 22:45:13 47275402185472 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.000002' at position 2206
slave2的错误日志：
2017-02-22 22:44:26 47107924593408 [Note] Slave: received end packet from server, apparent master shutdown:
2017-02-22 22:44:26 47107924593408 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.000002' at position 2206
上面报错显示IO线程读取主库的event失败,正在重新连接，这也说明了为什么slave的IO线程状态长时间一直处于Queueing master event to the relay log。问题发生的原因很简单了，那就是两个从库的server id都为2引起，mysql主从的server id一定不能相同,一般建议用ip最后一段+端口号，如IP为192.168.66.203，端口号为3306， 则server id为2033306,这样就能保证一个主从复制架构中每台mysql 实例的server id号不会相同。
server-id做什么用的呢？
1、 mysql的同步的数据中是包含server-id的，用于标识该语句最初是从哪个server写入的，所以server-id一定要有的;
2、 每一个同步中的slave在master上都对应一个master线程，该线程就是通过slave的server-id来标识的；每个slave在master端最多有一个master线程，如果两个slave的server-id 相
同，则后一个连接成功时，前一个将被踢掉。 这里至少有这么一种考虑：
slave主动连接master之后，如果slave上面执行了slave stop；则连接断开，但是master上对应的线程并没有退出；当slave start之后，master不能再创建一个线程而保留原来的线程，那
样同步就可能有问题.
四、问题解决：
将主库和从库的server id分别改为:
master--->2023306
slave1--->2033306
slave2--->2043306
主库：
MariaDB [(none)]> set global server_id =2023306
-> ;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show global variables like 'server_id';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| server_id | 2023306 |
+---------------+---------+
从库1：
set global server_id =2033306
MariaDB [(none)]> show global variables like 'server_id';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| server_id | 2033306 |
+---------------+---------+
从库2：
set global server_id =2043306
MariaDB [(none)]> show global variables like 'server_id';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| server_id | 2043306 |
+---------------+---------+
为防止重启mysql后，server_id改回原来设置，所以同时需要在my.cnf文件中作相应修改。两个从库都重启slave:
MariaDB [(none)]> stop slave;
MariaDB [(none)]> start slave;
slave1:
MariaDB [(none)]> show processlist; show slave status\G
| 2747608 | system user | | NULL | Connect | 204 | Waiting for master to send event | NULL | 0.000
|
| 2747609 | system user | | NULL | Connect | 202 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL | 0.000
|
3 rows in set (0.00 sec)
Master_Host: 192.168.66.202
Master_User: rep
Master_Port: 3306
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 2206
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 2206
Seconds_Behind_Master: 0
slave2:
MariaDB [(none)]> show processlist; show slave status\G
| 2749407 | system user | | NULL | Connect | 131 | Waiting for master to send event | NULL | 0.000
|
| 2749408 | system user | | NULL | Connect | 128 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL | 0.000
|
3 rows in set (0.00 sec)
Master_Host: 192.168.66.202
Master_User: rep
Master_Port: 3306
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 2206
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 2206
Seconds_Behind_Master: 0
1 row in set (0.00 sec)
现在从库IO线程状态显示Waiting for master to send event，这个才是正常状态。
master上现在可以看到两个从库有连接和注册上。
MariaDB [(none)]> show slave hosts;
+-----------+------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+------+------+-----------+
| 2033306 | | 3306 | 2023306 |
| 2043306 | | 3306 | 2023306 |
+-----------+------+------+-----------+
2 rows in set (0.00 sec)
最后，我们再用mha脚本检测一下复制是否正常:
masterha_check_repl --global_conf=/apps/conf/mha/masterha_base.cnf --conf=/apps/conf/mha/app1.cnf
......
192.168.66.202(192.168.66.202:3306) (current master)
+--192.168.66.203(192.168.66.203:3306)
+--192.168.66.204(192.168.66.204:3306)
Wed Feb 22 23:20:13 2017 - [info] Checking replication health on 192.168.66.203..
Wed Feb 22 23:20:13 2017 - [info] ok.
Wed Feb 22 23:20:13 2017 - [info] Checking replication health on 192.168.66.204..
Wed Feb 22 23:20:13 2017 - [info] ok.
Wed Feb 22 23:20:13 2017 - [info] Checking master_ip_failover_script status:
Wed Feb 22 23:20:13 2017 - [info] /apps/sh/mha/script/master_ip_failover --command=status --ssh_user=apps --orig_master_host=192.168.66.202 --
orig_master_ip=192.168.66.202 --orig_master_port=3306
Wed Feb 22 23:20:13 2017 - [info] OK.
Wed Feb 22 23:20:13 2017 - [warning] shutdown_script is not defined.
Wed Feb 22 23:20:13 2017 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
