今天其中一臺遊戲服務器的數據庫mysql master當機， 系統變爲只讀模式，重啓後進入安全模式，執行fsck後恢復正常。服務器起來之後mysql啓動正常，但一臺slave卻一直出現同步錯誤。
登錄後查看，發現以下錯誤：
mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Master_Host: 10.90.13.238 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000949 Read_Master_Log_Pos: 277562491 Relay_Log_File: mysql-relay-bin.001616 Relay_Log_Pos: 277562637 Relay_Master_Log_File: mysql-bin.000949 Slave_IO_Running: No Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 1 Exec_Master_Log_Pos: 277562491 Relay_Log_Space: 277562836 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1236 Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position; the first event 'mysql-bin.000949' at 277562491, the last event read from './mysql-bin.000949' at 4, the last byte read from './mysql-bin.000949' at 4.' Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 4 1 row in set (0.00 sec)
錯誤爲：
Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position; the first event 'mysql-bin.000949' at 277562491, the last event read from './mysql-bin.000949' at 4, the last byte read from './mysql-bin.000949' at 4.'
這個錯誤之前也遇到過，但沒有具體記錄下來，於是網上找資料。
參考了這幾個資料：
http://blog.ez2learn.com/2011/11/28/mysql-1236-solution/
http://blog.longwin.com.tw/2013/09/mysql-replication-error-1236-fix-2013/
出現這樣的錯誤原因很簡單，原本的slave在master當機前一直在執行同步的動作，當master當機重啓mysql恢復之後，會重新開一個新的binlog繼續寫，但slave不知道發生了這件事，所以還在問上次同步的那個binlog文件和讀到得那個位置。
要確定這個情況，我執行了如下的操作：
1. 檢查master的位置
mysql> show master status\G *************************** 1. row *************************** File: mysql-bin.000950 Position: 336492640 Binlog_Do_DB: Binlog_Ignore_DB: 1 row in set (0.00 sec) mysql> show master status;
2. 檢查master上binlog的大小和最新的修改時間：
[root@d1 ~]# ll /data/mysql/mysql-bin.* -rw-rw---- 1 mysql mysql 1073742473 Nov 17 10:38 /data/mysql/mysql-bin.000944 -rw-rw---- 1 mysql mysql 1073742022 Nov 18 12:44 /data/mysql/mysql-bin.000945 -rw-rw---- 1 mysql mysql 1073745576 Nov 19 15:31 /data/mysql/mysql-bin.000946 -rw-rw---- 1 mysql mysql 1073745324 Nov 21 05:03 /data/mysql/mysql-bin.000947 -rw-rw---- 1 mysql mysql 1073742027 Nov 22 16:09 /data/mysql/mysql-bin.000948 -rw-rw---- 1 mysql mysql 277553623 Nov 23 05:07 /data/mysql/mysql-bin.000949 -rw-rw---- 1 mysql mysql 337157571 Nov 23 18:04 /data/mysql/mysql-bin.000950 -rw-rw---- 1 mysql mysql 133 Nov 23 08:06 /data/mysql/mysql-bin.index
[root@d1 ~]# du /data/mysql/mysql-bin.* -sh 1.1G /data/mysql/mysql-bin.000944 1.1G /data/mysql/mysql-bin.000945 1.1G /data/mysql/mysql-bin.000946 1.1G /data/mysql/mysql-bin.000947 1.1G /data/mysql/mysql-bin.000948 265M /data/mysql/mysql-bin.000949 323M /data/mysql/mysql-bin.000950 4.0K /data/mysql/mysql-bin.index
從這裏可以發現，000949是mysql在系統崩潰的時候最後寫過的文件，在恢復之後重新建立了一個新的
000950，從時間和大小的條件可以判斷，正常情況下mysql-bin.000949應該會寫到1.1G的時候纔會重新建立新的文件繼續寫，現在的情況是服務器宕機導致binlog crash了，所以mysql啓動後會重新建立一個新的binlog文件。
3. 在slave上執行如下命令：
mysql> stop slave -> ; Query OK, 0 rows affected (0.00 sec) mysql> change master to master_host='10.90.13.238', master_user='slave' ,MASTER_PASSWORD='',MASTER_LOG_FILE='mysql-bin.000950',MASTER_LOG_POS=4; Query OK, 0 rows affected (0.09 sec)
就是在mysql上重新指定新的binlog和它的初始位置。然後啓動slave：
mysql> start slave;
觀察slave啓動正常了
mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 10.90.13.238 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000950 Read_Master_Log_Pos: 336968550 Relay_Log_File: mysql-relay-bin.000002 Relay_Log_Pos: 52752780 Relay_Master_Log_File: mysql-bin.000950 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 52752634 Relay_Log_Space: 336968852 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 31164 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 4 1 row in set (0.00 sec) mysql>