mysql binlog 筆記

binlog 是 mysql 對操作日誌的記錄，本身爲二進制文件，需要使用 mysqlbinlog 工具命令查看具體內容。包括三種模式：

• STATEMENT 記錄每一條修改語句，無需記錄每一條SQL 語句和每一行的數據變化，減少了日誌量；但某些場景下會導致 master-slave 中的數據不一致，如 sleep 函數，last_insert_id(),user define function等。
• ROW 不記錄每條 SQL 的上下文信息，僅需記錄哪條數據被修改了，修改成什麼樣了。缺點是會產生大量的日誌，alter 的時候會導致日誌暴漲。
• MIXED 混合模式，一般的複製使用 statement模式保存 binlog，對於 statement 模式無法複製的操作使用 row 模式保存 binlog，具體決策由 mysql 自動選擇。

日誌文件會根據配置進行更新，比如重啓 mysql，主動執行flush logs，日誌量、日誌天數到達配置觸發條件等操作，

查看當前mysql服務器是否開啓 binlog

# 看到 log-format 是 ROW
# 以及 log_bin 爲 ON 的狀態即可
show variables like '%log_%';

查看 master 有哪些日誌文件

show master logs;

查看當前 master 日誌記錄到了什麼狀態

show master status

查看 binlog 事件

# 日誌文件參考上一步
show binlog events in 'binlog.000002';

查看 binlog 具體內容，我們通常查看最新的 binlog 文件（編號最大的那個），在需要隔斷新日誌記錄的時候，執行下flush logs來避免服務器對舊的 binlog 文件的寫入。

mysqlbinlog --no-defaults --base64-output=decode-rows binlog.000002 

可以使用的參數有：

• --start-datetime='2020-02-27 15:00:00'
• --end-datetime='2020-02-28 00:00:00'
• --start-position=1232
• --stop-position=3434

可以根據區間或者配合 events 中顯示的某條命令的 position 進行區間選擇。

在MySQL5.5以下版本使用mysqlbinlog命令時如果報錯，就加上 “–no-defaults”選項

實際操作

root@vps90:/tmp#docker run -d -p 3333:3306 -e MYSQL_ROOT_PASSWORD=123456 --name=master-mysql mysql

root@vps90:/tmp# docker ps -a
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                      PORTS                                                            NAMES
698036278ab2        mysql                       "docker-entrypoint.s…"   3 seconds ago       Up 2 seconds                33060/tcp, 0.0.0.0:3333->3306/tcp                                master-mysql

root@vps90:/tmp# docker exec -it 698036278ab2 /bin/bash

root@698036278ab2:/# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.19 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.01 sec)

mysql> show master logs;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |   3084516 | No        |
| binlog.000002 |       155 | No        |
+---------------+-----------+-----------+
2 rows in set (0.00 sec)

mysql> create database ops;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE TABLE IF NOT EXISTS `member` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(16) NOT NULL,
`sex` enum('m','w') NOT NULL DEFAULT 'm',
`age` tinyint(3) unsigned NOT NULL,
`classid` char(6) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected, 3 warnings (0.02 sec)


mysql>  insert into member(`name`,`sex`,`age`,`classid`) values('wangshibo','m',27,'cls1'),('guohuihui','w',27,'cls2');
Query OK, 2 rows affected (0.01 sec)
Records: 2  Duplicates: 0  Warnings: 0


mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 | guohuihui | w   |  27 | cls2    |
+----+-----------+-----+-----+---------+
2 rows in set (0.00 sec)

手動模擬下對 mysql 的備份

root@698036278ab2:/# mysqldump -uroot -p -B -F -R -x --master-data=2 ops|gzip >/tmp/ops_$(date +%F).sql.gz
Enter password:
root@698036278ab2:/# ls /tmp
ops_2020-02-27.sql.gz
root@698036278ab2:/#

參數說明：

• B：指定數據庫
• F：刷新日誌
• R：備份存儲過程等
• x：鎖表
• –master-data：在備份語句裏添加CHANGE MASTER語句以及binlog文件及位置點信息

待到數據庫備份完成，就不用擔心數據丟失了，因爲有完全備份數據在！！

下面開始模擬業務對數據的修改

mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 | guohuihui | w   |  27 | cls2    |
+----+-----------+-----+-----+---------+
2 rows in set (0.00 sec)

mysql> show master status;
+---------------+----------+--------------+------------------+-------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000003 |      155 |              |                  |                   |
+---------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

mysql> insert into ops.member(`name`,`sex`,`age`,`classid`) values('yiyi','w',20,'cls1'),('xiaoer','m',22,'cls3'),('zhangsan','w',21,'cls5'),('lisi','m',20,'cls4'),('wangwu','w',26,'cls6');
Query OK, 5 rows affected (0.00 sec)
Records: 5  Duplicates: 0  Warnings: 0

mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 | guohuihui | w   |  27 | cls2    |
|  3 | yiyi      | w   |  20 | cls1    |
|  4 | xiaoer    | m   |  22 | cls3    |
|  5 | zhangsan  | w   |  21 | cls5    |
|  6 | lisi      | m   |  20 | cls4    |
|  7 | wangwu    | w   |  26 | cls6    |
+----+-----------+-----+-----+---------+
7 rows in set (0.00 sec)

mysql>
mysql>
mysql> update ops.member set name='' where id=4;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql>
mysql> update ops.member set name='' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 |           | w   |  27 | cls2    |
|  3 | yiyi      | w   |  20 | cls1    |
|  4 |           | m   |  22 | cls3    |
|  5 | zhangsan  | w   |  21 | cls5    |
|  6 | lisi      | m   |  20 | cls4    |
|  7 | wangwu    | w   |  26 | cls6    |
+----+-----------+-----+-----+---------+
7 rows in set (0.00 sec)

mysql>

可以看到數據庫中內容的確被修改了。然後模擬誤操作行爲

mysql> drop database ops;
Query OK, 1 row affected (0.02 sec)

1 正式的解決步驟：

# 到“服務器”上備份下之前的 binlog 文件
root@698036278ab2:/var/lib/mysql# cp binlog.000003 /tmp

2 然後執行flush logs 來避免新的操作對原來 binlog 文件的修改，讓新的操作都記錄到新的 binlog 文件中。

mysql> flush logs;
Query OK, 0 rows affected (0.01 sec)

mysql> show master logs
    -> ;
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |   3084516 | No        |
| binlog.000002 |      1152 | No        |
| binlog.000003 |      1406 | No        |
| binlog.000004 |       155 | No        |
+---------------+-----------+-----------+
4 rows in set (0.00 sec)

mysql> show master status;
+---------------+----------+--------------+------------------+-------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000004 |      155 |              |                  |                   |
+---------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

3. 讀取 binlog 分析問題原因

• mysqlbinlog binlog.00000x --方式，這個不是很直觀
• show binlog events in 'binlog.000003' 這個更加直觀。

mysql> show binlog events in 'binlog.000003';
+---------------+------+----------------+-----------+-------------+--------------------------------------+
| Log_name      | Pos  | Event_type     | Server_id | End_log_pos | Info                                 |
+---------------+------+----------------+-----------+-------------+--------------------------------------+
| binlog.000003 |    4 | Format_desc    |         1 |         124 | Server ver: 8.0.19, Binlog ver: 4    |
| binlog.000003 |  124 | Previous_gtids |         1 |         155 |                                      |
| binlog.000003 |  155 | Anonymous_Gtid |         1 |         234 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| binlog.000003 |  234 | Query          |         1 |         308 | BEGIN                                |
| binlog.000003 |  308 | Table_map      |         1 |         372 | table_id: 118 (ops.member)           |
| binlog.000003 |  372 | Write_rows     |         1 |         500 | table_id: 118 flags: STMT_END_F      |
| binlog.000003 |  500 | Xid            |         1 |         531 | COMMIT /* xid=66 */                  |
| binlog.000003 |  531 | Anonymous_Gtid |         1 |         610 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| binlog.000003 |  610 | Query          |         1 |         693 | BEGIN                                |
| binlog.000003 |  693 | Table_map      |         1 |         757 | table_id: 118 (ops.member)           |
| binlog.000003 |  757 | Update_rows    |         1 |         825 | table_id: 118 flags: STMT_END_F      |
| binlog.000003 |  825 | Xid            |         1 |         856 | COMMIT /* xid=68 */                  |
| binlog.000003 |  856 | Anonymous_Gtid |         1 |         935 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| binlog.000003 |  935 | Query          |         1 |        1018 | BEGIN                                |
| binlog.000003 | 1018 | Table_map      |         1 |        1082 | table_id: 118 (ops.member)           |
| binlog.000003 | 1082 | Update_rows    |         1 |        1153 | table_id: 118 flags: STMT_END_F      |
| binlog.000003 | 1153 | Xid            |         1 |        1184 | COMMIT /* xid=69 */                  |
| binlog.000003 | 1184 | Anonymous_Gtid |         1 |        1261 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| binlog.000003 | 1261 | Query          |         1 |        1362 | drop database ops /* xid=71 */       |
| binlog.000003 | 1362 | Rotate         |         1 |        1406 | binlog.000004;pos=4                  |
+---------------+------+----------------+-----------+-------------+--------------------------------------+
20 rows in set (0.00 sec)

可以看出 Pos=1261 時，執行了 drop 操作。而且出問題的 Position 範圍是 1261-1362，我們要進行數據恢復，只需要把數據恢復到 1261 之前就好。

4 恢復數據

先把之前備份的那個全量數據拿到，然後導入,爲了更加逼真，這裏新起一個服務器來測試

root@vps90:~# docker run -d -p 4444:3306 -e MYSQL_ROOT_PASSWORD=123456 --name=slave-mysql mysql
2a1967f3a6149be220a810b1eaccda5b84ebe398c5ed4096b5130b51d6d16a9b
root@vps90:~# docker ps -a | grep mysql
2a1967f3a614        mysql                       "docker-entrypoint.s…"   9 seconds ago       Up 8 seconds                33060/tcp, 0.0.0.0:4444->3306/tcp                                slave-mysql
698036278ab2        mysql                       "docker-entrypoint.s…"   20 minutes ago      Up 20 minutes               33060/tcp, 0.0.0.0:3333->3306/tcp                                master-mysql
root@vps90:~# docker cp 698036278ab2:/tmp/ops_2020-02-27.sql.gz 2a1967f3a614:/tmp/
copying between containers is not supported
root@vps90:~# docker cp 698036278ab2:/tmp/ops_2020-02-27.sql.gz /tmp
root@vps90:~# docker cp /tmp/ops_2020-02-27.sql.gz 2a1967f3a614:/tmp
root@vps90:~#

root@vps90:~# mysql -uroot -p
// 省略一些輸出
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

mysql> source /tmp/ops_2020-02-27.sql
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Database changed
Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| ops                |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

mysql> select * from ops;
ERROR 1146 (42S02): Table 'ops.ops' doesn't exist
mysql> use ops;
Database changed
mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 | guohuihui | w   |  27 | cls2    |
+----+-----------+-----+-----+---------+
2 rows in set (0.00 sec)

mysql>

然後就是根據 binlog 中出問題的語句進行 fix（就是刪掉對應的 drop 語句）注意這裏是走了新的服務器

root@vps90:/tmp# docker cp 698036278ab2:/tmp/binlog.000003.sql ./
root@vps90:/tmp# cp binlog.000003.sql binlog.000003.sql.bake
root@vps90:/tmp# vim binlog.000003.sql
root@vps90:/tmp# diff binlog.000003.sql binlog.000003.sql.bake
119a120
> drop database ops
root@vps90:/tmp#
root@vps90:/tmp# docker cp binlog.000003.sql 2a1967f3a614:/tmp
root@vps90:/tmp# docker exec -it 2a1967f3a614 /bin/bash
root@2a1967f3a614:/# tail /tmp/binlog.000003.sql
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 1261
#200227  7:50:50 server id 1  end_log_pos 1362 CRC32 0xd32877cf 	Query	thread_id=10	exec_time=0	error_code=0	Xid = 71
SET TIMESTAMP=1582789850/*!*/;
/*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
root@2a1967f3a614:/#
root@2a1967f3a614:/tmp# mysql -uroot -p -v ops < binlog.000003.sql
// 登陸進去，看下是否恢復
mysql> select * from member;
+----+-----------+-----+-----+---------+
| id | name      | sex | age | classid |
+----+-----------+-----+-----+---------+
|  1 | wangshibo | m   |  27 | cls1    |
|  2 |           | w   |  27 | cls2    |
|  3 | yiyi      | w   |  20 | cls1    |
|  4 |           | m   |  22 | cls3    |
|  5 | zhangsan  | w   |  21 | cls5    |
|  6 | lisi      | m   |  20 | cls4    |
|  7 | wangwu    | w   |  26 | cls6    |
+----+-----------+-----+-----+---------+
7 rows in set (0.00 sec)

拿到 binlog 手動修改失敗部分是一個方式，也可以使用--stop-position=1261的方式進行修改(前提是有database 和 member，也就是先前的備份數據ops_2020-02-27.sql 得是存在的)， 這一點經測試發現，還是很重要的。

/usr/bin/mysqlbinlog --stop-position=1261 --database=ops /var/lib/mysql/binlog.000003 | /usr/bin/mysql -uroot -p123456 -v ops

整理處理流程：

• 導入備份數據
• 分析binlog，導出 SQL
• 修改異常 SQL 語句
• 修改好的 SQL 導回 mysql
• 查看數據是否恢復

---

參考文獻：https://www.cnblogs.com/kevingrace/p/5907254.html