再深入一點|binlog和relay-log到底長啥樣？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上一篇mysql面試的文章之後收到不少朋友的意見，希望深入講講複製、日誌的格式這些，今天，我們就來深挖一下mysql的複製機制到底有哪一些，以及binlog和relay-log的結構到底是什麼樣子的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"binlog作用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"binlog的主要作用是記錄數據庫中表的更改，它只記錄改變數據的sql，不改變數據的sql不會寫入，比如select語句一般不會被記錄，因爲他們不會對數據產生任何改動。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用一個實際的場景看下binlog產生的過程，準備sql："}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"create table\ttest(text varchar(20));\ninsert into test values ('test_text');\nselect * from test;\nflush logs;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看binlog"}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"show binlog events in 'binlog.000029';"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯示的結果如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/61/61bffe4ea2ae8ef24c05c90518edbe03.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，也可以使用mysqlbinlog工具來查看binlog的內容："}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"show variables like 'log_%'; #查看日誌目錄\nmysqlbinlog --short-form --force-if-open --base64-output=never /usr/local/var/mysql/binlog.000029"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c2ada6e4d6c70092bea351e40c5e513.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0d33f7f9ce160cb97f41b2e317805413.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從日誌我們可以看到執行了創建表的語句以及一個Format_desc頭和Ratate輪換事件，這個我們會在後面講到，先看幾個字段代表的含義。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Log_name"},{"type":"text","text":"代表日誌文件的名稱，比如我這裏的查詢是直接查詢binlog.000029，默認的寫法是show binlog events，但是這樣只會查詢到第一個binlog，並不是當前激活狀態的binlog，如果你不知道binlog有哪些，可以用命令:"}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"show binary logs; #查看binlog列表\nshow master status; #查看最新的binlog"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/56/5681190b57ac013e3f0cf2d51fa44d8a.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Pos"},{"type":"text","text":"代表文件開始的位置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Event_type"},{"type":"text","text":"代表事件的類型。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Server_id"},{"type":"text","text":"是創建事件的服務器ID。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"End_log_pos"},{"type":"text","text":"代表事件在文件中的結束位置，以上面爲例，第一次查詢的結束位置是723，第二次insert之後文件的開始位置就是從723開始。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Info"},{"type":"text","text":"代表事件信息，是一段可讀的文本內容。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"binlog日誌結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"binlog日誌的結構大概是長這樣的，它由索引文件和binlog文件組成，其中binlog事件又包含通用頭、提交頭和事件體3個部分組成。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2e409f43bc8ad2f1f74ef3a85fd00d6f.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先說說索引文件，索引文件的每一行都包含了一個binlog文件的完整文件名(類似host-bin.001)，一些命令比如flush logs將所有日誌寫入磁盤會影響到索引文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個binlog文件以若干個binlog事件組成，以格式描述事件(Format_description)作爲文件頭(上面的binlog圖片Format_desc事件)，以日誌輪換事件(rotate)作爲文件尾。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Format_description"},{"type":"text","text":"包含binlog文件的服務器信息、文件狀態的關鍵信息等。如果服務器關閉或者重啓，則會創建一個新的binlog文件，同時寫入一個新的format_description。他的格式大致如下。"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"2 binlog-version\nstring[50] mysql-server version\n4 create timestamp\n1 event header length\nstring[p] event type header lengths"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"日誌輪換事件"},{"type":"text","text":"則包含下一個binlog的文件名以及開始讀取的位置，它由服務器寫完binlog後添加到文件尾，輪換事件並不會每次都存在，格式如下。"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"if binlog-version > 1 {\n8 position\n}\nstring[p] name of the next binlog"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"binlog事件"},{"type":"text","text":"包含若干個事務組成的組(group)，每個組對應一個事務，如果是create alter語句不屬於事務語句的話，則他們本身就是一個組，每個組要麼全部執行，要麼都不執行。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fc/fcbce10c92dc57469a734de3f68e056f.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"binlog事件結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個binlog事件由3個部分組成："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通用頭，包含binlog中所有事件具備的基本信息。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"提交頭，對於不同類型的事件來說，提交頭的內容也不盡相同"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"事件體，存儲事件的主要數據，同樣對於不同類型事件也不同。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"binlog輪換和清理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面的例子我們也可以看出來，binlog並非只有一個，而基於真實的場景來說，始終寫一個binlog文件肯定也是不可取的，而binlog輪換主要有3個場景："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"服務器啓動，每次服務器啓動都會生成一個新的binlog文件。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"達到最大大小，可以通過binlog-cache-size控制大小，達到最大大小後將更換。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"顯示刷新，flush logs將所有日誌寫入磁盤，這時候會創建一個新的文件寫入，從第一個例子也能看出來執行完之後生成了一個新的日誌binlog.000030的文件並且開始的位置是4。"}]}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5b/5b7fa746572fa7857e5f67db0d2afad1.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着時間的推移，我們的binlog文件會越來越多，這時候有兩種方式可以清除binlog："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通過設置expire-logs-days控制想保留的binlog日誌文件天數，系統將會自動清理。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通過PURGE BINARY LOGS手動清理"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"relay-log結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"relay-log中繼日誌是連接master和slave的核心，我們來深入瞭解一下它的結構和使用。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/9453d835271b18c44876ea9e3d4dce7a.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"relay-log的結構和binlog非常相似，只不過他多了一個master.info和relay-log.info的文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"master.info"},{"type":"text","text":"記錄了上一次讀取到master同步過來的binlog的位置，以及連接master和啓動複製必須的所有信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"relay-log.info"},{"type":"text","text":"記錄了文件複製的進度，下一個事件從什麼位置開始，由sql線程負責更新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上一篇文章我們提到了整個複製流程的過程大概是這個樣子："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c0/c02f1cadf309ae03de500472183f2990.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"知道binlog和relay-log的結構之後，我們重新梳理一下整個鏈路的流程，這裏我們假定master.info和relay-log.info都是存在的情況："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Master收到客戶端請求語句，在語句結束之前向二進制日誌寫入一條記錄，可能包含多個事件。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"此時，一個Slave連接到Master，Master的dump線程從binlog讀取日誌併發送到Slave的IO線程。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"IO線程從master.info讀取到上一次寫入的最後的位置。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"IO線程寫入日誌到relay-log中繼日誌，如果超過指定的relay-log大小，寫入輪換事件，創建一個新的relay-log。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"更新master.info的最後位置"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"SQL線程從relay-log.info讀取進上一次讀取的位置"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"SQL線程讀取日誌事件"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":8,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在數據庫中執行sql"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":9,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"更新relay-log.info的最後位置"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":10,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Slave記錄自己的binlog日誌"}]}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/72/720491d3eac905ce268cf74ad32df269.jpeg","alt":"","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是在這裏IO和SQL線程有會產生重複事件的問題，舉一個場景："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"先記錄中繼日誌，然後更新master.info位置"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"此時服務器崩潰，寫入master.info失敗"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"服務器恢復，再次同步從master.info獲取到的是上一次的位置，會導致事件重複執行"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"既然會有這個問題還爲什麼要這樣做呢？假設反過來，先更新master.info再記錄中繼日誌，這樣帶來的問題就是丟失數據了。而mysql認爲丟失比重複更嚴重，所以要先刷新日誌，保大還是保小mysql幫你做了決定。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e3e2b1d9d8cc8edce05f39dc757198f3.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"- END -"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}