MySQLのIbdata1異常及恢復
環境:CentOS5.4/MySQL-5.1.48
1.將優化後的my.cnf替換舊的my.cnf,導致MySQLd無法啓動
err文件提示如下:
InnoDB: Error: auto-extending data file /data/mysqldata/innodb/data/ibdata1 is of a different size
InnoDB: 65536 pages (rounded down to MB) than specified in the .cnf file:
InnoDB: initial 327680 pages, max 0 (relevant if non-zero) pages!
131205 9:55:36 InnoDB: Could not open or create data files.
131205 9:55:36 InnoDB: If you tried to add new data files, and it failed here,
131205 9:55:36 InnoDB: you should now edit innodb_data_file_path in my.cnf back
131205 9:55:36 InnoDB: to what it was, and remove the new ibdata files InnoDB created
131205 9:55:36 InnoDB: in this failed attempt. InnoDB only wrote those files full of
131205 9:55:36 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
131205 9:55:36 InnoDB: remove old data files which contain your precious data!
131205 9:55:36 [ERROR] Plugin 'InnoDB' init function returned error.
131205 9:55:36 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131205 9:55:36 [ERROR] Unknown/unsupported storage engine: InnoDB
131205 9:55:36 [ERROR] Aborting
從Error看,表空間ibdata1跑在舊的my.cnf下大小65536/64M(64個page=1M,1個page=16k)比現在my.cnf文件中配置的
ibdata1值要小,導致數據文件無法打開，同時InnoDB存儲引擎加載失敗;
解決方式:
1.刪除現在ibdata1數據文件,引起另外麻煩,mysqld啓動了，但是所有InnoDB表報廢，select時提示該表不存在---慎用；
2.註釋該設置的參數(InnoDB_data_file_path),MySQLd啓用默認設置ibdata1值大小;
3.ibdata1大小擴展方式,網上搜索一把,查詢是ibdata1如何瘦身,涉及參數:InnoDB_data_file_path;
2.ibdata1如何擴展?
如：
#Ibdata1存放路徑,若不設置,默認存放在datadir下
InnoDB_data_home_dir = /opt/mysql/data
innodb_data_file_path = ibdata1:10M:autoextend
若直接將此參數值修改比10M大的值，啓動mysqld,提示上述錯誤，即使MySQLd可以啓動，InnoDB引擎初始化失敗,涉及InnoDB
表無法使用;
假設ibdata1所在磁盤空間剩餘大小1G,過段時間該數據文件(ibdata1)長到988MB,那麼就必須增加一個新的數據文件，來保證有
富裕的表空間,增加如下:
1.innodb_data_file_path = ibdata1:988M:autoextend #這種方式導致InnoDB引擎無法正常加載
2.innodb_data_file_path = ibdata1:10M;ibdata2:998M:autoextend
MySQLd啓動時，創建新的數據文件:ibdata2
err記錄如下:
131203 18:19:36 InnoDB: Data file /opt/mysql/data2/ibdata2 did not exist: new to be created
131203 18:19:36 InnoDB: Setting file /opt/mysql/data2/ibdata2 size to 988 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900
131203 18:19:41 InnoDB: Started; log sequence number 0 48574
3.ibdata1如何瘦身?
方法:
1.dump出所有DB
2.正常關閉MySQLd
3.rm ibdata1 ib_ib_logfile* mysql-bin.index
4.配置新表空間InnoDB_data_home_dir/InnoDB_data_file_path
4.啓動MySQLd
5.導入dump文件，完成ibdata1瘦身;
觀察其跑的系統環境,發現一切正常,數據的讀取與操作完全正常,原因:mysqld在
運行狀態中，會保持這些文件爲打開狀態；
即使被刪除了，它們仍舊存在於文件系統中，mysqld仍然可以對其進行讀寫
*****************************
務必這會不要關閉或重啓mysqld
*****************************
查詢pid-file運行目錄:
ps -ef | grep mysql
mysql 5839 5696 0 18:21 pts/0 00:00:00 /opt/mysql/libexec/mysqld
--defaults-file=/opt/mysql/data2/my3307.cnf
--basedir=/opt/mysql --datadir=/opt/mysql/data2
--user=mysql
--log-error=/opt/mysql/data2/node3307.err
--pid-file=/opt/mysql/data2/node3307.pid
--socket=/opt/mysql/data2/mysql3307.sock
--port=3307
獲取mysqld的pid(進程ID)
cat /opt/mysql/data2/node3307.pid
5839
[root@node proc]# ls -la /proc/5839/fd/ | grep -e ibdata -e ib_
lrwx------ 1 root root 64 Dec 3 19:08 10 -> /opt/mysql/data2/ib_logfile1(deleted)
lrwx------ 1 root root 64 Dec 3 19:08 4 -> /opt/mysql/data2/ibdata1(deleted)
lrwx------ 1 root root 64 Dec 3 19:08 9 -> /opt/mysql/data2/ib_logfile0(deleted)
只要mysqld不結束,就可以通過proc文件系統找到這幾個被刪除的文件(已經被標示爲deleted狀態)
那麼是否只要把這幾個文件複製回/opt/mysql/data2就可以了嗎？--No
因爲，在innodb的buffer pool中，有許多dirty page(內存中數據已經被修改，與磁盤中數據不一致，但是沒有
寫回到文件中).
如果直接複製回去，可能數據丟失，或ibdata1文件損壞。
爲此，首先要停止更多的寫入、更新、刪除操作,而後等待innodb flush pages to disk.
停止寫入，可以把跑於該DB系統關閉對外服務.或lock tables;
mysql>flush tables with read lock;
這時等待它flush結束，怎樣知道結束沒有？觀看checkpoint age變化.
mysql>show engine innodb status \G;
---
LOG
---
Log SEQUENCE NUMBER 363096003
Log flushed up TO 363096003
LAST checkpoint at 363096003
--checkpoint age:Log SEQUENCE NUMBER減去LAST checkpoint at的值
若爲0，那麼所有page都flush到磁盤文件中了。
不過爲了加速這個flush過程,可以設置:
mysql>set global innodb_max_dirty_pages_pct=0;
此外，必須保證一些後臺的線程完成了它們的工作,
如，insert buffer thread.ibuf的大小應=1
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: SIZE 1, free list len 398, seg SIZE 400,
還有purge thread，應purge了全部transactions:
------------
TRANSACTIONS
------------
Trx id counter 0 16644
Purge done FOR trx's n:o < 0 16644 undo n:o < 0 0
還要確保innodb IO不再寫操作:
FILE I/O
--------
I/O thread 0 state: waiting FOR i/o request (INSERT buffer thread)
I/O thread 1 state: waiting FOR i/o request (log thread)
I/O thread 2 state: waiting FOR i/o request (READ thread)
I/O thread 3 state: waiting FOR i/o request (WRITE thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
332 OS file reads, 47 OS file writes, 32 OS fsyncs
0.00 reads/s, 0 avg bytes/READ, 0.00 writes/s, 0.00 fsyncs/s
上面都完成後，把文件複製回去：
[root@node proc]# cp /proc/5839/fd/10 /opt/mysql/data2/ib_logfile1
[root@node proc]# cp /proc/5839/fd/4 /opt/mysql/data2/ibdata1
[root@node proc]# cp /proc/5839/fd/9 /opt/mysql/data2/ib_logfile0
注意修改文件權限:
chown -R mysql:mysql ibdata1 ib_logfile* 重啓mysql
總結:
1,解決方案不明確時，不要進行操作，如重啓Mysqld，重啓服務器
2,有必要監控mysql的ibdata，ib_logfile*等文件存在
-----------------
如下一段信息是MySQL5.1.48在scripts/mysql_install_db後,執行bin/mysqld_safe生成內容:
131203 16:39:55 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/data2
131203 16:39:55 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please us
e '--skip-external-locking' instead.
InnoDB: The first specified data file /opt/mysql/data2/ibdata1 did not exist:
InnoDB: a new database to be created!
131203 16:39:55 InnoDB: Setting file /opt/mysql/data2/ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
131203 16:39:56 InnoDB: Log file /opt/mysql/data2/ib_logfile0 did not exist: new to be created
InnoDB: Setting log file /opt/mysql/data2/ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
131203 16:39:56 InnoDB: Log file /opt/mysql/data2/ib_logfile1 did not exist: new to be created
InnoDB: Setting log file /opt/mysql/data2/ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
131203 16:39:56 InnoDB: Started; log sequence number 0 0
131203 16:39:56 [Note] Event Scheduler: Loaded 0 events
...ommited...
......
131203 16:39:56 [Note] /opt/mysql/libexec/mysqld: ready for connections.
Version: '5.1.48-log' socket: '/opt/mysql/data2/mysql3307.sock' port: 3307 Source distribution
環境:CentOS5.4/MySQL-5.1.48
1.將優化後的my.cnf替換舊的my.cnf,導致MySQLd無法啓動
err文件提示如下:
InnoDB: Error: auto-extending data file /data/mysqldata/innodb/data/ibdata1 is of a different size
InnoDB: 65536 pages (rounded down to MB) than specified in the .cnf file:
InnoDB: initial 327680 pages, max 0 (relevant if non-zero) pages!
131205 9:55:36 InnoDB: Could not open or create data files.
131205 9:55:36 InnoDB: If you tried to add new data files, and it failed here,
131205 9:55:36 InnoDB: you should now edit innodb_data_file_path in my.cnf back
131205 9:55:36 InnoDB: to what it was, and remove the new ibdata files InnoDB created
131205 9:55:36 InnoDB: in this failed attempt. InnoDB only wrote those files full of
131205 9:55:36 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
131205 9:55:36 InnoDB: remove old data files which contain your precious data!
131205 9:55:36 [ERROR] Plugin 'InnoDB' init function returned error.
131205 9:55:36 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131205 9:55:36 [ERROR] Unknown/unsupported storage engine: InnoDB
131205 9:55:36 [ERROR] Aborting
從Error看,表空間ibdata1跑在舊的my.cnf下大小65536/64M(64個page=1M,1個page=16k)比現在my.cnf文件中配置的
ibdata1值要小,導致數據文件無法打開，同時InnoDB存儲引擎加載失敗;
解決方式:
1.刪除現在ibdata1數據文件,引起另外麻煩,mysqld啓動了，但是所有InnoDB表報廢，select時提示該表不存在---慎用；
2.註釋該設置的參數(InnoDB_data_file_path),MySQLd啓用默認設置ibdata1值大小;
3.ibdata1大小擴展方式,網上搜索一把,查詢是ibdata1如何瘦身,涉及參數:InnoDB_data_file_path;
2.ibdata1如何擴展?
如：
#Ibdata1存放路徑,若不設置,默認存放在datadir下
InnoDB_data_home_dir = /opt/mysql/data
innodb_data_file_path = ibdata1:10M:autoextend
若直接將此參數值修改比10M大的值，啓動mysqld,提示上述錯誤，即使MySQLd可以啓動，InnoDB引擎初始化失敗,涉及InnoDB
表無法使用;
假設ibdata1所在磁盤空間剩餘大小1G,過段時間該數據文件(ibdata1)長到988MB,那麼就必須增加一個新的數據文件，來保證有
富裕的表空間,增加如下:
1.innodb_data_file_path = ibdata1:988M:autoextend #這種方式導致InnoDB引擎無法正常加載
2.innodb_data_file_path = ibdata1:10M;ibdata2:998M:autoextend
MySQLd啓動時，創建新的數據文件:ibdata2
err記錄如下:
131203 18:19:36 InnoDB: Data file /opt/mysql/data2/ibdata2 did not exist: new to be created
131203 18:19:36 InnoDB: Setting file /opt/mysql/data2/ibdata2 size to 988 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900
131203 18:19:41 InnoDB: Started; log sequence number 0 48574
3.ibdata1如何瘦身?
方法:
1.dump出所有DB
2.正常關閉MySQLd
3.rm ibdata1 ib_ib_logfile* mysql-bin.index
4.配置新表空間InnoDB_data_home_dir/InnoDB_data_file_path
4.啓動MySQLd
5.導入dump文件，完成ibdata1瘦身;
-------
觀察其跑的系統環境,發現一切正常,數據的讀取與操作完全正常,原因:mysqld在
運行狀態中，會保持這些文件爲打開狀態；
即使被刪除了，它們仍舊存在於文件系統中，mysqld仍然可以對其進行讀寫
*****************************
務必這會不要關閉或重啓mysqld
*****************************
查詢pid-file運行目錄:
ps -ef | grep mysql
mysql 5839 5696 0 18:21 pts/0 00:00:00 /opt/mysql/libexec/mysqld
--defaults-file=/opt/mysql/data2/my3307.cnf
--basedir=/opt/mysql --datadir=/opt/mysql/data2
--user=mysql
--log-error=/opt/mysql/data2/node3307.err
--pid-file=/opt/mysql/data2/node3307.pid
--socket=/opt/mysql/data2/mysql3307.sock
--port=3307
獲取mysqld的pid(進程ID)
cat /opt/mysql/data2/node3307.pid
5839
[root@node proc]# ls -la /proc/5839/fd/ | grep -e ibdata -e ib_
lrwx------ 1 root root 64 Dec 3 19:08 10 -> /opt/mysql/data2/ib_logfile1(deleted)
lrwx------ 1 root root 64 Dec 3 19:08 4 -> /opt/mysql/data2/ibdata1(deleted)
lrwx------ 1 root root 64 Dec 3 19:08 9 -> /opt/mysql/data2/ib_logfile0(deleted)
只要mysqld不結束,就可以通過proc文件系統找到這幾個被刪除的文件(已經被標示爲deleted狀態)
那麼是否只要把這幾個文件複製回/opt/mysql/data2就可以了嗎？--No
因爲，在innodb的buffer pool中，有許多dirty page(內存中數據已經被修改，與磁盤中數據不一致，但是沒有
寫回到文件中).
如果直接複製回去，可能數據丟失，或ibdata1文件損壞。
備份mysql數據時，也不同直接備份這幾個文件，也是同樣的道理.
接下來需要做的事：
這會我們要做的事，就是保證所有buffer pool中的數據修改都保存在磁盤文件上面.爲此，首先要停止更多的寫入、更新、刪除操作,而後等待innodb flush pages to disk.
停止寫入，可以把跑於該DB系統關閉對外服務.或lock tables;
mysql>flush tables with read lock;
這時等待它flush結束，怎樣知道結束沒有？觀看checkpoint age變化.
mysql>show engine innodb status \G;
---
LOG
---
Log SEQUENCE NUMBER 363096003
Log flushed up TO 363096003
LAST checkpoint at 363096003
--checkpoint age:Log SEQUENCE NUMBER減去LAST checkpoint at的值
若爲0，那麼所有page都flush到磁盤文件中了。
不過爲了加速這個flush過程,可以設置:
mysql>set global innodb_max_dirty_pages_pct=0;
此外，必須保證一些後臺的線程完成了它們的工作,
如，insert buffer thread.ibuf的大小應=1
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: SIZE 1, free list len 398, seg SIZE 400,
還有purge thread，應purge了全部transactions:
------------
TRANSACTIONS
------------
Trx id counter 0 16644
Purge done FOR trx's n:o < 0 16644 undo n:o < 0 0
還要確保innodb IO不再寫操作:
FILE I/O
--------
I/O thread 0 state: waiting FOR i/o request (INSERT buffer thread)
I/O thread 1 state: waiting FOR i/o request (log thread)
I/O thread 2 state: waiting FOR i/o request (READ thread)
I/O thread 3 state: waiting FOR i/o request (WRITE thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
332 OS file reads, 47 OS file writes, 32 OS fsyncs
0.00 reads/s, 0 avg bytes/READ, 0.00 writes/s, 0.00 fsyncs/s
上面都完成後，把文件複製回去：
[root@node proc]# cp /proc/5839/fd/10 /opt/mysql/data2/ib_logfile1
[root@node proc]# cp /proc/5839/fd/4 /opt/mysql/data2/ibdata1
[root@node proc]# cp /proc/5839/fd/9 /opt/mysql/data2/ib_logfile0
注意修改文件權限:
chown -R mysql:mysql ibdata1 ib_logfile* 重啓mysql
總結:
1,解決方案不明確時，不要進行操作，如重啓Mysqld，重啓服務器
2,有必要監控mysql的ibdata，ib_logfile*等文件存在
-----------------
5.InnoDB涉及共享表空間和獨立表空間優/缺點，之後再瞭解?
如下一段信息是MySQL5.1.48在scripts/mysql_install_db後,執行bin/mysqld_safe生成內容:
131203 16:39:55 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/data2
131203 16:39:55 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please us
e '--skip-external-locking' instead.
InnoDB: The first specified data file /opt/mysql/data2/ibdata1 did not exist:
InnoDB: a new database to be created!
131203 16:39:55 InnoDB: Setting file /opt/mysql/data2/ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
131203 16:39:56 InnoDB: Log file /opt/mysql/data2/ib_logfile0 did not exist: new to be created
InnoDB: Setting log file /opt/mysql/data2/ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
131203 16:39:56 InnoDB: Log file /opt/mysql/data2/ib_logfile1 did not exist: new to be created
InnoDB: Setting log file /opt/mysql/data2/ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
131203 16:39:56 InnoDB: Started; log sequence number 0 0
131203 16:39:56 [Note] Event Scheduler: Loaded 0 events
...ommited...
......
131203 16:39:56 [Note] /opt/mysql/libexec/mysqld: ready for connections.
Version: '5.1.48-log' socket: '/opt/mysql/data2/mysql3307.sock' port: 3307 Source distribution
