一 故障描述
發現Zabbix mysql的磁盤空間滿了，之前升級Zabbix到3.2的時候，history_text這個表分區不起作用了，也沒有及時處理。導致這個表越來越大，佔了磁盤空間140多G。
於是使用mv 命令將history_text.ibd這個文件移動到其他目錄下，再使用ln -sf軟鏈接的方式連接到原來的目錄下。但是啓動mysql報錯了。
還有看ibdata1比較大，我也是用了軟連接方式把這個文件移動到其他目錄下。然後mysql就無法啓動了。
[Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details) [Warning] Buffered warning: option 'table_open_cache': unsigned value 8388608 adjusted to 524288 [Warning] Buffered warning: Changed limits: max_open_files: 102400 (requested 1078586) [Warning] Buffered warning: Changed limits: table_open_cache: 36195 (requested 524288) 7fdcafc77700 InnoDB: Warning: purge reached the head of the history list, InnoDB: but its length is still reported as 42949673139! Make a detailed bug InnoDB: report, and submit it to http://bugs.mysql.com InnoDB: Assertion failure in thread 140585818617600 in file trx0purge.cc line 699 InnoDB: Failing assertion: purge_sys->iter.trx_no <= purge_sys->rseg->last_trx_no InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:14:05 UTC - mysqld got signal 6 ;
二 故障解決
[Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details)
解決辦法：
在/etc/my.cnf中添加
explicit_defaults_for_timestamp=1
2.
[Warning] Buffered warning: option 'table_open_cache': unsigned value 8388608 adjusted to 524288
[Warning] Buffered warning: Changed limits: max_open_files: 102400 (requested 1078586)
[Warning] Buffered warning: Changed limits: table_open_cache: 36195 (requested 524288)
解決辦法：
這個3個警告信息都和系統文件打開數有關
ulimit -n 1024000
/etc/security/limits.conf
/etc/security/limits.d/90-nproc.conf
兩個文件中的nofile調大些
3.
7fdcafc77700 InnoDB: Warning: purge reached the head of the history list,
InnoDB: but its length is still reported as 42949673139! Make a detailed bug
InnoDB: report, and submit it to http://bugs.mysql.com
InnoDB: Assertion failure in thread 140585818617600 in file trx0purge.cc line 699
InnoDB: Failing assertion: purge_sys->iter.trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:14:05 UTC - mysqld got signal 6 ;
從日誌中可以看出MySQL的InnoDB崩潰了。
這時候可以設置MySQL的InnoDB爲恢復模式，恢復模式會使用數據庫只讀，用於不能更新、插入或者操作其他改變數據的操作。
innodb_force_recovery 有0~6個級別
設置
innodb_force_recovery=2
innodb_purge_threads=0
然後檢查有哪些表損壞
mysqlcheck -A -uroot -p
然後將需要的數據備份下來，重新導入到新的mysql庫中，
/opt/app/mysql/bin/mysqldump -uroot -p zabbix --ignore-table=zabbix.history --ignore-table=zabbix.history_uint --ignore-table=zabbix.history_str --ignore-table=zabbix.history_str > zabbixbackup20161215.sql
/opt/app/mysql/bin/mysql -uroot -p zabbix < /mnt/sdb1/zabbixbackup20161215.sql
參考文檔：
http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
http://blackbird.si/mysql-corrupted-innodb-tables-recovery-step-by-step-guide/