在一臺server中以各數據庫的備份文件爲數據文件啓動多個MySQL實例供SQL Review使用。
之前執行一直沒有問題(最多的時候有23個MySQL實例同一時候執行)。後來新配置了一臺server,啓動其相應的實例時失敗。
部分錯誤日誌例如以下:
……
140505 16:05:59 InnoDB: Using Linux native AIO
140505 16:05:59 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
140505 16:06:02 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
140505 16:06:02 InnoDB: Fatal error: cannot initialize AIO sub-system
140505 16:06:02 [ERROR] Plugin 'InnoDB' init function returned error.
140505 16:06:02 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140505 16:06:02 [ERROR] Unknown/unsupported storage engine: InnoDB
……
通過錯誤日誌瞭解到最早錯誤發生的地方爲 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts。
這個io_setup() failed with EAGAIN是關鍵。
我們man一下io_setup
NAME
io_setup - Create an asynchronous I/O context
……
DESCRIPTION
io_setup() creates an asynchronous I/O context capable of receiving at least maxevents. ctxp must not point to an AIO context that already exists, and must be
initialized to 0
prior to the call. On successful creation of the AIO context, *ctxp is filled in with the resulting handle.
RETURN VALUE
io_setup() returns 0 on success; otherwise, one of the errors listed in the "Errors" section is returned.
ERRORS
EINVAL ctxp is not initialized, or the specified maxevents exceeds internal limits. maxevents should be greater than 0.
EFAULT An invalid pointer is passed for ctxp.
ENOMEM Insufficient kernel resources are available.
EAGAIN The specified maxevents exceeds the user’s limit of available events.
ENOSYS io_setup() is not implemented on this architecture.
CONFORMING TO
……
看到io_setup用來創建異步I/O上下文環境用於特定目的,錯誤代碼EAGAIN意爲指定的maxevents 超出了用戶可用events的限制。
該server上已經執行了較多的MySQL實例,創建異步I/O的資源已經達到了臨界,所以新的實例啓動失敗。
最後通過在啓動MySQL實例時增加 --innodb_use_native_aio = 0攻克了問題。
也有通過更改系統設置來解決此問題的(待驗證)。
cat /proc/sys/fs/aio-max-nr能夠查看到當前的aio-max-nr的值一般爲65536(64k個)
可通過下述步驟改變該文件裏的值(上述文件不能直接編輯)
sudo vim /etc/sysctl.conf
改動或增加
fs.aio-max-nr=262144(256k個)
運行命令改動/proc/sys/fs/aio-max-nr
sysctl -p
能夠看到/proc/sys/fs/aio-max-nr中的值發生了變化
cat /proc/sys/fs/aio-max-nr
重新啓動MySQL實例
還有通過改動mysql源代碼來避免該問題的,但普通情況下不推薦也沒有必要這麼做。