最近客户那处理了一个case，case中客户现场的mysql经常长时间hold在那里。
如下是客户现场的 INNODB MONITOR OUTPUT，我们来分析下：
首先是
InnoDB: Warning: a long semaphore wait:
--Thread 139979394471680 has waited at log0log.ic line 320 for 896.00 seconds the semaphore:
Mutex at 0x243d84d8 created file log0log.c line 771, lock var 0
waiters flag 0
wait has ended
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
这个warning显示innodb工作线程139979394471680在准备写日志时有个长达896s的信号量等待。
lock var 0 表示该锁的状态为空闲，1表示该锁已被持有。
waiters flag 0 表示当前没有正在等待该锁的其他线程，若有1个那么该值为1。
wait has ended 表示 “mutex is already free for grabs but os has not yet scheduled thread”。
所以综上可以初步认为，运行mysql的机器负载太高了，内部进程和线程数太多了，导致系统频繁地进行 context switch。
接着我们看下当前mysql的负载
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 66488 1_second, 66092 sleeps, 6594 10_second, 6365 background, 6365 flush
srv_master_thread log flush and writes: 72129
1_second的数量和sleeps 的数量基本为 1：1，且 10_second 的数量基本为1_second的 1/10。
说明mysql的负载其实较小。
接着我们看下当前mysql统计的信号量信息
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 91544822, signal count 67071774
。。。
Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
RW-shared spins 26704, rounds 383132, OS waits 11585
RW-excl spins 90035, rounds 556267, OS waits 13349
Spin rounds per wait: 13.27 mutex, 14.35 RW-shared, 6.18 RW-excl
可以看到 Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
解释下这3个值：
Mutex spin waits 836769568 is the number of times a thread tried to get a mutex and it wasn’t available, so it waited in a spin-wait.
rounds 11105882188 is the number of times threads looped in the spin-wait cycle, checking the mutex.
OS waits 80039908 is the number of times the thread gave up spin-waiting and went to sleep instead.
这3个值都太高了，特别是 OS waits。 说明操作系统层面的锁等待现象非常严重。
再接着看下I/O
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
。。。
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 1; buffer pool: 0
9612 OS file reads, 2628919 OS file writes, 277446 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.17 writes/s, 0.04 fsyncs/s
fsyncs/s 的值非常的小，说明刷盘的频率很低，也验证了mysql负载较低的结论。
最后看下内存使用
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 21978152960; in additional pool allocated 0
Dictionary memory allocated 700351
Buffer pool size 1310719
Free buffers 1236849
Database pages 73175
Old database pages 26991
Modified db pages 23191
。。。
该mysql的 innodb buffer pool size 配置的是20G，与Buffer pool size 1310719相符。
Buffer pool size 表示的是block的个数，每个block大小为16k。
Free buffers表示空闲的block数，可以看出占了绝大部分。
所以说明数据库当前操作的数据量其实很小。
综上，我们可以确认，mysql本身负载不高，甚至较低，但运行mysql的机器的负载非常的高，导致mysql的工作线程无法被调度到。
参考：
http://www.mysqlperformanceblog.com/2011/09/02/understand-innodb-spin-waits-win-a-percona-live-ticket/
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/
转载请注明转自高孝鑫的blog。
如下是客户现场的 INNODB MONITOR OUTPUT，我们来分析下：
首先是
InnoDB: Warning: a long semaphore wait:
--Thread 139979394471680 has waited at log0log.ic line 320 for 896.00 seconds the semaphore:
Mutex at 0x243d84d8 created file log0log.c line 771, lock var 0
waiters flag 0
wait has ended
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
这个warning显示innodb工作线程139979394471680在准备写日志时有个长达896s的信号量等待。
lock var 0 表示该锁的状态为空闲，1表示该锁已被持有。
waiters flag 0 表示当前没有正在等待该锁的其他线程，若有1个那么该值为1。
wait has ended 表示 “mutex is already free for grabs but os has not yet scheduled thread”。
所以综上可以初步认为，运行mysql的机器负载太高了，内部进程和线程数太多了，导致系统频繁地进行 context switch。
接着我们看下当前mysql的负载
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 66488 1_second, 66092 sleeps, 6594 10_second, 6365 background, 6365 flush
srv_master_thread log flush and writes: 72129
1_second的数量和sleeps 的数量基本为 1：1，且 10_second 的数量基本为1_second的 1/10。
说明mysql的负载其实较小。
接着我们看下当前mysql统计的信号量信息
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 91544822, signal count 67071774
。。。
Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
RW-shared spins 26704, rounds 383132, OS waits 11585
RW-excl spins 90035, rounds 556267, OS waits 13349
Spin rounds per wait: 13.27 mutex, 14.35 RW-shared, 6.18 RW-excl
可以看到 Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
解释下这3个值：
Mutex spin waits 836769568 is the number of times a thread tried to get a mutex and it wasn’t available, so it waited in a spin-wait.
rounds 11105882188 is the number of times threads looped in the spin-wait cycle, checking the mutex.
OS waits 80039908 is the number of times the thread gave up spin-waiting and went to sleep instead.
这3个值都太高了，特别是 OS waits。 说明操作系统层面的锁等待现象非常严重。
再接着看下I/O
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
。。。
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 1; buffer pool: 0
9612 OS file reads, 2628919 OS file writes, 277446 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.17 writes/s, 0.04 fsyncs/s
fsyncs/s 的值非常的小，说明刷盘的频率很低，也验证了mysql负载较低的结论。
最后看下内存使用
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 21978152960; in additional pool allocated 0
Dictionary memory allocated 700351
Buffer pool size 1310719
Free buffers 1236849
Database pages 73175
Old database pages 26991
Modified db pages 23191
。。。
该mysql的 innodb buffer pool size 配置的是20G，与Buffer pool size 1310719相符。
Buffer pool size 表示的是block的个数，每个block大小为16k。
Free buffers表示空闲的block数，可以看出占了绝大部分。
所以说明数据库当前操作的数据量其实很小。
综上，我们可以确认，mysql本身负载不高，甚至较低，但运行mysql的机器的负载非常的高，导致mysql的工作线程无法被调度到。
参考：
http://www.mysqlperformanceblog.com/2011/09/02/understand-innodb-spin-waits-win-a-percona-live-ticket/
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/
转载请注明转自高孝鑫的blog。