DMESG查看日誌,發現設備問題。
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
device-mapper: table: 253:27: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:27: multipath: error getting device
device-mapper: ioctl: error adding target to table
查看db日誌,07 15:50:36 2013 出現了IO問題
Checkpoint not complete
Current log# 3 seq# 847 mem# 0: +CORE_REDO01/coredb/onlinelog/group_3.261.812494827
Current log# 3 seq# 847 mem# 1: +CORE_REDO02/coredb/onlinelog/group_3.261.812494831
Thread 2 advanced to log sequence 848 (LGWR switch)
Current log# 4 seq# 848 mem# 0: +CORE_REDO01/coredb/onlinelog/group_4.262.812494835
Current log# 4 seq# 848 mem# 1: +CORE_REDO02/coredb/onlinelog/group_4.262.812494837
Fri Jun 07 15:50:36 2013
Errors in file /oracle/app/oracle/diag/rdbms/coredb/coredb2/trace/coredb2_ora_46624.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 262144
WARNING: Write Failed. group:7 disk:0 AU:126728 offset:262144 size:262144
Errors in file /oracle/app/oracle/diag/rdbms/coredb/coredb2/trace/coredb2_ora_46624.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 7490 logical extent 0 of file 296 in group 7 on disk 0 allocation unit 126728
Fri Jun 07 15:50:40 2013
Thread 2 cannot allocate new log, sequence 849
Checkpoint not complete
Current log# 4 seq# 848 mem# 0: +CORE_REDO01/coredb/onlinelog/group_4.262.812494835
Current log# 4 seq# 848 mem# 1: +CORE_REDO02/coredb/onlinelog/group_4.262.812494837
Fri Jun 07 16:00:37 2013
WARNING:io_getevents timed out 600 sec
DD測試存儲
dd if=/dev/asm_core_data01 of=/dev/null bs=1024k conut=1
hang。。。。。。。。。。。。
聯繫存儲廠商解決:
原因:存儲快照分給的自由空間太小。寫滿走後導致LUN OFFLINE
解決:擴展存儲LUN 。