最近在生產庫上碰到這個錯誤。從錯誤號可以辨別是因爲存儲空間的問題。
由於生產環境是RAC，數據文件存於存儲上面。於是查看存儲空間情況。發現存儲上面的空間是足夠的。而且看了所有表空間狀態都是對的，沒有異常。無從查起，於是看節點的alert日誌。
終於在其中一個節點上看到如下信息：
- ORA-01114: <BD><AB><BF><E9>寫<C8><EB><CE>募<FE> 時<B3><F6><CF><D6> IO <B4><ED><CE><F3> (<BF><E9> # )
- Tue Apr 16 18:34:36 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_10993.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Could not open audit file: /u01/app/oracle/admin/center/adump/center2_ora_10993_1.aud
- Retry Iteration No: 1 OS Error: 28
- Retry Iteration No: 2 OS Error: 28
- Retry Iteration No: 3 OS Error: 28
- Retry Iteration No: 4 OS Error: 28
- Retry Iteration No: 5 OS Error: 28
- OS Audit file could not be created; failing after 5 retries
- Tue Apr 16 18:35:14 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_11034.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Could not open audit file: /u01/app/oracle/admin/center/adump/center2_ora_11034_1.aud
- Retry Iteration No: 1 OS Error: 28
- Retry Iteration No: 2 OS Error: 28
- Retry Iteration No: 3 OS Error: 28
- Retry Iteration No: 4 OS Error: 28
- Retry Iteration No: 5 OS Error: 28
- OS Audit file could not be created; failing after 5 retries
- Tue Apr 16 18:35:26 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_gcr0_10269.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Tue Apr 16 18:37:31 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_3430.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
從上面可以看到很重要的字眼： No space left on device 。於是用df查看磁盤空間，發現報以上錯誤的這個節點上面的根目錄空間已用完。
於是開始清理磁盤空間，發現是由於備份文件存放太多，沒有定期清理而把空間用完。清理空間後問題得以解決。