我們中心繫統使用nbu進行備份管理，今日發現有備份任務失敗，發現了status 6的錯誤，下面是本人簡單排查的一個過程
可以查看一下nbu備份的詳細信息
記得在賽門鐵克的nbu手冊裏面有一段記錄是這樣的：
那麼我們可以查看我們的Oracle主機裏備份腳本所輸出的log日誌進行下一步的診斷~
jszwdb01 [/]$ cd /oracle/nbu_scripts
jszwdb01 [/oracle/nbu_scripts]$ ls
hot_database_backup.sh hot_database_backup_arch.sh
hot_database_backup.sh.out hot_database_backup_arch.sh.out
jszwdb01 [/oracle/nbu_scripts]$ tail -n 200 hot_database_backup.sh.out
input datafile fno=00043 name=/dev/rjs_settindex12
input datafile fno=00089 name=/dev/rjs_settle46
input datafile fno=00135 name=/dev/rjs_statdata031
input datafile fno=00181 name=/dev/rjs_statdata077
input datafile fno=00221 name=/dev/rjs_statindex018
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34031_1_793845665 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:11
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00044 name=/dev/rjs_settindex13
input datafile fno=00090 name=/dev/rjs_settle47
input datafile fno=00136 name=/dev/rjs_statdata032
input datafile fno=00182 name=/dev/rjs_statdata078
input datafile fno=00222 name=/dev/rjs_statindex019
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34032_1_793845711 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00045 name=/dev/rjs_settle02
input datafile fno=00091 name=/dev/rjs_settle48
input datafile fno=00137 name=/dev/rjs_statdata033
input datafile fno=00183 name=/dev/rjs_statdata079
input datafile fno=00223 name=/dev/rjs_statindex020
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34033_1_793845736 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00046 name=/dev/rjs_settle03
input datafile fno=00092 name=/dev/rjs_settle49
input datafile fno=00138 name=/dev/rjs_statdata034
input datafile fno=00184 name=/dev/rjs_statdata080
input datafile fno=00002 name=/dev/rjs_undotbs01
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34034_1_793845772 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00047 name=/dev/rjs_settle04
input datafile fno=00093 name=/dev/rjs_settle50
input datafile fno=00139 name=/dev/rjs_statdata035
input datafile fno=00185 name=/dev/rjs_statdata081
input datafile fno=00224 name=/dev/rjs_undotbs02
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34035_1_793845797 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:11
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00005 name=/dev/rjs_commdata001
input datafile fno=00051 name=/dev/rjs_settle08
input datafile fno=00097 name=/dev/rjs_settle54
input datafile fno=00143 name=/dev/rjs_statdata039
input datafile fno=00001 name=/dev/rjs_system
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34036_1_793845833 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:11
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00006 name=/dev/rjs_settindex01
input datafile fno=00052 name=/dev/rjs_settle09
input datafile fno=00098 name=/dev/rjs_settle55
input datafile fno=00144 name=/dev/rjs_statdata040
input datafile fno=00003 name=/dev/rjs_sysaux
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34037_1_793845869 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00007 name=/dev/rjs_settle01
input datafile fno=00053 name=/dev/rjs_settle10
input datafile fno=00099 name=/dev/rjs_settle56
input datafile fno=00145 name=/dev/rjs_statdata041
input datafile fno=00226 name=/dev/rjs_sysaux01
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34038_1_793845904 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00008 name=/dev/rjs_statdata001
input datafile fno=00054 name=/dev/rjs_settle11
input datafile fno=00100 name=/dev/rjs_settle57
input datafile fno=00146 name=/dev/rjs_statdata042
input datafile fno=00227 name=/dev/rjs_sysaux02
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34039_1_793845930 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00009 name=/dev/rjs_statindex001
input datafile fno=00055 name=/dev/rjs_settle12
input datafile fno=00101 name=/dev/rjs_settle58
input datafile fno=00147 name=/dev/rjs_statdata043
input datafile fno=00225 name=/dev/rjs_rman
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34040_1_793845965 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00010 name=/dev/rjs_commindex01
input datafile fno=00056 name=/dev/rjs_settle13
input datafile fno=00102 name=/dev/rjs_settle59
input datafile fno=00148 name=/dev/rjs_statdata044
input datafile fno=00004 name=/dev/rjs_user
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34041_1_793845991 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00048 name=/dev/rjs_settle05
input datafile fno=00094 name=/dev/rjs_settle51
input datafile fno=00140 name=/dev/rjs_statdata036
input datafile fno=00186 name=/dev/rjs_statdata082
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34042_1_793846027 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00049 name=/dev/rjs_settle06
input datafile fno=00095 name=/dev/rjs_settle52
input datafile fno=00141 name=/dev/rjs_statdata037
input datafile fno=00187 name=/dev/rjs_statdata083
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34043_1_793846052 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00050 name=/dev/rjs_settle07
input datafile fno=00096 name=/dev/rjs_settle53
input datafile fno=00142 name=/dev/rjs_statdata038
input datafile fno=00188 name=/dev/rjs_statdata084
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34044_1_793846088 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch00: starting incremental level 1 datafile backupset
channel ch00: specifying datafile(s) in backupset
including current control file in backupset
channel ch00: starting piece 1 at 13-SEP-12
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34045_1_793846113 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:01
channel ch01: starting incremental level 1 datafile backupset
channel ch01: specifying datafile(s) in backupset
including current SPFILE in backupset
channel ch01: starting piece 1 at 13-SEP-12
channel ch00: finished piece 1 at 13-SEP-12
piece handle=bk_34046_1_793846149 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:01:01
channel ch01: finished piece 1 at 13-SEP-12
piece handle=bk_34047_1_793846175 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 00:01:00
Finished backup at 13-SEP-12
sql statement: alter system archive log current
released channel: ch00
released channel: ch01
allocated channel: ch00
channel ch00: sid=5693 devtype=SBT_TAPE
channel ch00: Veritas NetBackup for Oracle - Release 7.0 (2010070805)
sent command to channel: ch00
Starting backup at 13-SEP-12
current log archived
released channel: ch00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 09/13/2012 00:50:54
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /ora_arch/1_143570_734312913.arc
ORA-27037: unable to obtain file status
IBM AIX RISC System/6000 Error: 2: A file or directory in the path name does not exist.
Additional information: 3
RMAN> RMAN>
Recovery Manager complete.
Script /oracle/nbu_scripts/hot_database_backup.sh
==== ended in error on Thu Sep 13 00:50:55 GMT+08:00 2012 ====
發現了大量的RMAN備份error，其中我們關注一下ora-19625
jszwdb01 [/oracle/nbu_scripts]$ cd /ora_arch
jszwdb01 [/ora_arch]$ ls
1_144167_734312913.arc 1_144179_734312913.arc 1_144191_734312913.arc 1_144203_734312913.arc 1_144215_734312913.arc 1_144227_734312913.arc
1_144168_734312913.arc 1_144180_734312913.arc 1_144192_734312913.arc 1_144204_734312913.arc 1_144216_734312913.arc addmrpt_1_14314_14316.txt
1_144169_734312913.arc 1_144181_734312913.arc 1_144193_734312913.arc 1_144205_734312913.arc 1_144217_734312913.arc awrrpt_1_13612_13625.html
1_144170_734312913.arc 1_144182_734312913.arc 1_144194_734312913.arc 1_144206_734312913.arc 1_144218_734312913.arc awrrpt_1_13614_13617.html
1_144171_734312913.arc 1_144183_734312913.arc 1_144195_734312913.arc 1_144207_734312913.arc 1_144219_734312913.arc awrrpt_1_13618_13621.html
1_144172_734312913.arc 1_144184_734312913.arc 1_144196_734312913.arc 1_144208_734312913.arc 1_144220_734312913.arc full_str.dump
1_144173_734312913.arc 1_144185_734312913.arc 1_144197_734312913.arc 1_144209_734312913.arc 1_144221_734312913.arc jzjs_ora_4588116.trc
1_144174_734312913.arc 1_144186_734312913.arc 1_144198_734312913.arc 1_144210_734312913.arc 1_144222_734312913.arc lost+found
1_144175_734312913.arc 1_144187_734312913.arc 1_144199_734312913.arc 1_144211_734312913.arc 1_144223_734312913.arc ora_back_bitmap
1_144176_734312913.arc 1_144188_734312913.arc 1_144200_734312913.arc 1_144212_734312913.arc 1_144224_734312913.arc
1_144177_734312913.arc 1_144189_734312913.arc 1_144201_734312913.arc 1_144213_734312913.arc 1_144225_734312913.arc
1_144178_734312913.arc 1_144190_734312913.arc 1_144202_734312913.arc 1_144214_734312913.arc 1_144226_734312913.arc
jszwdb01 [/ora_arch]$ ls
jszwdb01 [/ora_arch]$ ls 1_143570_734312913.arc
ls: 0653-341 The file 1_143570_734312913.arc does not exist.
原因找到了，操作系統下刪除了這個歸檔日誌，這個日誌還沒備份過，rman去備份的時候，找不到這個歸檔就出錯了，Oracle在controlfile中記錄着archivelog信息，當我們把這些物理文件delete掉或異常變動後，在controlfile中仍然記錄着這些archivelog的信息，當我們手工清除archive目錄下的文件後，這些記錄並沒有被我們從controlfile中清除掉，此時Oracle並不知道這些文件已經不存在。我們進入RMAN中執行crosscheck archivelog all；進行交叉檢驗，讓oracle控制文件同步歸檔文件的信息。