Oracle 11.2 OGG-01117 Received signal: OMVS Shutdown (33)

AIX: 'GGS ERROR 190 ... : Received Signal: OMVS Shutdown (33).' (Doc ID 969596.1)

 

This message means that AIX had a system-wide paging problem, which it reported to the GoldenGate component, in this case the Collector process. The AIX system sent an unexpected UNIX SIGDANGER signal to the Collector process. Solution Details:Current GoldenGate versions are aware of this UNIX signal only at shutdown of a z/OS UNIX System Services (OMVS) environment. However, AIX sends SIGDANGER before an imminent system failure that is caused by a shortage of free paging-space pages. 

 

metalink上的文檔說是pagespace的問題，看了一下errpt，發現確實有相關報錯。

 

進一步用topas發現

oraagent的pagespace用的非常高，高過同類進程100倍以上。

 

於是又發現如下文章：

 

• <GI_HOME>/log/<node>/agent/crsd/oraagent_grid/oraagent_grid.log

2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.    ====>> resource missing
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0 
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} ASM Dedicated Thread }

 

• "<GI_HOME>/bin/crsctl stat res -t" shows no 'ora.LISTENER.lsnr' resource

 

• "<GI_HOME>/bin/crsctl stat res ora.asm -p" shows ora.asm depends on 'ora.LISTENER.lsnr' resource

NAME=ora.asm
....
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)

  

 

Cause

Dependent resource ' ora.LISTENER.lsnr' was removed.

The issue is being investigated in bug 18261183 and bug 18427298

Solution

While the bugs are still being worked by Development, the workaround is to add the missing "ora.LISTENER.lsnr" listener or set correct "listener" resource for ora.asm as grid user:

In the case of "ora.LISTENER.lsnr" is gone, add the listener back:

$ srvctl add listener -l LISTENER -p <port>

 

In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.ASM_LISTENER.lsnr"

$ srvctl modify asm -l ASM_LISTENER

 

確實如上文所說，缺少ora.LISTENER.lsnr，其實名字改了，但asm依賴的listener名字還是LISTENER

嘗試srvctl modify asm -l <mylistener>

 

發現報錯：

PRCA-1032: ASM Listener LISTENER Does Not Exist On 11.2 SIHA/OHAS Configuration.

 

只好繼續查，找到 (Doc ID 1428284.1)

發現要解決這個問題，需要先添加名爲listener的listener，再改，改完只好再刪除這個新加的listener

 

srvctl add listener -l LISTENER -p 1522                        因爲1521的已經被佔用了。

srvctl modify asm -l <mylistener>    

現在可以了。

srvctl remove listener -l LISTENER

 

再將has重啓一下，發現oraagent的pagespace就正常了。