hi all
今天在回滾xcrmdb41上的補丁重啓集羣時gipcd.trc報如下錯誤，集羣無法正常啓動
2019-11-12 17:10:09.019842 :GIPCXCPT:3: gipcmodNetworkProcessBind: failed to bind endp 600000000036c760 [0000000000000030] { gipcEndpoint : localAddr 'ipc://xcrmdb41_DBG_GIPCD', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0
, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0000000000000000, ready 0, wobj 600000000034de30, sendp 60000000003a1020 status 13flags 0x20008612, flags-2 0x0, usrFlags 0x20020 }, addr 600000000036c9a0 [0000000000000032] { gipcAddres
s : name 'ipc://xcrmdb41_DBG_GIPCD', objFlags 0x0, addrFlags 0x5 }
2019-11-12 17:10:09.019896 :GIPCXCPT:3: gipcmodNetworkProcessBind: slos op : sgipcnDSBindHelper
2019-11-12 17:10:09.019901 :GIPCXCPT:3: gipcmodNetworkProcessBind: slos dep : Address already in use (226)
2019-11-12 17:10:09.019906 :GIPCXCPT:3: gipcmodNetworkProcessBind: slos loc : bind
2019-11-12 17:10:09.019910 :GIPCXCPT:3: gipcmodNetworkProcessBind: slos info: failed to /tmp/.oracle/ora_gipc_xcrmdb41_DBG_GIPCD
2019-11-12 17:10:09.020135 :GIPCXCPT:3: gipcListenF [clsdms_thdmai : clsdms.c : 642]: EXCEPTION[ ret gipcretAddressInUse (20) ] failed to listen on endp 600000000036c760 [0000000000000030] { gipcEndpoint : localAddr 'ipc://xcrmdb41_DBG
_GIPCD', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0000000000000000, ready 0, wobj 600000000034de30, sendp 60000000003a1020 status 13flags 0x20008612, flags-2 0x0, usrFla
gs 0x20020 }, flags 0x0
2019-11-12 17:10:09.020146 : CLSDMT:3: Fail to listen to ipc://xcrmdb41_DBG_GIPCD
2019-11-12 17:10:09.020151 : CLSDMT:3: Terminating process
2019-11-12 17:10:09.020159 : GIPCD:3: gipcd_ExitCB: Received a shutdown message from agent framework
2019-11-12 17:10:09.020184 : GIPCLIB:3: gipclibMapSearch: gipcMapSearch() -> gipcMapGetNodeAddr() failed: ret:gipcretKeyNotFound (36), ht:6000000000011590, idxPtr:60000000002008d0, key:9fffffffff02fc88, flags:0x0
2019-11-12 17:10:09.020198 :GIPCXCPT:3: gipcObjectLookupF [gipcPostF : gipc.c : 2251]: search found no matching oid 0000000000000000, ret gipcretKeyNotFound (36), ret gipcretInvalidObject (3)
2019-11-12 17:10:09.020222 :GIPCXCPT:3: gipcPostF [gipcd_ExitCB : gipcd.c : 1054]: EXCEPTION[ ret gipcretInvalidObject (3) ] failed to post obj 0000000000000000, flags 0x0
2019-11-12 17:10:09.020230 : GIPCLIB:3: gipclibMapSearch: gipcMapSearch() -> gipcMapGetNodeAddr() failed: ret:gipcretKeyNotFound (36), ht:6000000000011590, idxPtr:60000000002008d0, key:9fffffffff02fc88, flags:0x0
2019-11-12 17:10:09.020238 :GIPCXCPT:3: gipcObjectLookupF [gipcPostF : gipc.c : 2251]: search found no matching oid 0000000000000000, ret gipcretKeyNotFound (36), ret gipcretInvalidObject (3)
2019-11-12 17:10:09.020254 :GIPCXCPT:3: gipcPostF [gipcd_ExitCB : gipcd.c : 1055]: EXCEPTION[ ret gipcretInvalidObject (3) ] failed to post obj 0000000000000000, flags 0x0
2019-11-12 17:10:09.020261 : GIPCLIB:3: gipclibMapSearch: gipcMapSearch() -> gipcMapGetNodeAddr() failed: ret:gipcretKeyNotFound (36), ht:6000000000011590, idxPtr:60000000002008d0, key:9fffffffff02fc88, flags:0x0
2019-11-12 17:10:09.020269 :GIPCXCPT:3: gipcObjectLookupF [gipcPostF : gipc.c : 2251]: search found no matching oid 0000000000000000, ret gipcretKeyNotFound (36), ret gipcretInvalidObject (3)
2019-11-12 17:10:09.020285 :GIPCXCPT:3: gipcPostF [gipcd_ExitCB : gipcd.c : 1056]: EXCEPTION[ ret gipcretInvalidObject (3) ] failed to post obj 0000000000000000, flags 0x0
Trace file /oracle/app/grid/diag/crs/xcrmdb41/crs/trace/gipcd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
2019-11-12 17:11:09.028848 : GIPCD:3: gipcd_ExitCB: one or more of gipcdThreads failed to come into offline in 60 seconds of time, aborting the gipcd process
CLSB:3: Oracle Clusterware infrastructure error in GIPCD (OS PID 11463): Fatal signal 6 has occurred in program gipcd thread 3; nested signal count is 1
Incident 89 created, dump file: /oracle/app/grid/diag/crs/xcrmdb41/crs/incident/incdir_89/gipcd_i89.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Trace file /oracle/app/grid/diag/crs/xcrmdb41/crs/trace/gipcd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
CLSB:1: Argument count (argc) for this daemon is 1
CLSB:1: Argument 0 is: /oracle/app/12.1.0/grid/bin/gipcd.bin
經與mos比對，符合文檔gipcd not coming up (文檔 ID 2034669.1)所述
參考solution，使用root執行以下命令
rm -rf /usr/tmp/.oracle/* /var/tmp/.oracle/* /tmp/.oracle/*
重啓crs後，gipcd.trc繼續報錯
2019-11-12 17:32:32.661755 : GIPCD:1: gipcdMain: gipcd Started
2019-11-12 17:32:32.662486 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name CLNTCOMM
2019-11-12 17:32:32.662502 : GIPCD:1: gipcdConfig_vir2phyID_setunit: to set trace level to 2 for gipclib module CLNTCOMM
2019-11-12 17:32:32.662547 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name NODECOMM
2019-11-12 17:32:32.662552 : GIPCD:1: gipcdConfig_vir2phyID_setunit: to set trace level to 2 for gipclib module NODECOMM
2019-11-12 17:32:32.662567 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name MONTCOMM
2019-11-12 17:32:32.662572 : GIPCD:1: gipcdConfig_vir2phyID_setunit: to set trace level to 2 for gipclib module MONTCOMM
2019-11-12 17:32:32.662587 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name GIPCD
2019-11-12 17:32:32.662602 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name GIPCDNDE
2019-11-12 17:32:32.662612 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name GIPCDCLT
2019-11-12 17:32:32.662622 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 2, module name GIPCDMON
2019-11-12 17:32:32.662632 : GIPCD:1: gipcdConfig_vir2phyID_setunit: level 0, module name CLSINET
2019-11-12 17:32:33.036939 : CLSDMT:3: PID for the Process [15522], connkey 13
2019-11-12 17:32:33.557358 : GIPCD:1: gipcd_CLSDMInit: clsdm inited
2019-11-12 17:32:33.557366 : GIPCD:3: gipcd_CheckCB: GIPC Daemon is NOT yet READY for business ...
2019-11-12 17:32:34.498251 : GPNP:1: clsgpnp_getCachedProfileEx: [at clsgpnp.c:689] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00107:)Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
2019-11-12 17:32:34.498264 : GPNP:1: clsgpnp_getCachedProfileEx: [at clsgpnp.c:700] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00108:)Failed to get offline GPnP service profile.
2019-11-12 17:32:34.565701 : GIPCD:3: gipcd_CheckCB: GIPC Daemon is NOT yet READY for business ...
2019-11-12 17:32:35.051922 : GIPCLIB:1: gipclibGetClusterGuid: retrieved cluster guid 759bf747111d6f2dbfb3f6d80570e7a4
2019-11-12 17:32:35.575283 : GIPCD:3: gipcd_CheckCB: GIPC Daemon is NOT yet READY for business ...
2019-11-12 17:32:35.727462 : GPNP:1: clsgpnp_getCachedProfileEx: [at clsgpnp.c:689] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00107:)Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
2019-11-12 17:32:35.727469 : GPNP:1: clsgpnp_getCachedProfileEx: [at clsgpnp.c:700] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00108:)Failed to get offline GPnP service profile.
2019-11-12 17:32:35.739680 : GIPCLIB:1: gipclibGetClusterGuid: retrieved cluster guid 759bf747111d6f2dbfb3f6d80570e7a4
2019-11-12 17:32:35.814649 :GIPCDCLT:1: gipcdMain: acquired exclusive lock on existing map smem://xcrmdb41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4
2019-11-12 17:32:35.814712 :GIPCDCLT:1: gipcdMain: It is fresh start, creating brand new shared memory
2019-11-12 17:32:35.814784 :GIPCXCPT:1: gipcmodSharedSegCreate: failed create smem 6000000000252d80 [0000000000000174] { gipcMemory : name 'smem://xcrmdb41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4', len 0, objFlags 0x0 }, name 'xcrmdb
41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4-0000000000', len 10485960
2019-11-12 17:32:35.814836 :GIPCXCPT:1: gipcmodSharedSegCreate: slos op : sgipcCreateSharedMemory
2019-11-12 17:32:35.814840 :GIPCXCPT:1: gipcmodSharedSegCreate: slos dep : Not owner (1)
2019-11-12 17:32:35.814845 :GIPCXCPT:1: gipcmodSharedSegCreate: slos loc : unlink
2019-11-12 17:32:35.814850 :GIPCXCPT:1: gipcmodSharedSegCreate: slos info: failed to cleanup map /var/opt/oracle/maps/xcrmdb41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4-0000000000
2019-11-12 17:32:35.814924 :GIPCXCPT:1: gipcMemoryF [gipcdMain : gipcd.c : 549]: EXCEPTION[ ret gipcretFail (1) ] failed memory create ctx 6000000000193a20 [00000000000000e7] { gipcContext : traceLevel 2, fieldLevel 0x0, numDead 0, num
Pending 0, numZombie 0, numObj 1, numWait 0, numReady 0, wobj 600000000003a3d0, hgid 00000000000000e8, flags 0x1a, objFlags 0x0 }, name 'smem://xcrmdb41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4', flags 0x203400
2019-11-12 17:32:35.814932 :GIPCDCLT:1: gipcdMain: Failed to create new gipcMemory() 1
2019-11-12 17:32:35.825389 : GIPCD:1: gipcdMain: All threads terminated
2019-11-12 17:32:35.825450 : GIPCD:1: gipcdMain: GIPCD terminated
將/var/opt/oracle/maps/xcrmdb41_gipcd1318_759bf747111d6f2dbfb3f6d80570e7a4-0000000000清理掉後，gipc進程可正常啓動