環境：HP-UNIX , oracle 11.2.0.3 版本的三 node RAC
問題描述：問題發生前幾天公司機房網絡升級萬兆網絡，整個數據庫服務器停機。網絡升級完成後，
RAC環境開啓正常。三天後應用的同事反映部分應用不能連接數據庫，應用後臺報：Listener refused
the connection with the following error:ORA-12514, TNS:listener does not currently know of
service requested in connect descriptor
嘗試用本地的plsqldev 工具連接，也提示 ora-12505。
注：應用與本地plsqldev 的連接字符串配置都使用的是 scan ip
思路：
1、檢查數據庫實例是否正常；
2、檢查service是否正常；
3、檢查監聽是否正常；
4、是否超過最大連接數
5、網絡連接是否正常
6、查alert，listener ，集羣，OS 日誌
線索：
所有節點實例都正常，服務正常(共三個service，srv1,srv2,啓在node1 上，srv3 在node3 上).
監聽都是有啓動，連接數足夠充分。服務器端及客戶端 ping scan IP 都能ping 通。
ping scan IP
# ping 10.1.194.208
PING 10.1.194.208: 64 byte packets
64 bytes from 10.1.194.208: icmp_seq=0. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=1. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=2. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=3. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=4. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=5. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=6. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=7. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=8. time=0. ms
----10.1.194.208 PING Statistics----
9 packets transmitted, 9 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/0/0
telnet scan ip :
# telnet 10.1.194.208 1521
Trying...
telnet: Unable to connect to remote host: Connection refused
發現不通
數據庫 alert 日誌沒有任何錯誤。
監聽log ：
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:41 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44171)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44174)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(SERVICE_NAME=querydb)(CID=(PROGRAM=python)(HOST=monitorcenter.yto56.com.cn)(USER=zabbix))) * (
ADDRESS=(PROTOCOL=tcp)(HOST=10.1.203.87)(PORT=60204)) * establish * querydb * 0
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv1)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.82)(PORT=46107)) * establish * srv1 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45333)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Sat Feb 08 13:20:43 2014
08-FEB-2014 13:20:43 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45335)) * establish * srv2 * 12514
有大量這類錯誤。
OS日誌 無錯誤。
CRS日誌正常。
crs_stat -t -v 發現所有RAC資源都正常。
鑑於以上原因難以確定，嘗試重啓監聽，未能解決問題，步驟就不貼了。
於是想到重啓 scan IP 。
# . /home/grid/.profile
# crsctl stop res ora.scan1.vip -f
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'expquery03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'expquery03' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'expquery03'
CRS-2677: Stop of 'ora.scan1.vip' on 'expquery03' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 OFFLINE OFFLINE
# crsctl start res ora.scan1.vip
CRS-2672: Attempting to start 'ora.scan1.vip' on 'expquery02'
CRS-2676: Start of 'ora.scan1.vip' on 'expquery02' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE expquery02
但發現還是不能解決問題。
於是，想到網管必修的終極殺招————重啓。
於是就把服務切到其它節點，先重啓node1 ，咦，神奇的發現，重啓之後，竟然可以連接了。
---------
廢話這麼多，其實還是沒有找到真正的原因。關於這個問題，本人懷疑是因爲網絡的異常的緣故，導致scan ip 資源異常。
問題描述：問題發生前幾天公司機房網絡升級萬兆網絡，整個數據庫服務器停機。網絡升級完成後，
RAC環境開啓正常。三天後應用的同事反映部分應用不能連接數據庫，應用後臺報：Listener refused
the connection with the following error:ORA-12514, TNS:listener does not currently know of
service requested in connect descriptor
嘗試用本地的plsqldev 工具連接，也提示 ora-12505。
注：應用與本地plsqldev 的連接字符串配置都使用的是 scan ip
思路：
1、檢查數據庫實例是否正常；
2、檢查service是否正常；
3、檢查監聽是否正常；
4、是否超過最大連接數
5、網絡連接是否正常
6、查alert，listener ，集羣，OS 日誌
線索：
所有節點實例都正常，服務正常(共三個service，srv1,srv2,啓在node1 上，srv3 在node3 上).
監聽都是有啓動，連接數足夠充分。服務器端及客戶端 ping scan IP 都能ping 通。
ping scan IP
# ping 10.1.194.208
PING 10.1.194.208: 64 byte packets
64 bytes from 10.1.194.208: icmp_seq=0. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=1. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=2. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=3. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=4. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=5. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=6. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=7. time=0. ms
64 bytes from 10.1.194.208: icmp_seq=8. time=0. ms
----10.1.194.208 PING Statistics----
9 packets transmitted, 9 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/0/0
telnet scan ip :
# telnet 10.1.194.208 1521
Trying...
telnet: Unable to connect to remote host: Connection refused
發現不通
數據庫 alert 日誌沒有任何錯誤。
監聽log ：
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:41 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44171)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.69)(PORT=44174)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(SERVICE_NAME=querydb)(CID=(PROGRAM=python)(HOST=monitorcenter.yto56.com.cn)(USER=zabbix))) * (
ADDRESS=(PROTOCOL=tcp)(HOST=10.1.203.87)(PORT=60204)) * establish * querydb * 0
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv1)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.82)(PORT=46107)) * establish * srv1 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
08-FEB-2014 13:20:42 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45333)) * establish * srv2 * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Sat Feb 08 13:20:43 2014
08-FEB-2014 13:20:43 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=wasadmin))(SERVER=DEDICATED)(SERVICE_NAME=srv2)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.1.194.81)(PORT=45335)) * establish * srv2 * 12514
有大量這類錯誤。
OS日誌 無錯誤。
CRS日誌正常。
crs_stat -t -v 發現所有RAC資源都正常。
鑑於以上原因難以確定，嘗試重啓監聽，未能解決問題，步驟就不貼了。
於是想到重啓 scan IP 。
# . /home/grid/.profile
# crsctl stop res ora.scan1.vip -f
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'expquery03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'expquery03' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'expquery03'
CRS-2677: Stop of 'ora.scan1.vip' on 'expquery03' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 OFFLINE OFFLINE
# crsctl start res ora.scan1.vip
CRS-2672: Attempting to start 'ora.scan1.vip' on 'expquery02'
CRS-2676: Start of 'ora.scan1.vip' on 'expquery02' succeeded
# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora.DATA_DG.dg ora....up.type 0/5 0/ ONLINE ONLINE expquery01
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE expquery01
ora....N1.lsnr ora....er.type 0/5 0/0 OFFLINE OFFLINE
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE expquery01
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE expquery01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE expquery01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE expquery01
ora....y01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y01.ons application 0/3 0/0 ONLINE ONLINE expquery01
ora....y01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE expquery02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE expquery02
ora....y02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y02.ons application 0/3 0/0 ONLINE ONLINE expquery02
ora....y02.vip ora....t1.type 0/0 1/0 ONLINE ONLINE expquery02
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE expquery03
ora....03.lsnr application 0/5 0/0 ONLINE ONLINE expquery03
ora....y03.gsd application 0/5 0/0 OFFLINE OFFLINE
ora....y03.ons application 0/3 0/0 ONLINE ONLINE expquery03
ora....y03.vip ora....t1.type 0/0 0/0 ONLINE ONLINE expquery03
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE expquery01
ora.oc4j ora.oc4j.type 0/1 0/2 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE expquery01
ora.querydb.db ora....se.type 0/2 0/1 ONLINE ONLINE expquery02
ora....rc1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc2.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery01
ora....rc3.svc ora....ce.type 0/0 0/0 ONLINE ONLINE expquery03
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE expquery02
但發現還是不能解決問題。
於是，想到網管必修的終極殺招————重啓。
於是就把服務切到其它節點，先重啓node1 ，咦，神奇的發現，重啓之後，竟然可以連接了。
---------
廢話這麼多，其實還是沒有找到真正的原因。關於這個問題，本人懷疑是因爲網絡的異常的緣故，導致scan ip 資源異常。
