informatica 拋出ora-12592 :包錯誤 排查過程和解決辦法

作者：lianghc

1.環境說明

informatica server 版本：9.6.1

informatica server上的oracle客戶端：11.2.0.3

informatica server安裝的操作系統：win server 2008 X64

oralce server :Oracle Database 11g Release 11.2.0.4.0 (oracle 環境是rac環境，有兩個節點)

oralce server 安裝在操作系統：redhat linux 6.4

2.問題描述

        最近在數據採集時總是出現ora-12592: TNS 包錯誤，出現的頻率大概每週一次，這兩天頻繁出現，錯誤出現的情況隨機不定，這一刻失敗，下一刻重啓又能成功，經過網上查詢簡單定位，此類錯誤我們初步定位爲網絡，防火牆的相關原因，諮詢過客戶網絡管理員，管理員回覆：服務器是內網連接，應該沒有問題，服務期間沒有防火牆。

3.informatica knowledge base

   在informatica 的社區找到一個kb-262530,描述如下，主要的操作是

添加客戶端和服務器sqlnet.ora文件，並重新啓動監聽器和任何現有的工作/連接： 
SQLNET.SEND_TIMEOUT=600

修改了相關的sqlnet後依然沒有效果。

4.問題排查做出的嘗試

在informatica 技術超羣中資訊相關大牛，大多依然將問題定位爲oracle 問題，其中有位說可能是informatica 服務端所依賴的oracle客戶端有問題，我覺得說的有道理，於是將客戶端換掉了，也有聲音說是informatica的client（9.6.0）和server（9.6.1）的版本不一致造成的，我想笑的版本差異應該不是問題。

這時我的同事懷疑是session和process的問題，並做出如下解決：

1.查看當前連接進程數
SQL>selectcount(*) from v$process;
2.查看連接數上限
SQL>selectvalue from v$parameter where name = 'processes';
 
3.查看當前數據庫的processes設置
SQL>show parameter processes
NAME                                TYPE        VALUE 
db_writer_processes                 integer     1
gcs_server_processes                integer     0
job_queue_processes                 integer     10
log_archive_max_processes           integer     2
processes                           integer     150 
 
SQL> show parameter sessions
NAME                                TYPE        VALUE
java_soft_sessionspace_limit        integer     0l
icense_max_sessions                integer     0
license_sessions_warning            integer     0
logmnr_max_persistent_sessions      integer     1
sessions                            integer     170
shared_server_sessions              integer
 
4.只要會話連接數超過上面的process數150或者sessions數170，再來一個的會話進程，就會產生12516錯誤。因此可以修改一下該值：
sessions=1.1*processes+5;//這個是sessions值和processes值的關係，最好按照這樣做，
因此如果要將processes數設置爲500，則sessions數必須爲1.1*500+5=555
 
SQL>alter system set processes=500 scope=spfile;
系統已更改。
SQL> alter system set sessions=555 scope=spfile;
系統已更改。
 
5.重新啓動數據庫服務即可。
SQL>shutdown immediate
```
SQL>startup

5.INFA售後 給出的答覆

努力嘗試了很多方法後，問題依然沒有進展，最後我們資訊了infa售後技術支持以及DBA

ORA-12592 錯誤是 Oracle 自身的 error message，是 TNS 軟件在檢測到不規範的包時發出的錯誤提示。
 
    由於 InformaticaPowerExchange for Oracle Database 在抽取或裝載數據到 Oracle 數據庫時，採用的是 Oracle 的 Native driver（即原生驅動），通信的建立和控制管理，都是由Oracle自身的通信機制來完成的，所以，遇到這個問題，首先考慮到的，就是從 Oracle 客戶端與服務器間的通信診斷開始。我在網上也查過，幾乎所有與ora-12592 錯誤有關的網頁，都認爲問題出在底層網絡上，或者 Oracle 通信參數的設定上。儘管，用戶的管理員一再表示這是內網環境，防火牆也未啓用，但鑑於這一明顯的錯誤提示，如果條件允許，還是請客戶的網絡管理員來配合，聯合診斷一下（比如開啓 tracing，然後借用 Oracle 客戶端通過網絡往 Oracle 數據庫服務器裏進行 insert 操作，再分析該過程中記錄下來的信息）。

6.DBA給出的答覆

關於這個錯誤和網絡關係比較大，如果服務器間沒有防火牆，檢查一下系統的防火牆是不是起來了。
關於這個錯誤：
   錯誤12592意味着所提供的信息是不可接受的two-task層。它可以發生，如果數據本身是無效的或者包實際上是失敗從Oracle Net（SQL * NET）層。 
現在，如果數據包被損壞，Oracle網絡層將報告錯誤發送信息到two-task層之前。錯誤本身發生在負責數據轉換的數據庫層中。這裏是網絡層後的層，是報告網絡問題。這可能會出現一系列的原因： 


1。查詢本身會導致失敗。 
意思是不正確的查詢。這可以被跟蹤，如果轉儲發生重複，所有出現在相同類型的選擇語句發生。 

2。有一個進程失敗（無論是客戶端/服務器進程）。 
如果應用程序掛在前端客戶端上，併發送不完整的數據到 
服務器進程。這是最難跟蹤的，因爲我們永遠不知道在服務器端錯誤轉儲時客戶端正在做什麼。 

3。在網絡層有一個間歇性的通信損耗。 
意思是一包信息沒有傳遞。當發生較大的查詢發生時，大多數情況發生，只有部分數據到達服務器進程。雙任務嘗試執行數據轉換，但信息丟失。
這也是軌道非常困難因爲唯一現實的選擇是SQL * NET跟蹤從的角度找出發生了什麼的時候出現錯誤。我們看到底層的TCP或Winsock錯誤如果Windows客戶參與。
如果是這樣的話，那些可能會導致數據包傳送的問題。 


4 在某些情況下，客戶端（可能是一個應用程序或ETL工作或進出口/ sqlloader）將間歇性失敗與ora-12592而數據庫報告任何錯誤或問題。 
a 解決方案：應用補丁18841764可在不同平臺上的11.2.0.4。 
b 解決方法：添加以下行客戶端和服務器sqlnet.ora文件，並重新啓動監聽器和任何現有的工作/連接： 
SQLNET.SEND_TIMEOUT=600

7.最終解決問題

我們在DBA 的指導下，給數據庫打了補丁18841764，在oracle的官網下載補丁：p18841764_112040_Linux-x86-64.zip

打補丁的指導文檔：

Oracle Database 11g Release 11.2.0.4.0

ORACLE NET Patch for Bug# 18841764 for Linux-x86-64 Platforms

This patch is RAC Rolling Installable  - Please read My Oracle Support Document 244241.1 https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=244241.1 
Rolling Patch - OPatch Support for RAC.

This patch is non-Data Guard Standby-First Installable - Please read My Oracle Support Note 1265700.1 https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1265700.1
Oracle Patch Assurance - Data Guard Standby-First Patch Apply for details on how to remove risk and reduce downtime when applying this patch. 

Released: Thu Jun 26 02:54:06 2014
   
This document describes how you can install the ORACLE NET interim patch for bug#  18841764 on your Oracle Database 11g Release 11.2.0.4.0


Note: This patch must be applied on the Oracle home of Oracle Database as well as Oracle Grid Infrastructure. 

 
(I) Prerequisites
--------------------
Before you install or deinstall the patch, ensure that you meet the following requirements:

Note: In case of an Oracle RAC environment, meet these prerequisites on each of the nodes.

1.	Ensure that the Oracle home on which you are installing the patch or from which you are rolling back the patch is Oracle Database 11g Release 11.2.0.4.0.

2.      Oracle recommends that you use the latest version available for 11g Release 11.2.0.4.0. If you do not have  OPatch 11g Release 11.2.0.3.0 or the latest version available for 11g Release 11.2.0.4.0,then download it from patch# 6880880 for 11.2.0.4.0 release.

	For information about OPatch documentation, including any known issues, see My Oracle Support Document 293369.1 OPatch documentation list:
	https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1

3.	Ensure that you set (as the home user) the ORACLE_HOME environment variable to the Oracle home.

4.	Ensure that the $PATH definition has the following executables: make, ar, ld and nm. The location of these executables depends on your operating system. On many operating systems, they are located in /usr/ccs/bin.	

5.	Ensure that you verify the Oracle Inventory because OPatch accesses it to install the patches. To verify the inventory, run the following command.
       $ opatch lsinventory 
	Note:
	-	If this command succeeds, it will list the Top-Level Oracle Products and one-off patches if any that are installed in the Oralce Home.
			- Save the output so you have the status prior to the patch apply.
	-	If the command displays some errors, then contact Oracle Support and resolve the issue first before proceeding further.

6.	(Only for Installation) Maintain a location for storing the contents of the patch ZIP file. In the rest of the document, this location (absolute path) is referred to as <PATCH_TOP_DIR>. Extract the contents of the patch ZIP file to the location (PATCH_TOP_DIR) you have created above. To do so, run the following command:
	$ unzip -d <PATCH_TOP_DIR>  p18841764_112040_Linux-x86-64.zip 


7.	(Only for Installation) Determine whether any currently installed interim patches conflict with this patch 18841764 as shown below:
	$ cd <PATCH_TOP_DIR>/18841764
	$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
	
	The report will indicate the patches that conflict with this patch and the patches for which the current 18841764 is a superset.
	
	Note:
	When OPatch starts, it validates the patch and ensures that there are no conflicts with the software already installed in the ORACLE_HOME. OPatch categorizes conflicts into the following types: 
	-	Conflicts with a patch already applied to the ORACLE_HOME that is a subset of the patch you are trying to apply  - In this case, continue with the patch installation because the new patch contains all the fixes from the existing patch in the ORACLE_HOME. The subset patch will automatically be rolled back prior to the installation of the new patch.
	-	Conflicts with a patch already applied to the ORACLE_HOME - In this case, stop the patch installation and contact Oracle Support Services.

8.	Ensure that you shut down all the services running from the Oracle home.
	Note:
		-	For a Non-RAC environment, shut down all the services running from the Oracle home. 
		-	For a RAC environment, shut down all the services (database, ASM, listeners, nodeapps, and CRS daemons) running from the Oracle home of the node you want to patch. After you patch this node, start the services on this node.Repeat this process for each of the other nodes of the Oracle RAC system. OPatch is used on only one node at a time.
  
9.      oracle executible has to be relinked and client shared library to be relinked



(II) Installation  
-----------------
To install the patch, follow these steps:

1.	Set your current directory to the directory where the patch is located and then run the OPatch utility by entering the following commands:

	$ cd <PATCH_TOP_DIR>/18841764

	$ opatch apply

2.	Verify whether the patch has been successfully installed by running the following command:

	$ opatch lsinventory

3.	Start the services from the Oracle home.


  
(III) Postinstallation
---------------------- 

none .


(IV) Deinstallation
----------------------
Ensure to follow the Prerequsites (Section I). To deinstall the patch, follow these steps:

1.	Deinstall the patch by running the following command:

	$ opatch rollback -id 18841764
  
2.	Start the services from the Oracle home.

3.	Ensure that you verify the Oracle Inventory and compare the output with the one run before the patch installation and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command:

	$ opatch lsinventory



 
(V)Postdeinstallation
--------------------------


After you deinstall the patch, reload the packages into the Oracle Database.
none .


 (VI) Bugs Fixed by This Patch
---------------------------------
The following are the bugs fixed by this patch:
  18841764: ORA-12592 TNS:BAD PACKET OR ORA-3137 OR ORA-1


--------------------------------------------------------------------------
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
--------------------------------------------------------------------------

8.打完補丁遇到ora-12514錯誤

補丁 18841764 打完之後，數據庫啓動監聽拋出ora-12514錯誤，在網上搜索都是說需要在Network/admin目錄下找到listener.ora ，將標紅添加進文件裏（ORCL爲監聽的實例名）。

# listener.ora Network Configuration File: D:\..\network\admin\listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = E:\..)
      (PROGRAM = extproc)
    )
   (SID_DESC =
    (GLOBAL_DBNAME = ORCL)
    (ORACLE_HOME = D:\..)
    (SID_NAME = ORCL)
    )
 )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
      (ADDRESS = (PROTOCOL = TCP)(HOST = onest)(PORT = 1521))
    )
  )

而實際上是因爲打完補丁導致數據庫存儲的權限錯誤

9.修復數據庫存儲的權限錯誤

oracle 報錯如下：

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=10.0.60.124)(PORT=1521))' SCOPE=MEMORY SID='oradc1';
NOTE: Loaded library: System 
ORA-15025: could not open disk "/dev/raw/raw1"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied

---解決辦法---
[root@datacenter1 ~]# ls -ltr /dev/raw/raw1
crw-rw---- 1 grid asmadmin 162, 1 Aug 11 15:05 /dev/raw/raw1

[grid@datacenter1 ~]$ cd /u01/product/grid/11.2.0/bin/
[grid@datacenter1 bin]$ ./setasmgidwrap o=/u01/product/oracle/11.2.0/db_1/bin/oracle