作者 阿九【轉載時請務必以超鏈接形式標明文章原始出處和作者信息】
今天應用的人反映個稅系統無法登陸，查看weblogic後臺日誌，發現有兩個比較多的報錯 <BEA-000337> <BEA-101215>
錯誤日誌：
|
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '3' for queue: 'weblogic.kernel.Default' has been busy for "909" seconds working on the request
"Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '8' for queue: 'weblogic.kernel.Default' has been busy for "907" seconds working on the request
"Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '15' for queue: 'weblogic.kernel.Default' has been busy for "908" seconds working on the request
"Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "873" seconds working on the request
"Http Request: /bimsweb/slpt/accept/queryDeclareAcceptInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '28' for queue: 'weblogic.kernel.Default' has been busy for "926" seconds working on the request
"Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '39' for queue: 'weblogic.kernel.Default' has been busy for "909" seconds working on the request
"Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午09時58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '46' for queue: 'weblogic.kernel.Default' has been busy for "929" seconds working on the request
"Http Request: /bimsweb/slpt/accept/queryDeclareAcceptInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<2012-11-23 上午10時00分59秒 CST> <Error> <HTTP> <sxdsrjk> <myserver> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101215> <Malformed Request "null". Request parsing failed, Code: -10>
|
日誌顯示，新連接的會話由於系統繁忙導致等待。
檢查後臺數據庫情況，在告警日誌中沒有記錄任何異常報錯，查看等待會話，發現有大量的
“global cache cr request”、“db file scattered read”、“db file sequential read”、“buffer busy waits”等待事件。
通過操作系統topas命令也看到磁盤使用率100%，CPU使用率很高，52%在使用、36%在等待、CPU沒有空閒。
通過查看後臺視圖發現大量等待會話執行的SQL語句均爲“SELECT”語句，將這些會話殺死後，系統恢復正常。
初步判斷是這些SQL語句大量的、頻繁的執行導致數據庫出現過多的“global cache cr request”、“db file scattered read”、“db file sequential read” 等待事件，導致後續連接的會話不可用。
相關SQL語句，還要在跟蹤分析。
