由weblogic相關日誌分析後認爲2020-1-19 下午03時04分43秒宕機原因爲weblogic的線程阻塞,引起weblogic宕機;
####<2020-1-19 下午02時53分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416823035> <BEA-310002> <49% of the total memory in the server is free>
####<2020-1-19 下午02時55分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416943036> <BEA-310002> <32% of the total memory in the server is free>
####<2020-1-19 下午02時56分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417003036> <BEA-310002> <43% of the total memory in the server is free>
####<2020-1-19 下午03時00分11秒 CST> <Error> <HTTP> <VM_35_2_redhat> <nc1> <[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1579417211420> <BEA-101017> <[ServletContext@620021214[app:nc65 module:/ path: spec-version:2.5]] Root cause of ServletException.
javax.servlet.ServletException: not supported
at nc.bs.framework.comn.serv.CommonServletDispatcher.doGet(CommonServletDispatcher.java:39)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at nc.bs.framework.server.LoggerServletFilter.doFilter(LoggerServletFilter.java:35)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<2020-1-19 下午03時04分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417483038> <BEA-310002> <32% of the total memory in the server is free>
經排查後進行以下優化:
1)優化JDK內存;
2)修改StuckThreadMaxTime參數;
3) 增大線程數,防止線程阻塞問題。
優化JDK內存
setDomainEnv.sh
修改前:
if [ "${JAVA_VENDOR}" = "Sun" ] ; then
WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"
export WLS_MEM_ARGS_64BIT
WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"
export WLS_MEM_ARGS_32BIT
else
WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"
export WLS_MEM_ARGS_64BIT
WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"
export WLS_MEM_ARGS_32BIT
fi
MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"
MEM_PERM_SIZE_32BIT="-XX:PermSize=48m"
MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=256m"
MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=128m"
MEM_ARGS="-Xms512m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi -Dfile.encoding=utf-8"
修改後:
if [ "${JAVA_VENDOR}" = "Sun" ] ; then
WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"
export WLS_MEM_ARGS_64BIT
WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"
export WLS_MEM_ARGS_32BIT
else
WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"
export WLS_MEM_ARGS_64BIT
WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"
export WLS_MEM_ARGS_32BIT
fi
MEM_PERM_SIZE_64BIT="-XX:PermSize=1024m"
MEM_PERM_SIZE_32BIT="-XX:PermSize=512m"
MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=2048m"
MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=1024m"
MEM_ARGS="-Xms4096m -Xmx4096m -XX:PermSize=1024m -XX:MaxPermSize=2048m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi -Dfile.encoding=utf-8"
修改StuckThreadMaxTime參數
config.xml 配置文件
修改所有 server下stuck-thread-max-time 參數
修改前:
<stuck-thread-max-time>600</stuck-thread-max-time>
修改後:
<stuck-thread-max-time>60000</stuck-thread-max-time>
增大線程數(64位操作系統 3000 ~ 5000 左右)
在所有server中增加參數
<self-tuning-thread-pool-size-min>2000</self-tuning-thread-pool-size-min>
<self-tuning-thread-pool-size-max>4000</self-tuning-thread-pool-size-max>