tomcat進程意外掛掉(crash)(未解決的問題)

問題:tomcat在運行過程中會莫名掛掉,表現如被Killed -9,系統中沒有任何日誌(命令行無Kill指令記錄、/var/log/message中無OOM Killer日誌、項目本地目錄無core dump日誌、無java crash日誌、應用中記錄了eixt操作,日誌中沒有看到該記錄)。不知道大夥兒遇到過這種情況不,有啥建議不?

下面是我排查的情況:
軟件環境
$ uname -a
Linux server-224 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 6.3 (Final)

tomcat 7.0.39
$ java -version
java version "1.7.0_15"
Java(TM) SE Runtime Environment (build 1.7.0_15-b03)

測試流程:
crontab中設置每隔15分不斷重啓tomcat
啓動腳本如下:
$ cat notasktest.sh
#!/bin/sh
source /etc/profile
PROJECT="notask"
COUNT=`ps aux|grep java|grep $PROJECT|awk '{print $2}'|wc -l`
if [ $COUNT -le 0 ]
then
echo process not exist
else
cd $project_dir
bin/shutdown.sh
sleep 5
ps aux|grep "java"|grep $PROJECT|awk '{print $2}'|xargs kill -9
sleep 5
bin/startup.sh
fi

啓動成功後的進程信息:
/usr/bin/java -Djava.util.logging.config.file=$project_dir/gameserver.notask/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xms4096M -Xmx4096M -XX:PermSize=128M -XX:MaxPermSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:HeapDumpPath=$project_dir/gameserver.notask/log/heap.hprof -Dproject=notask -Djava.rmi.server.hostname=ip -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1108 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:$project_dir/gameserver.notask/log/gc.log -XX:ErrorFile=$project_dir/gameserver.notask/log/java_error_%p.log -Djava.endorsed.dirs=$project_dir/gameserver.notask/endorsed -classpath $project_dir/gameserver.notask/bin/bootstrap.jar:$project_dir/gameserver.notask/bin/tomcat-juli.jar -Dcatalina.base=$project_dir/gameserver.notask -Dcatalina.home=$project_dir/gameserver.notask -Djava.io.tmpdir=$project_dir/gameserver.notask/temp org.apache.catalina.startup.Bootstrap start

現象:
tomcat進程在啓動成功後,可能突然掛掉。效果類似被kill -9
難點:
1、是否掛掉不確定(所以比較難復現,導致很難用回滾法定位問題所在)
2、掛掉時間不確定,有可能剛啓動就掛掉,也可能運行一陣子才掛掉。
3、掛掉後,系統中沒有任何日誌(tomcat自帶的日誌logs下什麼都沒有,當前項目目錄也什麼都沒有)
分析:
被kill(系統、其他進程、人爲、自身)、 異常(系統、自身)
努力:
1 捕捉未處理的異常
Thread.setDefaultUncaughtExceptionHandler(new UncaughtExceptionHandler() {
UncaughtExceptionHandler old = Thread
.getDefaultUncaughtExceptionHandler();

@Override
public void uncaughtException(Thread t, Throwable e) {
java.io.StringWriter s = new StringWriter(80);
e.printStackTrace(new PrintWriter(s));
logger.error(
"線程異常結束:" + t.getName() + "("
+ Long.toHexString(t.getId()) + ")\n"
+ s.toString(), e);
if (old != null) {
old.uncaughtException(t, e);
}
}
});

2 捕捉代碼中的exit
System.setSecurityManager(new MySecurityManager());

//------------------MySecurityManager.java關鍵代碼如下
public class MySecurityManager extends SecurityManager {

@Override
public void checkExit(int status) {
logger.error("嚴重bug,系統退出!"+ new Exception());
throw new SecurityException("not allow to call System.exit");
}

3 在啓動指令中(詳情見上)指定了XX:+HeapDumpOnOutOfMemoryError XX:HeapDumpPath=$project_dir/gameserver.notask/log/heap.hprof
Xloggc:$project_dir/gameserver.notask/log/gc.log -XX:ErrorFile=$project_dir/gameserver.notask/log/java_error_%p.log
捕捉堆棧信息、crash信息

4 開啓了系統日誌
service rsyslog start

5 開啓coredump日誌
cat /etc/security/limits.conf
* soft nofile 32768
* hard nofile 65536
* soft core unlimited
* hard core unlimited
# End of file

$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62825
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

3.6 通過history查看當時沒有命令行執行的指令記錄,沒看到kill指令
export HISTTIMEFORMAT='%F %T - ' #開啓每條指令的時間/etc/profile
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章