最近服務器 一直不太正常,經常cpu佔到200%,load到2,隨時都有可能報警
通過 top -H -p xx看到
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27987 admin 25 0 1559m 542m 13m R 100.2 27.0 15995:57 java
27988 admin 25 0 1559m 542m 13m R 100.2 27.0 16027:01 java
看來cpu完全消耗到了2個線程上,通過jstack threaddump,發現
"pool-4-thread-1" prio=10 tid=0x0000000059fa8000 nid=0x6d53 runnable [0x0000000043ce0000..0x0000000043ce0a90]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at java.util.HashSet.add(HashSet.java:200)
at com.xx.smsgw.domain.MessagePool.put(MessagePool.java:29)
at com.xx.smsgw.domain.SmsMTQueue.putSmsMT(SmsMTQueue.java:43)
at com.xx.smsgw.smc.mt.MTChannelImpl.put(MTChannelImpl.java:45)
at com.xx.smsgw.smc.route.MTRouteTask.run(MTRouteTask.java:49)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-4-thread-2" prio=10 tid=0x000000005a621000 nid=0x6d54 runnable [0x00000000455f9000..0x00000000455f9a10]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:484)
at java.util.HashMap.resize(HashMap.java:463)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at java.util.HashSet.add(HashSet.java:200)
at com.xx.smsgw.domain.MessagePool.put(MessagePool.java:29)
at com.xx.smsgw.domain.SmsMTQueue.putSmsMT(SmsMTQueue.java:43)
at com.xx.smsgw.smc.mt.MTChannelImpl.put(MTChannelImpl.java:45)
at com.xx.smsgw.smc.route.MTRouteTask.run(MTRouteTask.java:49)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
這就奇怪了,耗在HashMap上了,更奇怪的是我多次threaddump,這個信息完全保持不變,就好像thread僵在那一樣
從下面的這篇日誌找到答案:
https://jira.jboss.org/jira/browse/JGRP-525
說是併發的HashSet訪問導致,解決方案是
- 同步HashSet/HashMap
- HashSet/HashMap-->ConcurrentHashMap
我個人認爲如果每個線程一個單獨的HashSet/HashMap實例也可以解決,關鍵是要避免在線程不安全的環境中使用HashSet/HashMap