問題起因：
java工程消費的內容延遲嚴重，發現負責消費的dispatcher 進程阻塞，且佔用的CPU%很高。
開始按照https://www.cnblogs.com/paul8339/p/7464206.html 文中提到的方法排查。
一、追查
第一步：先獲取到需要排查的進程PID
查詢前十個消耗CPU做多的進程命令：
ps aux|head -1;ps aux|grep -v PID|sort -rn -k +3|head
獲取java進程的資源消費情況：
ps -ef | grep java
第二步：獲取線程信息，找到耗CPU的線程
ps -mp pid -o THREAD,tid,time | sort -rn 6
第三步：打印線程的堆棧信息
先將需要的線程ID轉換爲16進制格式
printf "%x\n" tid
jstack pid |grep tid -A 30
踩坑1:
我們的工程運行在docker環境下，需要進入docker下執行前兩步，否則在執行jstack的時候報錯：
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:411)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
解決辦法：
執行 docker ps -a 找到需要進入的docker CONTAINER ID
執行 docker exec -it CONTAINER ID /bin/bash 進入docker
踩坑2:
linux沒有默認開啓trace debug功能，執行jstack的時候提示：
Can't attach to the process
解決辦法：
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
參考https://www.cnblogs.com/wscit/p/6803354.html
執行jstack後發現好多線程都是BLOCKED狀態，堆棧信息如下：
Thread 171: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
--
- org.apache.commons.httpclient.HttpMethodBase.readStatusLine(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=36, line=1973 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodBase.readResponse(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=21, line=1735 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodBase.execute(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=68, line=1098 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(org.apache.commons.httpclient.HttpMethod) @bci=135, line=398 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodDirector.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=288, line=171 (Compiled frame)
- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HostConfiguration, org.apache.commons.httpclient.HttpMethod, org.apache.commons.httpclient.HttpState) @bci=114, line=397 (Compiled frame)
- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=14, line=323 (Compiled frame)
- com.weibo.qa.util.RequestUtil.execHttpMethod(org.apache.commons.httpclient.HttpMethod, int) @bci=38, line=361 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String, int) @bci=18, line=319 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String) @bci=4, line=340 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.lang.String) @bci=11, line=272 (Compiled frame)
- com.weibo.qa.terminator.service.FriendService.getUserInforBatch(java.lang.Long[]) @bci=255, line=126 (Compiled frame)
二、分析
分析上面的信息可以看到，是調用了HttpClient之後線程鎖死。
看工程的log日誌，發現較多的warning：
WARN org.apache.commons.httpclient.HttpMethodBase - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
參考https://blog.csdn.net/xiaoshuji/article/details/71077619，提到HttpClient建議使用InputStream getResponseBodyAsStream()代替byte[]getResponseBody()。對於返回結果很大或無法預知的情況，就需要使用InputStreamgetResponseBodyAsStream()，避免byte[] getResponseBody()可能帶來的內存的耗盡問題。
三、解決
1. 一方面優化工程代碼，減少接口調用；
2. 修改HttpClient接口的Response獲取方式。