问题起因：
java工程消费的内容延迟严重，发现负责消费的dispatcher 进程阻塞，且占用的CPU%很高。
开始按照https://www.cnblogs.com/paul8339/p/7464206.html 文中提到的方法排查。
一、追查
第一步：先获取到需要排查的进程PID
查询前十个消耗CPU做多的进程命令：
ps aux|head -1;ps aux|grep -v PID|sort -rn -k +3|head
获取java进程的资源消费情况：
ps -ef | grep java
第二步：获取线程信息，找到耗CPU的线程
ps -mp pid -o THREAD,tid,time | sort -rn 6
第三步：打印线程的堆栈信息
先将需要的线程ID转换为16进制格式
printf "%x\n" tid
jstack pid |grep tid -A 30
踩坑1:
我们的工程运行在docker环境下，需要进入docker下执行前两步，否则在执行jstack的时候报错：
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:411)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
解决办法：
执行 docker ps -a 找到需要进入的docker CONTAINER ID
执行 docker exec -it CONTAINER ID /bin/bash 进入docker
踩坑2:
linux没有默认开启trace debug功能，执行jstack的时候提示：
Can't attach to the process
解决办法：
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
参考https://www.cnblogs.com/wscit/p/6803354.html
执行jstack后发现好多线程都是BLOCKED状态，堆栈信息如下：
Thread 171: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
--
- org.apache.commons.httpclient.HttpMethodBase.readStatusLine(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=36, line=1973 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodBase.readResponse(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=21, line=1735 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodBase.execute(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=68, line=1098 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(org.apache.commons.httpclient.HttpMethod) @bci=135, line=398 (Compiled frame)
- org.apache.commons.httpclient.HttpMethodDirector.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=288, line=171 (Compiled frame)
- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HostConfiguration, org.apache.commons.httpclient.HttpMethod, org.apache.commons.httpclient.HttpState) @bci=114, line=397 (Compiled frame)
- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=14, line=323 (Compiled frame)
- com.weibo.qa.util.RequestUtil.execHttpMethod(org.apache.commons.httpclient.HttpMethod, int) @bci=38, line=361 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String, int) @bci=18, line=319 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String) @bci=4, line=340 (Compiled frame)
- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.lang.String) @bci=11, line=272 (Compiled frame)
- com.weibo.qa.terminator.service.FriendService.getUserInforBatch(java.lang.Long[]) @bci=255, line=126 (Compiled frame)
二、分析
分析上面的信息可以看到，是调用了HttpClient之后线程锁死。
看工程的log日志，发现较多的warning：
WARN org.apache.commons.httpclient.HttpMethodBase - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
参考https://blog.csdn.net/xiaoshuji/article/details/71077619，提到HttpClient建议使用InputStream getResponseBodyAsStream()代替byte[]getResponseBody()。对于返回结果很大或无法预知的情况，就需要使用InputStreamgetResponseBodyAsStream()，避免byte[] getResponseBody()可能带来的内存的耗尽问题。
三、解决
1. 一方面优化工程代码，减少接口调用；
2. 修改HttpClient接口的Response获取方式。