公司一項目在高峯期報異常,且請求響應時間比預期明顯長很多,排查問題發現和httpclient有關,在這裏描述一下3.1版本中容易踩到的坑
問題:生產環境異常信息:
2016-12-19 14:43:27,697 ERROR - Http post occur error!url=XXXXXX
org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:497)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
……
爲了排除由於網絡延時,httpclient請求訪問的系統響應時間太長等其他因素的影響,寫了一個mock應用實現:(1)接收httpclient發送過來的http請求,(2)等待200ms,(3)返回信息“請求成功”
排查過程:
1.單線程調用httpclient方法,循環執行100次,無異常
2.多線程併發調用httpclient,10個線程,循環執行100次,報異常ConnectionPoolTimeoutException: Timeout waiting for connection,問題復現
3.檢查連接時間設置5000ms,遠大於200ms(mock程序等待時間),無問題
4.檢查連接數設置DEFAULT_MAX_TOTAL_CONNECTIONS = 20;大於10個線程,無問題
5.觀察mock應用線程狀態,發現只有2個http連接,找到另外一個可疑參數DEFAULT_MAX_HOST_CONNECTIONS = 2;每個主機最大連接數,重新賦值後,解決 Timeout waiting for connection問題。
總結:org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection這個問題通常是由這三個參數配置不合理導致的
連接時間、最大連接數、每個主機最大連接數,通常每個主機最大連接數DEFAULT_MAX_HOST_CONNECTIONS容易被我們忽略。
httpclient4.5版本中也有類似相關參數的配置,需要合理設置:
cm = new PoolingHttpClientConnectionManager(registry);
// 將最大連接數增加到200
cm.setMaxTotal(200);
// 將每個路由基礎的連接增加到20
cm.setDefaultMaxPerRoute(20);
httpclient4.5連接數設置不合理拋出異常信息:
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
……