httpClient4.5.9 連接池及重試配置處理

近期改進項目中的httpClient請求，由原來的複用單連接，改爲使用連接池，解決併發調用的問題，最近又報出以下異常。

e:org.apache.http.NoHttpResponseException: 【ip:port】 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)

經排查，是服務器每秒5次調用服務接口，猜測可能是服務器端負載過大，導致在收到請求後無法處理的結果，查閱資料

其他博友的說法是：

博友1：https://www.cnblogs.com/756623607-zhang/p/10960782.html

在某些情況下，通常是在高負載下，web服務器可能能夠接收請求，但無法處理它們。缺乏足夠的資源(比如工作線程)就是一個很好的例子。這可能導致服務器在不給出任何響應的情況下斷開到客戶機的連接。HttpClient在遇到這種情況時拋出NoHttpResponseException。在大多數情況下，重試使用NoHttpResponseException失敗的方法是安全的。

官網給出的解決辦法就是：重試調用失敗的方法

博友2：https://blog.csdn.net/liubenlong007/article/details/78180333

當服務器端由於負載過大等情況發生時，可能會導致在收到請求後無法處理(比如沒有足夠的線程資源)，會直接丟棄鏈接而不進行處理。此時客戶端就回報錯：NoHttpResponseException。
官方建議出現這種情況時，可以選擇重試。但是重試一定要限制重試次數，避免雪崩。

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
import org.apache.http.Consts;
import org.apache.http.HeaderElement;
import org.apache.http.HeaderElementIterator;
import org.apache.http.HttpEntity;
import org.apache.http.HttpEntityEnclosingRequest;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.NoHttpResponseException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.config.ConnectionConfig;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.conn.ConnectionKeepAliveStrategy;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.message.BasicHeaderElementIterator;
import org.apache.http.protocol.HTTP;
import org.apache.http.protocol.HttpContext;
import org.apache.http.ssl.SSLContexts;
import org.apache.http.ssl.TrustStrategy;
import org.apache.http.util.EntityUtils;
import org.springframework.core.io.ClassPathResource;
import org.springframework.stereotype.Service;

import javax.annotation.PostConstruct;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InterruptedIOException;
import java.net.SocketTimeoutException;
import java.net.UnknownHostException;
import java.security.KeyStore;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.TimeUnit;


@Service
@Slf4j
public class HttpService {

    private static int MAX_FAIL_RETRY_COUNT = 3;
    private HttpClient httpClient;


    //優化併發場景 HttpClient 單線程問題
    @PostConstruct
    public void init2() throws Exception {

        try {
            // 1. 根據證書來調用
            SSLContext sslContext = sslContent();
            SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE);

            //存活的策略
            ConnectionKeepAliveStrategy myStrategy = ka();

            // 設置協議http和https對應的處理socket鏈接工廠的對象
            Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
                    .register("http", PlainConnectionSocketFactory.INSTANCE)
                    .register("https", sslConnectionSocketFactory)
                    .build();

            //創建ConnectionManager，添加Connection配置信息
            //最大連接數
            //例如默認每路由最高50併發，具體依據業務來定,一般和setMaxTotal 一致
            PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
            connectionManager.setMaxTotal(200);
            connectionManager.setDefaultMaxPerRoute(200);
            //檢測有效連接的間隔 2s
            connectionManager.setValidateAfterInactivity(2000);

            RequestConfig requestConfig = RequestConfig.custom()
                    //.setConnectionRequestTimeout(6000)//設定連接服務器超時時間
                    //.setConnectTimeout(2000)//設定從連接池獲取可用連接的時間
                    //.setSocketTimeout(6000)//設定獲取數據的超時時間
                    .build();

            ConnectionConfig connectionConfig = ConnectionConfig.custom()
                    .setBufferSize(4128)
                    .build();

            httpClient = HttpClients.custom()
                    .setConnectionManager(connectionManager)
                    .setKeepAliveStrategy(myStrategy)
                    .setDefaultRequestConfig(requestConfig)
                    .setSSLHostnameVerifier(new NoopHostnameVerifier())
                    //不使用這種方式，不方便看日誌，使用下面自定義的retry
                    //.setRetryHandler(new DefaultHttpRequestRetryHandler(3,true))
                    .setRetryHandler(new MyRetryHandler())
                    .setDefaultConnectionConfig(connectionConfig)
                    .build();

            log.info("註冊https證書成功");
        } catch (Exception e) {
            log.error("註冊https證書失敗,e:{}", e);
        }
    }


  
    public String postJSON(String url, String json) throws IOException {
        InputStream inStream = null;
        HttpEntity entity = null;
        try {
            HttpPost httpPost = new HttpPost(url);
            httpPost.setEntity(new StringEntity(json, Consts.UTF_8));
            HttpResponse httpResponse = httpClient.execute(httpPost);
            entity = httpResponse.getEntity();

            if (httpResponse.getStatusLine().getStatusCode() == 200) {
                inStream = entity.getContent();
                return IOUtils.toString(inStream);
            } else {
                throw new IOException("Unexpected code " + httpResponse);
            }
        } finally {
            EntityUtils.consume(entity);
            if (inStream != null) {
                inStream.close();
            }
        }
    }


    /**
     * 請求重試處理器
     */
    private static class MyRetryHandler implements HttpRequestRetryHandler {

        @Override
        public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {

            HttpClientContext clientContext = HttpClientContext.adapt(context);
            HttpRequest request = clientContext.getRequest();
            String uri = request.getRequestLine().getUri();

            if (executionCount >= MAX_FAIL_RETRY_COUNT) {
                log.warn("{}-{}重試次數大於等於3次", uri, executionCount);
                return false;
            }

            boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
            if (idempotent) {
                // 如果請求被認爲是冪等的，則重試
                log.warn("冪等接口重試：{},次數：{}", uri, executionCount);
                return true;
            }

            //NoHttpResponseException 重試
            if (exception instanceof NoHttpResponseException) {
                log.warn("NoHttpResponseException 異常重試，接口：{},次數：{} ", uri, executionCount);
                return true;
            }

            //連接超時重試
            if (exception instanceof ConnectTimeoutException) {
                log.warn("ConnectTimeoutException異常重試 ，接口：{},次數：{} ", uri, executionCount);
                return true;
            }

            // 響應超時不重試，避免造成業務數據不一致
            if (exception instanceof SocketTimeoutException) {
                return false;
            }

            if (exception instanceof InterruptedIOException) {
                // 超時
                return false;
            }
            if (exception instanceof UnknownHostException) {
                // 未知主機
                return false;
            }

            if (exception instanceof SSLException) {
                // SSL handshake exception
                return false;
            }

            return false;
        }
    }

    private ConnectionKeepAliveStrategy ka() {

        //就是通過Keep-Alive頭信息中，獲得timeout的值，作爲超時時間；單位毫秒；
        //如請求頭中 Keep-Alive: timeout=5, max=100
        //DefaultConnectionKeepAliveStrategy strategy = DefaultConnectionKeepAliveStrategy.INSTANCE;
        ConnectionKeepAliveStrategy myStrategy = new ConnectionKeepAliveStrategy() {
            @Override
            public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
                HeaderElementIterator it = new BasicHeaderElementIterator
                        (response.headerIterator(HTTP.CONN_KEEP_ALIVE));
                while (it.hasNext()) {
                    HeaderElement he = it.nextElement();
                    String param = he.getName();
                    String value = he.getValue();
                    if (value != null && param.equalsIgnoreCase("timeout")) {
                        return Long.parseLong(value) * 1000;
                    }
                }
                //如果沒有約定，則默認定義時長爲60s
                return 60 * 1000;
            }
        };
        return myStrategy;
    }

    private SSLContext sslContent() throws Exception {

        SSLContext sslcontext = null;
        InputStream instream = null;
        KeyStore trustStore = null;
        try {
            trustStore = KeyStore.getInstance("jks");
            instream = new ClassPathResource("config/release/test.jks").getInputStream();
            trustStore.load(instream, "123456".toCharArray());

            // 相信自己的CA和所有自簽名的證書
            TrustStrategy ts = new TrustStrategy() {
                @Override
                public boolean isTrusted(X509Certificate[] chain, String authType) {
                    return true;
                }
            };
            sslcontext = SSLContexts.custom()
                    //.loadTrustMaterial(trustStore, new TrustSelfSignedStrategy())
                    .loadTrustMaterial(trustStore, ts)
                    .build();
        } finally {
            try {
                instream.close();
            } catch (IOException e) {
            }
        }

        return sslcontext;
    }
}

httpClient4.5.9 連接池及重試配置處理

crc32 散列均衡

httpClient4.5.9 連接池及重試配置處理

mysql DDL 建表語句結構導出到EXCEL

springboot + 攜程apllo 配置實時生效(2)-全局生效

攜程apllo 配置實時生效

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結