OkHttp工具類在微服高併發場景中問題實踐總結

問題場景

我的應用是一箇中間業務應用XXApp，一個交易請求進來需要依賴下游應用，採用http協議通訊方式，需要調用3-4次下游請求。
老XXApp在生產上有大量Close_wait問題，（可以請求交易沒有及時關閉，會造成系統有限資源port被佔用完）導致“too many open files” SocketExcepiton.
我改成以前常用的OKhttp3，有個高併發場景的應用Utils類。但是讓我沒想到的在壓力測試環境下20qps，XXApp系統不能連接後端服務，異常表現是大量的“time out”。
開始判斷是後端服務能力不夠或者是業務服務線程數不夠，

OKHttp的部分代碼

  private static final OkHttpClient OK_HTTP_CLIENT =new OkHttpClient().newBuilder().connectTimeout(5,TimeUnit.SECONDS).build();


 /**
     * POST請求，參數爲json格式。
     *
     * @param url
     * @param json
     * @return Optional<String>
     */
    public static String post(String url, String json, int timeout) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(JSON, json);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(timeout).newCall(request).execute().body().string();
        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }


    }

    public static OkHttpClient getHttpClient(int timeout) {

            if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
                return OK_HTTP_CLIENT;
            } else {
                OkHttpClient okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();

                return okHttpClient;
            }
        }

爲了支持不同業務調用需要不同timeout，提供自定義okHttpClient與默認的OK_HTTP_CLIENT。
XXApp在業務調用的時候採用不同接口自定義timeout時間。

壓測現象分析

top -H -P 進程ID，查看進程的線程數，發現totalThread數上15000多，還在不停的增加，這就有些奇怪了，以前高併發應用也沒有這麼高的線程數。
netstat -n | awk ‘/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’ ；發現也有CLOSE_WAIT的連接狀態在增長。

[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-CGSrfyGC-1582808024865)(./image/close_wait_issue.jpg)]

[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-I65oGUiM-1582808024867)(./image/many_conn.jpg)]

3. 開始懷疑是不是每個請求都創建okHttpClient實例造成線程數量增加。然後在關閉連接過程造成CLOSE_WAIT數量波動在3000-4000多。
4. 查詢一下http協議1.1 keep-alive長連接應用可以減少tcp 連接，關閉的時間。OKHttp也有一個ConnectionPool類來支持併發請求，查看builder類，果然給okHttpClient創建一個默認
5. 如果每次都要創建okHttpClient的確不很好提升性能，所有先改成默認OK_HTTP_CLIENT實例試下，壓測效果瞬間提升到100qps。進程的線程數也穩定在500左右，CLOSE_WAIT問題也沒有了.

再次壓測現象分析

[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-Mp6JHBlo-1582808024867)(./image/ok-conn-count.jpg)]

各種壓測指標，監控指標都正常了。

繼續分析源碼來總結，查看OkHttpClients源碼

  * Factory for {@linkplain Call calls}, which can be used to send HTTP requests and read their
  * responses.
  *
  * <h3>OkHttpClients should be shared</h3>
  *
  * <p>OkHttp performs best when you create a single {@code OkHttpClient} instance and reuse it for
  * all of your HTTP calls. This is because each client holds its own connection pool and thread
  * pools. Reusing connections and threads reduces latency and saves memory. Conversely, creating a
  * client for each request wastes resources on idle pools.
  *
  * <p>Use {@code new OkHttpClient()} to create a shared instance with the default settings:
  * <pre>   {@code
  *
  *   // The singleton HTTP client.
  *   public final OkHttpClient client = new OkHttpClient();
  * }</pre>
  *

已經明確描述希望OkHttpClient是一個單例，因爲它有自己連接池與線程池。也解釋爲什麼不停調用會thread增加。

總結

我在構建一個utils的時候，除參考網絡上的代碼例子，還應該參考官方example，及api的comments。
https://github.com/square/okhttp/blob/master/okhttp/src/test/java/okhttp3/OkHttpClientTest.java
關注點

注意測試工具的timeout問題，一定要寫單元測試。
注意工具的連接關閉問題，參考官方demo是最好的。netstat 觀察
瞭解工具調用連接結構，方便調優，已解決在高併發場景下的調優。

修改後的OKHttp3Utils3


import lombok.extern.slf4j.Slf4j;
import okhttp3.*;
import okhttp3.internal.Util;
import okio.BufferedSink;
import okio.Okio;
import okio.Source;

import java.io.IOException;
import java.io.InputStream;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;

@Slf4j
public final class OKHttp3Utils3 {

    public static int DEFAULT_TIME_OUT = 10;

    /**
     * 全局實例可以保持http1.1 連接複用，線程池複用， 減少tcp的網絡連接，關閉，
     * 如果每次一個請求，在高併發下，thread增多到1W，close_wait持續增加到6k。
     */
    private static final OkHttpClient OK_HTTP_CLIENT = new OkHttpClient.Builder()
            .connectionPool(new ConnectionPool(50, 5, TimeUnit.MINUTES))
            .connectTimeout(10, TimeUnit.SECONDS).readTimeout(10, TimeUnit.SECONDS).writeTimeout(10, TimeUnit.SECONDS).build();

    private static final MediaType JSON = MediaType.parse("application/json; charset=utf-8");

    private static final MediaType FORM_TYPE = MediaType.parse("application/x-www-form-urlencoded; charset=UTF-8");


    /**
     * 不同timeout的連接池
     */
    public static ConcurrentHashMap<Integer, OkHttpClient> cacheClients = new ConcurrentHashMap();


    public static OkHttpClient getHttpClient(int timeout) {

        if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
            return OK_HTTP_CLIENT;
        } else {
            OkHttpClient okHttpClient = cacheClients.get(timeout);
            if (okHttpClient == null) {
                return syncCreateClient(timeout);
            }
            return okHttpClient;
        }
    }

    private static synchronized OkHttpClient syncCreateClient(int timeout) {
        OkHttpClient okHttpClient;

        okHttpClient = cacheClients.get(timeout);
        if (okHttpClient != null) {
            return okHttpClient;
        }

        okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();
        cacheClients.put(timeout, okHttpClient);
        return okHttpClient;

    }


    /**
     * GET請求
     *
     * @param url
     * @return Optional<String>
     */
    public static String get(String url, int timeout) throws Exception {
        Request request = new Request.Builder().url(url)
                .build();
        return getHttpClient(timeout).newCall(request).execute().body().string();

    }

    public static String get(String url) throws Exception {
        return get(url, 0);
    }

    /**
     * POST請求，參數爲json格式。
     *
     * @param url
     * @param json
     * @return Optional<String>
     */
    public static String post(String url, String json, int timeout) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(JSON, json);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(timeout).newCall(request).execute().body().string();
        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }


    }


    public static String post(String url, String json) throws Exception {
        return post(url, json, 0);
    }


    public static String postByFormType(String url, String form) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(FORM_TYPE, form);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(0).newCall(request).execute().body().string();

        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }
    }


    /**
     * 根據不同的類型和requestbody類型來接續參數
     *
     * @param url
     * @param mediaType
     * @param inputStream
     * @return
     * @throws Exception
     */
    public static String post(String url, MediaType mediaType, InputStream inputStream) throws Exception {
        RequestBody body = createRequestBody(mediaType, inputStream);
        Request request = new Request.Builder().url(url).post(body).build();
        return OK_HTTP_CLIENT.newCall(request).execute().body().string();
    }

    private static RequestBody createRequestBody(final MediaType mediaType, final InputStream inputStream) {
        return new RequestBody() {
            // @Nullable
            @Override
            public MediaType contentType() {
                return mediaType;
            }

            @Override
            public long contentLength() throws IOException {
                try {
                    return inputStream.available();
                } catch (IOException e) {
                    return 0;
                }
            }

            @Override
            public void writeTo(BufferedSink sink) throws IOException {
                Source source = null;
                try {
                    source = Okio.source(inputStream);
                    sink.writeAll(source);
                } finally {
                    Util.closeQuietly(source);
                }
            }
        };
    }

}

OkHttp工具類在微服高併發場景中問題實踐總結