k8s集羣ingress controller壓測記錄

前言

這兩天老大讓做k8s測試集羣內ingress controller的壓測報告，記錄下。ingress-controller中nginx結構爲1master4worker，結構如下：

短連接測試

最開始使用jmeter，配置如下：

四個線程組（其實和單線程組的效果一致），從1000開始增加，得到的CPU使用率結果如下：

當請求次數超過9500次之後，CPU負載不再增加，在結果樹中可以看到大量失敗返回：

可以知道是由大量短連接導致的timewait過多，無法建立新連接，在http頭中開啓長連接，發現依然有這個問題，查閱資料後發現,jmeter會主動去關閉連接，於是使用HttpClient進行測試，最開始使用單連接進行調用，結果發現請求數量到4000以上，CPU使用率不再提高：

於是使用4000個連接開啓測試：

測試源碼如下：

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CountDownLatch;

public class testCPU {
    static int count = 0;
    static int num = 0;
    static int requestPerConnection = 16;
    static int totalRequest = requestPerConnection * 4000;
    public static void send(CloseableHttpClient httpclient, HttpUriRequest request, CountDownLatch countDownLatch) throws IOException{

        // Create a custom response handler
        ResponseHandler<String> responseHandler = (HttpResponse response) -> {
                HttpEntity entity = response.getEntity();
                if (response.getStatusLine().getStatusCode() != 200) {
                    count++;
                }else {
                    num++;
                }
                return entity != null ? EntityUtils.toString(entity) : null;
        };
        String responseBody = httpclient.execute(request, responseHandler);
        countDownLatch.countDown();
    }

    public static void main(String[] args) throws InterruptedException, IOException {
        CountDownLatch countDownLatch = new CountDownLatch(totalRequest);
        for (int i = 0; i < 4000; i++) {
            // create custom http headers for httpclient
            List<Header> defaultHeaders = Arrays.asList(
                    new BasicHeader("X-Default-Header", "default header httpclient"));
            //創建一個HttpClient對象
            CloseableHttpClient httpclient = HttpClients
                    .custom()
                    .setDefaultHeaders(defaultHeaders)
                    .build();
            HttpUriRequest request = RequestBuilder.get()
                    .setUri("http://xiaoxingchen-demo.com:30102/index")
                    .setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
                    .setHeader(HttpHeaders.CONNECTION, "Keep-Alive")
                    .setHeader(HttpHeaders.CACHE_CONTROL, "no-cache")
                    .build();

            Runnable runnable = () -> {
                try {
                    send(httpclient, request, countDownLatch);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            };
            for (int j = 0; j < requestPerConnection; j ++) {
                Thread thread = new Thread(runnable);
                thread.start();
            }
        }
        countDownLatch.await();
        System.out.println("成功請求個數：" + num + "個");
        System.out.println("成功請求個數：" + + count + "個");

    }
}

負載又沒有衝上去，因爲用了countdownlatch計算相應的成功率，其實邏輯上還是單線程的調用，於是去掉countdownlatch：

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

public class testCPU {
    static int requestPerConnection = 20;
    public static void send(CloseableHttpClient httpclient, HttpUriRequest request) throws IOException{

        // Create a custom response handler
        ResponseHandler<String> responseHandler = (HttpResponse response) -> {
                HttpEntity entity = response.getEntity();
                if (response.getStatusLine().getStatusCode() != 200) {
                    count++;
                }else {
                    num++;
                }
                return entity != null ? EntityUtils.toString(entity) : null;
        };
        String responseBody = httpclient.execute(request, responseHandler);
    }

    public static void main(String[] args) throws InterruptedException, IOException {
        for (int i = 0; i < 8000; i++) {
            // create custom http headers for httpclient
            List<Header> defaultHeaders = Arrays.asList(
                    new BasicHeader("X-Default-Header", "default header httpclient"));
            //創建一個HttpClient對象
            CloseableHttpClient httpclient = HttpClients
                    .custom()
                    .setDefaultHeaders(defaultHeaders)
                    .build();
            HttpUriRequest request = RequestBuilder.get()
                    .setUri("http://xiaoxingchen-demo.com:30102/index")
                    .setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
                    .setHeader(HttpHeaders.CONNECTION, "Keep-Alive")
                    .setHeader(HttpHeaders.CACHE_CONTROL, "no-cache")
                    .build();

            Runnable runnable = () -> {
                try {
                    send(httpclient, request);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            };
            for (int j = 0; j < requestPerConnection; j ++) {
                Thread thread = new Thread(runnable);
                thread.start();
            }
        }
    }
}

測試結果如下：

負載衝到了百分之二十後又衝不上去了，並出現大量連接超時，查看pod負載和連接數如下：

可以看到測試的tomcat服務已經打滿了，但是nginx服務負載還沒衝上去，被主管一語道破，用ngnix去壓tomcat是不合理的！臨近雙11，機器緊張，節後再申請10臺機器部署服務繼續測試~~~~

節後再更，先做DB負載均衡服務的壓測