性能測試誤差對比研究(一)

之前思考過兩種統計模式的各種誤差來源,以及如何避免這些可能產生的誤差。又做了一些具體的框架改進,如下列文章所示:

今天分享一下在性能測試統計中,各種參數和性能指數對性能測試誤差的影響,以及各種減少誤差方法效果,以便知道以後的性能測試改如何改進。

演示Demo

我寫了一個模擬正常單接口性能測試的時候的Demo,我使用了ThreadLimitTimesCount<String>模型,使用sleep()方法模擬請求和響應,具體代碼如下:


/**
 * 性能測試統計誤差測試類
 * by:FunTester
 */

class DeviationTest extends SourceCode {


    static void main(String[] args) {
        def ts = []

        10.times {
            ts << new FunTester(StringUtil.getString(10), 60)
        }

        new Concurrent(ts, "FunTester誤差分析").start()

    }

    private static class FunTester extends ThreadLimitTimesCount<String{


        FunTester(String s, int times) {
            super(s, times, null)
        }

        @Override
        protected void doing() throws Exception {
            sleep(0.1)
        }

        @Override
        ThreadBase clone() {
            new FunTester(StringUtil.getString(10), times)
        }
    }

}

下面開始針對之前提到的誤差來源進行對比分析了。

由於模擬的時間比較短,這裏就不進行標準的演示了。在我自己測試空轉的過程中也很難在ms級別統計代碼運行,所以我也放棄了對代碼運行時間的對比。

線程數

首先來研究一下,線程數對性能測試誤差的影響。首先我們先來一組配置20線程、50請求次數在固定請求時間的情況下的模擬,看一下結果。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"3.29%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":1000,
>  ① . "qps2":187.79342723004694,
>  ① . "total":1000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 15:40:58",
>  ① . "endTime":"2021-04-05 15:41:04",
>  ① . "mark":"FunTester誤差分析051540",
>  ① . "table":"eJztjzEKwkAQRfvA3mEOkEC0sMgBPIDkAgEHXHBXySYQSxElpVVKj2BK7xOIeAxX7SzcFUQZmeHDm2r+PBGAx4xLnaIpML+0bX869vXufNjDMIZilmM29TkBIhCvuyZolgttEFKpMIEqMpjLbA66VCGsIoVTmWlXh/sPJTU8biWDOAZlQpVVdh3Z9QMWjumajY0DWxs/1Db2p65Z28AzPKreLr43fhs3w58UsyEbsiEb/hHYkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD5EcAVmaHb1"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

實測平均耗時103ms,總的誤差是3.29%,不算很高。下面看一下配置40線程、50請求次數在固定請求時間的情況下的模擬,看一下結果。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":40,
>  ① . "deviation":"2.7%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":377.8575477045154,
>  ① . "total":2000,
>  ① . "qps":388.3495145631068,
>  ① . "startTime":"2021-04-05 15:43:19",
>  ① . "endTime":"2021-04-05 15:43:24",
>  ① . "mark":"FunTester誤差分析051543",
>  ① . "table":"eJztz08KgkAUBvC94B3eARQUooUH6ADhBQQHEpopHANbRv9ctnLZEXLZfQSjYzTVrmiUCsbFNzz43Pje97Mt6vBGCxEymbH0WlXN+dQUu8vxQAOPsknKorjLCrItW39rzOR8JiSjMOEsoNyVLE2iKYkFd2jpchYnkWi70d6DJ4KeuwLf84hLh0d54PtD9fkHxa+vLgs1nSS9vFKXKzWfQnf1wy9rNW+xVdMt9mq+iUdTXdhWXW7U0Gvo2rRu7VPchT2oASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEWuENvcRZ6Q=="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

實測平均耗時103ms,總的誤差是2.7%,可以說很相近,再試一組100線程的,其他條件不變。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":102,
>  ① . "failRate":0.0,
>  ① . "threads":100,
>  ① . "deviation":"3.7%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":5000,
>  ① . "qps2":944.1087613293051,
>  ① . "total":5000,
>  ① . "qps":980.3921568627451,
>  ① . "startTime":"2021-04-05 15:44:33",
>  ① . "endTime":"2021-04-05 15:44:38",
>  ① . "mark":"FunTester誤差分析051544",
>  ① . "table":"eJztzz0KwjAUwPE9kDu8A7RQF4cewANILxAwYMBEaVqoo4jS0amjR7Cj9ylUPIYpblXaih/w5IUHv0x5+XMGA84kNZG0iYyvZVmfT3W+vxwPMAoCSOaxFLMhb3DGu3dNpV0tjZUQKS1DyHwrYyUWYFLtwdrXcqaE6dvR/w+tDNzfCpsAbT0tMncdu+sHKrpOVWzdPLBz8xK5mxacVcXGDbR5urGLt77xRZrC326kQiqkQir8P6gQP1SIHyrEDxXihwrxQ4X4oUL8UCF+qBA/VIgfKsQPFeKHsxsIDnrM"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

這次雖然平均響應時間更準了,但誤差反而更高了,我們再試一組200線程的。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":200,
>  ① . "deviation":"3.2%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":10000,
>  ① . "qps2":1879.6992481203008,
>  ① . "total":10000,
>  ① . "qps":1941.7475728155339,
>  ① . "startTime":"2021-04-05 15:45:27",
>  ① . "endTime":"2021-04-05 15:45:33",
>  ① . "mark":"FunTester誤差分析051545",
>  ① . "table":"eJztjz0KwkAQRvvA3mEOkEC0MwfwAJILBFww4K6STUBLxZ+UVik9gpbeR4h4DEctRDBmQVFGZ/ngTbPzzRMOWLx2pkNpUpkct9tytynzxWG9gqbvQ9pLZNS12SEc8byrI81woI2EMFYygJFnZBJHfdCZcmHsKdmNI13XUX+HijVcdwUNFFDGVdEIxxaOb7B49e2LHGNlcvsywdRghqnCHGOHy233EM6+mGKgAvbLl5gqPCj+GM6G3+xnQzZkQzb8BbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhffyD4QlUrRlG"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

目前來看,線程數對於誤差影響不大,幾乎可以忽略。

時間&次數

再來看看測試時間和請求次數的影響,因爲這次我選擇的固定請求次數的模型,所以主要研究參數還是請求次數。

首先來一組,20線程、20請求次數在固定請求耗時的模擬結果。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":105,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"5.28%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":400,
>  ① . "qps2":180.42399639152006,
>  ① . "total":400,
>  ① . "qps":190.47619047619048,
>  ① . "startTime":"2021-04-05 15:48:19",
>  ① . "endTime":"2021-04-05 15:48:21",
>  ① . "mark":"FunTester誤差分析051548",
>  ① . "table":"eJztkksKwjAYhPdC7/AfwEIrrnoADyC9QKEBAyZKU0GX4nPpqkuPoEvvU6h4DOMDQbDaWjFdzM/AZDUzH8RqUIHrjKTPVMyi036fHXbZenncbqjlUNyLWBAWiSCrYb3v6jI1HEjFyOeCeTS2FYt40Cc5Ek2a2IKFPJCfOj7vEFzSLctzHYeEaopg7Lntln7+gKLqpclaqxDJ9xULrf80XSvQhKavmtJkqpVnM608u//w0rbSyrPn5Wky16KHlY57kVonuxDWYAYIQQhC8zNACEIQmp8BQhCC0PwMEIIQhOZngBCEIDQ/A4SVCM8WpxSi"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

平均響應時間105ms,誤差5.28%。現在將請求次數增加到50

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":104,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"2.79%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":1000,
>  ① . "qps2":186.9508319312021,
>  ① . "total":1000,
>  ① . "qps":192.30769230769232,
>  ① . "startTime":"2021-04-05 15:49:42",
>  ① . "endTime":"2021-04-05 15:49:48",
>  ① . "mark":"FunTester誤差分析051549",
>  ① . "table":"eJztz0sKwjAUBdB5oXt4C1BoxVEX4ALEDQQaMGCiNBbqUPx16MihS9Ch+xEUl2HUgeAvhRZi4YYHN4M27x7fowKnk6oe12OeXPb702F3ypfn7ZpaAY37CWdxkSfI9/zfu7pcj4ZKc+oJySPKmpongg1IpbJBk6bksWDKtsPeQwpFj7eiMAhI6oZkWRS2Q3OtQFH2HDe5mUISbMGWum05bqZm3mJmxhJzM5ZYmKks7uU/hu+9frsy84zvf9YlbsI/qAEhhBC6rwEhhBC6rwEhhBC6rwEhhBC6rwEhhBC6rwFhKeEVVbcQ1Q=="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

平均響應時間和誤差都有所下降,現在增加到100請求,看看結果。


~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"1.51%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":191.2411550965768,
>  ① . "total":2000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 15:50:56",
>  ① . "endTime":"2021-04-05 15:51:06",
>  ① . "mark":"FunTester誤差分析051550",
>  ① . "table":"eJztzzEKwjAUBuC9kDu8A7RQHRx6AA8gvUDAgAETpWmhjiJKR6eOHsGO3qdQ8RhG3EqxlQpx+MOD/015/8c8GvDmmY6FSUXyqKrmdm2K0/1ypmlI6SoRfDnkC2Ie+3xrIcx2o42gWCoRUR4YkUi+Jp0pn3aBEkvJdd+N/h5Kanr/FU3CkJTxFc/tOrPrDxTdry4PdnriaOerKOy0gnl1ubdD7Rh5v+OUo3gJ/6AGhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhKOET/9xgkk="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

平均響應時間和誤差進一步降低了,看來增加請求次數是可以降低誤差的,這也是符合直覺的。下面再進行一組200請求次數的測試。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"0.69%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":4000,
>  ① . "qps2":192.8268414963363,
>  ① . "total":4000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 15:52:23",
>  ① . "endTime":"2021-04-05 15:52:44",
>  ① . "mark":"FunTester誤差分析051552",
>  ① . "table":"eJztzzEKwjAUBuC9kDu8A7RQHRx6AA8gvUDAgAETpWmhjiJKR6eOHsGO3qdQ8RhGRBBBW7EQhz88+N+U93/Mow5vnOlYmFQkl6pqTsem2J0PexqGlM4SwaddviDmsc+3JsIsF9oIiqUSEeWBEYnkc9KZ8mkVKDGVXLfdaO+hpKb7X9EgDEkZX/HcriO79qB4++pyY6cltna+isLOI5hXl2s79Br9HH4+5Shuwj+oASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvAeFPwisab36D"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

相信結論已經很明顯了。增加測試請求次數是可以有效降低誤差的。

請求耗時

現在模擬一下請求耗時對誤差的影響,直覺來講,耗時越長,誤差越小,因爲系統誤差一定的情況下,耗時越長,誤差的所佔的比例也就越小了。

先進行一組耗時50ms的測試,線程數20,請求次數設置100


~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":52,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"3.99%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":369.2762186115214,
>  ① . "total":2000,
>  ① . "qps":384.61538461538464,
>  ① . "startTime":"2021-04-05 15:56:36",
>  ① . "endTime":"2021-04-05 15:56:42",
>  ① . "mark":"FunTester誤差分析051556",
>  ① . "table":"eJztz08KgkAYBfC94B2+AyhIUAsP0AGiCwgOJDRTOAq27C8uW7nsCLnsPoLRMRpqV6QTSRP05MFzNd/72RZpfMNUjJlMWHwpy/p0rPPd+bCnnkfJJGZBqPME2ZbdfGvE5HwmJKNxxJlPmStZHAVTEil3aOFyFkaBaLuhMYRHgu6P+X2PuHR4kPn9gfrrQvHeVxVrFb3KVfSEDydWKq9qq9JSt7v69bSwKpYqetX9nA/Ltqpio0LN9d1RnQt/YAaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEP658ApAgBUd"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

下面增大響應耗時到100ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"1.25%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":191.75455417066155,
>  ① . "total":2000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 15:58:00",
>  ① . "endTime":"2021-04-05 15:58:10",
>  ① . "mark":"FunTester誤差分析051558",
>  ① . "table":"eJztzzEKwjAUgOE9kDu8A7RQHRx6AA8gvUCgAQMmStNCHUWUjk4dPYIdvU+h4jGMiOBSW6EoT1548GXKy88Z9DjTzETSpjK5VlVzPjXF/nI8wDiAdJ5IEfd5Ajjj73fNpF0tjZUQKS1DyH0rEyUWYDLtwdrXMlbCdO3o/odWBh5vhaMgAG09LXJ3nbjrABWtpy63bjrYufmIws0Tzupy4wZaGGb/68Zvcy/85X4qpEIqpMJ/gArxQ4X4oUL8UCF+qBA/VIgfKsQPFeKHCvFDhfihQvxQIX44uwGetH51"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

誤差下降了一半還說,再次說明了直覺是對的。現在繼續增加響應耗時到200ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":203,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"0.95%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":97.58477677482313,
>  ① . "total":2000,
>  ① . "qps":98.52216748768473,
>  ① . "startTime":"2021-04-05 15:59:25",
>  ① . "endTime":"2021-04-05 15:59:46",
>  ① . "mark":"FunTester誤差分析051559",
>  ① . "table":"eJztzzEKwjAUgOE9kDu8A7RQOjj0AB5AeoFAHxgwUZoW6iha6ejk6BHs6H2Eiscw4CbW1sHhyQsPvpAhyS8FjFjT0qboCszvbdtdzl2zv50OEEdQzHNU2ZgrQAr5+a0ZutXSOoRUG0ygCh3mWi3AliaAdWgw08oOvTH8D6MtPO9K4igC4wKjKr+d+O2XFdfjxk8fWz997Pz0Ufv5LY2fF6R4e/xHcCF9uJA+XEgfLqQPF9KHC+nDhfThQvpwIX24kD5cSB8upA8X0ocL6cOF9JHiAVtAtsc="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

誤差繼續下降,只不過不是下降趨勢不如剛纔,基本上也能印證直覺得出的結論是對的,響應耗時越長,誤差越小。

響應時間離散係數

實際工作中響應時間都非固定的響應時間,對於某些結果,響應時間可能相差巨大,甚至好幾倍的差距。下面我們來看看響應時間的離散程度對性能測試誤差的影響。

我引入一個隨機的(0,1]的隨機數來模擬響應時間的離散係數。

下面是20線程、100請求次數的模擬結果,隨機休眠配置是sleep(0.1 + getRandomDouble() / 10),理論平均休眠時間是150ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":153,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"3.56%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":126.07160867372667,
>  ① . "total":2000,
>  ① . "qps":130.718954248366,
>  ① . "startTime":"2021-04-05 16:06:34",
>  ① . "endTime":"2021-04-05 16:06:50",
>  ① . "mark":"FunTester誤差分析051606",
>  ① . "table":"eJztlU0KgkAYhveCd/gOoGDRygN0gOgCQgMJzRSOQS37d9nKZUfIZfcRjI7RB4EUWJo/afUNL3yzmvd5GH9UBTKs7lT0mXSZcwmC6HSMvO35sIe2Ae7QYdYgyxGgKurrrh6Tk7GQDPo2ZybMdMkc2xqBmHIN5jpnA9sSaR3pHNwWcDvLbBkd4FLj1sxsGwZuS7DIv0J/gcGxweDwMJmM8lTddzyOUhtDf4nBscYk91XSHvc+96yUIvEu844CNKVylEAV+itMJTyF6GKgHaZKrpyU2V+gemkbwfkGdfw8fvbeC9M3njvFouD3uUk23+mQYPT2v/pbhqr8wh2lGTYAgwzJkAzrxyBDMiTD+jHIkAzJsH4MMiTDPze8AjKhbpE="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

下面將隨機數離散係數增加一倍,配置sleep(0.1 + getRandomDouble() / 5),理論平均休眠時間200ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":202,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"5.3%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":93.76465072667604,
>  ① . "total":2000,
>  ① . "qps":99.00990099009901,
>  ① . "startTime":"2021-04-05 16:10:32",
>  ① . "endTime":"2021-04-05 16:10:54",
>  ① . "mark":"FunTester誤差分析051610",
>  ① . "table":"eJztlk0KwjAQRvdC7zAHUFAXQnsADyBeoGDAgonSKOjSf5euuvQIdul9CorHcEAUFKmxybQVEgaSVb730gmpUwGF0Z6ILpNjFl7j+Hw6nneby2EPzTqM+yHzeypbgFNx0rM6TI6GQjLoBpx5MK1JFgb+AMSEV2FW46wX+OJbxncOHgi47+U16i3gssr9qdd0XVwasNAaSbTAwmmHpWSTMWaN9YihS0uiJdZbDmFqEs2xPusRpyfRCkslk4RAXZmG4rVxNSc9Gs2jMEr0bMgtllmkzGQ/X5J8CY12Eg1l1rueJ+nzGMk6zyhxbt1oiLrYFtCiL+FRKxso/FCUcHo1oX8TcjX6z0+SavUvN+QXszK9B1SGJcCwhtbQGhaPYQ2toTUsHsMaWsNUwxsm4B0O"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

可以看到,誤差明顯增加了一些,下面再試試配置sleep(0.1 + getRandomDouble() / 2),理論休眠時間350ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":355,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"4.85%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":53.60493165371214,
>  ① . "total":2000,
>  ① . "qps":56.33802816901409,
>  ① . "startTime":"2021-04-05 16:13:14",
>  ① . "endTime":"2021-04-05 16:13:51",
>  ① . "mark":"FunTester誤差分析051613",
>  ① . "table":"eJztlk1qwzAQhfcB32EOkEASSKE+QA5QcgFDBDFUSrESSJdtfpdZaZkjJMveR5DSY2Sg0EUJthyN5FlYPLBXet+T5GclHXAY46WaCL0Qxc/lcv06Xw+779MRhn1YzAqRTV2mgKSTlHu9CP02V1rAJJcihVVPiyLPXkEtZRfee1JM80xVeVRzyFzB71zpYDAEqbsyW6Wj5yd8JUjhNaz5QOHjgHJKQ2AT1M2aT9Q/q3CO1mxQ9+0CuVqzRZVahnB2t6V1t2aNquFLT+GJQEXicNZi0dQ+DAGJ/splj6JF8iEj3S1auhC7R0VY1t+MSO/+0XjSNruitYlpyjYqdezviYicL3YlPQu8BxOwqA7/JJxqxTMNw8rxSRTt5hEvFfuqeixZwDtr0+lYYLQJ24RtwuYx2oReCW9YS+Wj"
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

沒想到反而有了點下降的趨勢,我們再將離散係數增加一倍,配置sleep(0.1 + getRandomDouble()),看一下效果。

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":590,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"10.51%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":30.336584403961957,
>  ① . "total":2000,
>  ① . "qps":33.898305084745765,
>  ① . "startTime":"2021-04-05 16:20:41",
>  ① . "endTime":"2021-04-05 16:21:47",
>  ① . "mark":"FunTester誤差分析051620",
>  ① . "table":"eJztl01qwkAYhvdC7vAdQMEfkJID9ADFCwQcMOBMS0bBLv136WqWHkGXvc+ApcfoQKkgtsn8fclEDC8kq3mfh8nMJFEDNK7nKRsQPiHZ1+l0/jied5vPwx66bZiMMpIMdYaAqBHld70Q/vbKOIFBSkkMsxYnWZqMgU1pE95blAzThBV1FGLQlMHPUHGn2wPKmzSZxZ32U189e7BwvaTYqeiYOFRsVdCbpFir/Nagtv1ZhdQoxVwltxKj+VL7fx9WuxQLFYNqvwTG5kgUdvUeKRwRvJA4ToZvIttXE5EKEcmFDB/Lmk6KlUoJaHaEZeNZUGocRpWT6p+bQRBLsVSpmNOMOkBkHfJwsQvpK9oafBkEsRbdLK7/Zepyuza57Ix1VLk1Ku1LozyruqxzQ7OarX8Tu/ubsVvDADAehg/DXMNv5swe1g=="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

這下誤差增加了不少,我都開始懷疑上一次實驗是否是偶然情況下才導致誤差並沒有升高。不過這個問題跟之後會講到的同步結束有關聯。到時候我修復完異步結束導致的誤差增加後,在回過來重複一下離散係數這個實驗。

基本結論還是,接口請求響應時間離散係數越大,誤差越大。

before & after

下面進行本期最後一個環節,就是beforeafter兩個方法中進行耗時操作,對誤差產生的影響。其實這裏不用進行試驗也能得出結論,耗時越長,誤差越大。原因就是在請求響應誤差一定的情況下,beforeafter執行所消耗的時間,就是增加了總時間T,請求次數相同,總時間T除以總請求次數,QPS肯定是降低的,再加上,QPS=T/count(req)本身就是低QPS計算公式,所以總的結論就是:在beforeafter方法中執行約耗時的操作,誤差越大。

下面再before中進行sleep(5)的操作,配置20線程,100請求次數,固定休眠時間100ms

~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"9.82%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":175.1006828926633,
>  ① . "total":2000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 16:32:31",
>  ① . "endTime":"2021-04-05 16:32:43",
>  ① . "mark":"FunTester誤差分析051632",
>  ① . "table":"eJztz0sKwjAQgOF9IXeYA7TQuuwBPID0AoUGDJgoTQt167NLV116BLv0PoWKxzA+ltUGtODAhIEvq0l+5oDFGecq4jrj6bWu2/OpLXeX4wFGPmTTlMeJzQpgDvv81oTrxVxpDpGQPITC0zwV8QxULl1YepInIlZ9b/T/QwoFz11h4PsgtSvjIgyC+/UHFd+epirNWJW83bAy08PGjB1bMx2/aqq1GTteKzrZm7Hj8YMhYM6Ay/8CKsQPFeKHCvFDhfihQvxQIX6oED9UiB8qxA8V4ocK8UOF+KFC/FAhfphzAzQA+zM="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~```

現在講休眠時間增加到`5s`,再看看結果。


```shell
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
>  {
>  ① . "rt":103,
>  ① . "failRate":0.0,
>  ① . "threads":20,
>  ① . "deviation":"33.29%",
>  ① . "errorRate":0.0,
>  ① . "executeTotal":2000,
>  ① . "qps2":129.5336787564767,
>  ① . "total":2000,
>  ① . "qps":194.1747572815534,
>  ① . "startTime":"2021-04-05 16:33:10",
>  ① . "endTime":"2021-04-05 16:33:25",
>  ① . "mark":"FunTester誤差分析051633",
>  ① . "table":"eJztz0EKgkAUgOG9MHd4B1DQVuEBOkB4AaGBBpopHAVbRlQuW7nsCLnsPoLRMRpqE4SOlIseveHB58aZ9zMHepxJpiKuU57cqqq5nJtifz0dYeRDOk94POtzBTCHdb815Xq1VJpDJCQPIfc0T0S8AJVJF9ae5DMRK9sb9j2kUPC8Kwx8H6R2ZZyHQTA2nwNUfHvqsjDTq+SfXqnLjRkLWzNvPFbo2KTlty52ZiwczLTxutGnMGeYe34XKsQPFeKHCvFDhfihQvxQIX6oED9UiB8qxA8V4ocK8UOF+KFC/DDnDnuZ/C4="
>  }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~

結論符合理論值,就不進行更多驗證了。

下期將分享一下,異常處理正則異步結束加鎖資源等因素對誤差的影響,歡迎持續關注。本來打算錄視頻演示的。可惜時間不是很多,沒法準備,以後有機會開個直播聊。


FunTester騰訊雲年度作者Boss直聘簽約作者GDevOps官方合作媒體,非著名測試開發。

點擊閱讀原文,查看公衆號歷史文章


本文分享自微信公衆號 - FunTester(NuclearTester)。
如有侵權,請聯繫 [email protected] 刪除。
本文參與“OSC源創計劃”,歡迎正在閱讀的你也加入,一起分享。

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章