kubernetes flannel neutron calico 网络性能测试分析
环境准备
- OS: Ubuntu 14.04
- kubernetes 版本：1.3.5
- flannel 版本： 0.5.5
- calico 版本: 1.6.0
- neutron 版本：6.0.0
- iperf 版本：2.0.5
- 物理机配置：
4 Intel(R) Core(TM) i5-4460S CPU @ 2.90GHz
8G memory千兆网卡
搭建kubernetes相关组件，并配置不同的网络模式
|
节点类型
|
ip地址
|
备注
|
|
master/node
|
10.100.100.244
|
k8s的master跟node节点共用
|
|
node
|
10.100.100.245
|
|
|
neutron网络节点
|
10.100.100.246
|
为k8s提供neutron网络
|
root@hc0:~# kubectl get nodeNAME STATUS AGE
10.100.100.244 Ready 6d
10.100.100.245 Ready 6d
性能评测指标：
- ping延迟: 用ping测试hosts之间和pods之间的延迟
- 带宽测试: 用iperf测试hosts之间和pods之间的带宽
- HTTP性能测试: 部署单进程nginx server并使用apache benchmark(ab)测试
物理机性能评测
ping延迟
登录到10.100.100.244，运行ping 10.100.100.245测试延迟：
root@hc0:~# ping 10.100.100.245PING 10.100.100.245 (10.100.100.245) 56(84) bytes of data.
64 bytes from 10.100.100.245: icmp_seq=1 ttl=64 time=0.247 ms
64 bytes from 10.100.100.245: icmp_seq=2 ttl=64 time=0.221 ms
64 bytes from 10.100.100.245: icmp_seq=3 ttl=64 time=0.200 ms
64 bytes from 10.100.100.245: icmp_seq=4 ttl=64 time=0.234 ms
64 bytes from 10.100.100.245: icmp_seq=5 ttl=64 time=0.211 ms
64 bytes from 10.100.100.245: icmp_seq=6 ttl=64 time=0.212 ms
64 bytes from 10.100.100.245: icmp_seq=7 ttl=64 time=0.249 ms
64 bytes from 10.100.100.245: icmp_seq=8 ttl=64 time=0.227 ms
64 bytes from 10.100.100.245: icmp_seq=9 ttl=64 time=0.245 ms
64 bytes from 10.100.100.245: icmp_seq=10 ttl=64 time=0.278 ms
64 bytes from 10.100.100.245: icmp_seq=11 ttl=64 time=0.206 ms
平均值为： 0.244842ms
注：可以通过下面的脚本例子计算ping延迟的平均值
root@hc0:~#
ping 10.100.100.245 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'0.244842ms
相互之间多次ping测试。延迟一般在0.22ms-0.26ms之间
iperf带宽
apt-get install iperf 安装iperf后，在10.100.100.245上启动iperf server:
iperf -s，在10.100.100.244上启动客户端，执行下面命令:root@hc0:~# iperf -c 10.100.100.245 -i 1
------------------------------------------------------------
Client connecting to 10.100.100.245, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.244 port 49980 connected with 10.100.100.245 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 113 MBytes 945 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 942 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 942 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 943 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 943 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 941 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 942 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 943 Mbits/sec
[ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec
root@hc0:~# iperf -c 10.100.100.245 -u -t 60 -b 1000M
------------------------------------------------------------
Client connecting to 10.100.100.245, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.244 port 53440 connected with 10.100.100.245 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.67 GBytes 811 Mbits/sec
[ 3] Sent 4139127 datagrams
[ 3] Server Report:
[ 3] 0.0-60.0 sec 5.66 GBytes810 Mbits/sec 0.009 ms 7469/4139126 (0.18%)
[ 3] 0.0-60.0 sec 1 datagrams received out-of-order
root@hc1:~# iperf -c 10.100.100.244 -u -t 60 -b 2000M
------------------------------------------------------------
Client connecting to 10.100.100.244, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.245 port 46402 connected with 10.100.100.244 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.67 GBytes 811 Mbits/sec
[ 3] Sent 4138997 datagrams
[ 3] Server Report:
[ 3] 0.0-60.0 sec 5.66 GBytes810 Mbits/sec 0.043 ms 3929/4138996 (0.095%)
[ 3] 0.0-60.0 sec 1 datagrams received out-of-order
物理机iperf带宽 942 Mbits/sec
nginx
benchmark
先在10.100.100.244上，执行
docker run --net=host nginx启动一个单进程的nginx服务，然后在10.100.100.245上使用ab
-n 90000 -c 50 http://10.100.100.244/开始压测，结果如下：root@hc1:~# ab -n 90000 -c 50 http://10.100.100.244/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.100.100.244 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 10.100.100.244
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 7.235 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 12439.61 [#/sec] (mean)
Time per request: 4.019 [ms] (mean)
Time per request: 0.080 [ms] (mean, across all concurrent requests)
Transfer rate: 10265.11 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 2
Processing: 1 4 1.0 4 9
Waiting: 1 4 1.0 4 9
Total: 1 4 0.9 4 9
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 5
90% 5
95% 5
98% 5
99% 6
100% 9 (longest request)
neutron模式评测
ping延迟
在pod之间的ping延迟，启动两个iperf pod，启动在不同的host上。然后执行下面的命令获得ping延迟:
root@hc0:~/yaml# kubectl get pod -owide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
net1-sub1 net-test-4v9av 1/1 Running 0 29s 192.168.11.33 10.100.100.245
net1-sub1 net-test-ztu5r 1/1 Running 0 29s 192.168.11.34 10.100.100.244
root@hc0:~# kubectl exec -it net-test-ztu5r bash --namespace=net1-sub1
root@net-test-ztu5r:/#
ping
192.168.11.33 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'0.747789ms
相互之间多次ping测试，延迟一般在0.71ms-0.77ms之间
iperf带宽
pod之间的iperf带宽:
net-test-4v9av作为server。然后从net-test-ztu5r访问net-test-4v9av的IP来测试带宽。
root@net-test-ztu5r:/# iperf -c 192.168.11.33 -u -t 60 -b 2000M
------------------------------------------------------------
Client connecting to 192.168.11.33, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.11.34 port 40863 connected with 192.168.11.33 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.66 GBytes 810 Mbits/sec
[ 3] Sent 4131942 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
nginx benchmark
pod内的nginx性能：
用RC创建Nginx pod，以及service:
kubectl create -f nginx-rc.yamlkubectl create -f nginx-svc.yamlroot@hc0:~# kubectl get pod --namespace=net1-sub1 -owide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-4v9av 1/1 Running 0 25m 192.168.11.33 10.100.100.245
net-test-ztu5r 1/1 Running 0 25m 192.168.11.34 10.100.100.244
nginx-0v5r8 1/1 Running 0 30s 192.168.11.36 10.100.100.244
nginx-pnxzt 1/1 Running 0 30s 192.168.11.35 10.100.100.245
net-test-4v9av pod使用podIP访问
ab
-n 90000 -c 50 http://192.168.11.36/，性能相近。：root@net-test-4v9av:/# ab -n 90000 -c 50 http://192.168.11.36/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.11.36 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 192.168.11.36
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 9.928 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 9065.38 [#/sec] (mean)
Time per request: 5.515 [ms] (mean)
Time per request: 0.110 [ms] (mean, across all concurrent requests)
Transfer rate: 7480.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.3 0 12
Processing: 0 4 1.7 5 11
Waiting: 0 4 1.7 5 11
Total: 0 5 1.4 6 21
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 7
98% 9
99% 9
100% 21 (longest request)
使用flannel vxlan模式评测：
pod之间的ping延迟：
操作步骤同上
root@hc0:~/yaml# kubectl get pod -owide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
net1-sub1 net-test-ipgc5 1/1 Running 0 9s 172.16.21.2 10.100.100.244
net1-sub1 net-test-iyfds 1/1 Running 0 9s 172.16.78.2 10.100.100.245
root@hc0:~# kubectl exec -it net-test-ipgc5
bash --namespace=net1-sub1
root@net-test-ipgc5:/#
ping 172.16.78.2
| head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.379456ms
平均延迟：0.379456ms
pod之间的带宽
操作步骤同上
root@net-test-iyfds:/#
iperf -c 172.16.21.2
-t 60------------------------------------------------------------
Client connecting to 172.16.21.2,
TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local172.16.78.2
port 45646 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.28 GBytes 910 Mbits/sec
flannel的iperf带宽 910 Mbits/sec
nginx性能
操作步骤同上
root@net-test-b917n:/# ab -n 90000 -c 50 http://172.16.21.3/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking
172.16.21.3 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname:
172.16.21.3
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 8.945 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 10804.44 [#/sec] (mean)
Time per request: 4.928 [ms] (mean)
Time per request: 0.093 [ms] (mean, across all concurrent requests)
Transfer rate: 8814.89 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 5
Processing: 1 4 1.3 5 10
Waiting: 1 4 1.3 5 9
Total: 2 5 1.3 5 10
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 5
95% 6
98% 6
99% 6
100% 10 (longest request)
Calico网络性能
- 根据http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/#configuring-kubernetes完成集群的部署
ping延迟
评测方法同上
root@hc0:# kubectl get pod -owide --namespace=net1-sub1
NAME READY STATUS RESTARTS AGE IP NODE
net-test-b917n 1/1 Running 0 11m 192.168.145.132 10.100.100.245
net-test-w98x9 1/1 Running 0 11m 192.168.202.4 10.100.100.244
root@net-test-w98x9:/# ping 192.168.145.132 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.267684ms
多次测试平均延迟：0.25ms-0.31ms之间
iperf带宽
评测方法同上
tcp：
root@net-test-w98x9:/# iperf -s ------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.202.4 port 5001 connected with 192.168.145.132 port 54228
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.0 sec 3.29 GBytes 941 Mbits/sec
root@net-test-b917n:/# iperf -c 192.168.202.4 -t 60
------------------------------------------------------------
Client connecting to 192.168.202.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.145.132 port 54255 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.58 GBytes 942 Mbits/sec
udp：
root@net-test-b917n:/# iperf -c 192.168.202.4 -u -b 1000M -t 10
------------------------------------------------------------
Client connecting to 192.168.202.4, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.145.132 port 60316 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 968 MBytes 812 Mbits/sec
[ 3] Sent 690167 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 966 MBytes 811 Mbits/sec 0.028 ms 762/690166 (0.11%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
calico的iperf带宽 941 Mbits/sec
nginx benchmark
评测方法同上
root@net-test-b917n:/# ab -n 90000 -c 50 http://192.168.202.6/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.202.6 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 192.168.202.6
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 8.330 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 10031.23 [#/sec] (mean)
Time per request: 4.886 [ms] (mean)
Time per request: 0.093 [ms] (mean, across all concurrent requests)
Transfer rate: 8915.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 5
Processing: 1 4 1.3 5 10
Waiting: 1 4 1.3 5 9
Total: 2 5 1.3 5 10
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 5
95% 6
98% 6
99% 6
100% 10 (longest request)
结论
| 网络类型 | 延迟 | 带宽 | nginx(QPS/延迟) |
| 物理 | 0.244842ms | 942Mb/s | 12439.61/4.019 |
| neutron | 0.747789ms | 907Mb/s | 9065.38/5.515 |
| flannel vxlan | 0.379456ms | 910Mb/s | 10031.23/4.886 |
| Calico | 0.267684ms | 941Mb/s | 10804.44/4.628 |
TODO
- ovs网络性能测试
- ovn网络性能测试
/***************************2月24日更新**************************/
ovs vxlan网络性能
ping延迟
评测方法同上
root@hc0:# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-278vs 1/1 Running 0 25s 172.16.75.2 10.100.100.244
net-test-vtva6 1/1 Running 0 25s 172.16.61.2 10.100.100.245
root@net-test-278vs:/# ping 172.16.61.2 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.35475ms
iperf带宽
评测方法同上
tcp：
root@net-test-278vs:/# iperf -c 172.16.61.2 -t 20
------------------------------------------------------------
Client connecting to 172.16.61.2, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.75.2 port 39574 connected with 172.16.61.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-20.0 sec 2.14 GBytes 917 Mbits/sec
ovs的iperf带宽 917 Mbits/sec
nginx benchmark
评测方法同上
root@net-test-278vs:/# ab -n 90000 -c 50 http://172.16.61.3/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 172.16.61.3 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 172.16.80.2
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 7.855 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second:11457.32 [#/sec] (mean)
Time per request:4.364 [ms] (mean)
Time per request: 0.087 [ms] (mean, across all concurrent requests)
Transfer rate: 9454.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 0.3 2 4
Processing: 1 2 0.5 2 13
Waiting: 1 2 0.5 2 13
Total: 2 4 0.4 4 15
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 5
80% 5
90% 5
95% 5
98% 5
99% 5
100% 15 (longest request)
结论
| 网络类型 | 延迟 | 带宽 | nginx(QPS/延迟) |
|---|---|---|---|
| 物理 | 0.244842ms | 942Mb/s | 12439.61/4.019 |
| neutron | 0.747789ms | 907Mb/s | 9065.38/5.515 |
| flannel vxlan | 0.727158ms | 910Mb/s | 10031.23/4.886 |
| Calico | 0.267684ms | 941Mb/s | 10804.44/4.628 |
| ovs vxlan | 0.35475ms | 917Mb/s | 11457.32/4.364 |