gRPC client端请求响应速度慢问题


1.问题描述

1.开发板验证测试
服务端

/usrdata/grpc_test # ./grpc_server_timestamp
SyncServer listening on localhost:50051
[runtime] Create and run the server cost time 0.37311s
[runtime] The call to the server-side GetInfo interface ends and cost time 0.2816s

客户端


/usrdata/grpc_test # time ./grpc_client_timestamp 1 1
[runtime] The call to the client-side GetInfo interface ends and cost time 3.154608s
[runtime] The client RPC requests and accepts the server response total cost time 3.161419s
real    0m 3.20s
user    0m 0.08s
sys     0m 0.02s

开发板内部运行server与client程序,client响应速度要3秒。
PC端Ubuntu系统通过以太网通讯验证,居然更长。

root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client_timestamp 1 1
[runtime] The call to the client-side GetInfo interface ends and cost time 75.134398s
[runtime] The client RPC requests and accepts the server response total cost time 75.161900s

2.问题分析与解决

到底是哪里阻塞这么长时间呢?
GRPC_VERBOSITY=INFO的信息不足以分析问题,使用GRPC_VERBOSITY=DEBUG来详细查看grpc的运行日志。

/usrdata/grpc_test # GRPC_VERBOSITY=DEBUG ./grpc_client_timestamp 1 1
D0619 14:16:15.642466857    2556 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0619 14:16:15.643895763    2556 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0619 14:16:15.644129930    2556 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0619 14:16:15.644314930    2556 ev_posix.cc:174]            Using polling engine: epoll1
D0619 14:16:15.646215242    2556 dns_resolver_ares.cc:490]   Using ares dns resolver
D0619 14:16:18.679968522    2559 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0619 14:16:18.787043158    2556 subchannel.cc:1055]         New connected subchannel at 0x74b78 for subchannel 0xb5003968

发现dns_resolver_ares.cc:490] Using ares dns resolver这里居然阻塞了3秒,查看grpc源码。


大致定位到dns解析的问题。查看开发板/etc/resolv.conf 默认就为空。原来设计为拨号成功后配置运营商提供的dns,由于我只是在开发板单纯验证grpc功能,没插sim卡也没运行拨号程序。
推测是因为没有dns服务器导致解析很慢的。
echo “nameserver 8.8.8.8” >> /etc/resolv.conf 通过该命令配置dns服务器

/usrdata/grpc_test # echo "nameserver 8.8.8.8" >> /etc/resolv.conf
/usrdata/grpc_test #
/usrdata/grpc_test #
/usrdata/grpc_test # GRPC_VERBOSITY=DEBUG ./grpc_client_timestamp 1 1
D0619 14:24:26.808571357    2998 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0619 14:24:26.811212919    2998 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0619 14:24:26.811701253    2998 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0619 14:24:26.811869534    2998 ev_posix.cc:174]            Using polling engine: epoll1
D0619 14:24:26.813298388    2998 dns_resolver_ares.cc:490]   Using ares dns resolver
D0619 14:24:26.842104638    2998 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0619 14:24:26.937539013    2998 subchannel.cc:1055]         New connected subchannel at 0x7d9c8 for subchannel 0x72830

[runtime] The call to the client-side GetInfo interface ends and cost time 0.128943s
[runtime] The client RPC requests and accepts the server response total cost time 0.141080s

果然快很多。

于是我在pc端也验证了一下。

root@chenwr-VirtualBox:/media/sf_share# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8
root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client 1 1
D0622 10:43:59.845292688    3451 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0622 10:43:59.845550764    3451 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0622 10:43:59.845617206    3451 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0622 10:43:59.845680554    3451 ev_posix.cc:174]            Using polling engine: epoll1
D0622 10:43:59.846227281    3451 dns_resolver_ares.cc:490]   Using ares dns resolver
D0622 10:45:14.870530939    3451 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0622 10:45:14.973018726    3451 subchannel.cc:1055]         New connected subchannel at 0x204b390 for subchannel 0x7fecc4005120

[runtime] The call to the client-side GetInfo interface ends and cost time 75.134907s
[runtime] The client RPC requests and accepts the server response total cost time 75.142769s

但还是很慢。
因为我开发板和笔记本是通过以太网通讯,是否因为无法访问网络才导致域名解析速度慢?但是同样没有拨号为何开发板上的客户端就很快?

于是我将pc端的/etc/resolv.conf的域名ip配置成我开发板运行grpc server端的ip.

root@chenwr-VirtualBox:/media/sf_share# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
#nameserver 8.8.8.8
nameserver 192.168.1.3
root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client 1 1
D0622 10:48:41.357512939    3460 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0622 10:48:41.357584289    3460 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0622 10:48:41.357593292    3460 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0622 10:48:41.357602656    3460 ev_posix.cc:174]            Using polling engine: epoll1
D0622 10:48:41.358144920    3460 dns_resolver_ares.cc:490]   Using ares dns resolver
D0622 10:48:41.404296216    3460 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0622 10:48:41.511149159    3460 subchannel.cc:1055]         New connected subchannel at 0x1d4d650 for subchannel 0x1d4cc90

[runtime] The call to the client-side GetInfo interface ends and cost time 0.161359s
[runtime] The client RPC requests and accepts the server response total cost time 0.188645s

速度果然就正常了。
我的推测是nameserver 192.168.1.3的作用相当于域名解析的操作在192.168.1.3这台主机上进行,因为开发板server端建立的域名与192.168.1.3映射关系。因此pc端client程序请求连接的域名也需要对应到192.168.1.3中。

3.DNS知识点补充

3.1 什么是dns?

DNS(Domain Name System,域名系统),因特网上作为域名和IP地址相互映射的一个分布式数据库,能够使用户更方便的访问互联网,而不用去记住能够被机器直接读取的IP数串。通过主机名,最终得到该主机名对应的IP地址的过程叫做域名解析(或主机名解析)。

简单理解为ip太难记了,比如14.215.177.38和www.baidu.com 很显然后者更容易记住。
这个转换过程就由DNS来完成。

3.2 主机名与域名是不是同一个概念?

主机名就是每台主机的名字,就跟人名一样。
域名可以认为是主机在公网环境中的标识,在公网下,对应一个唯一的IP,例如我们访问百度的主页:www.baidu.com。
其实两者的作用都是一样的,都能够对应到唯一的IP。只不过应用场景不同叫法不一样,比如局域网叫主机名,就像你在家里,家人叫你小名一样。广域网(公网)就叫域名,外面的陌生人肯定要叫你大名了呀。

3.3 Linux端如何配置域名解析?

/etc/resolv.conf —DNS域名解析配置文件。/etc/resolv.conf是DNS客户机配置文件,用于设置DNS服务器的IP地址及DNS域名,还包含了主机的域名搜索顺序。

每行以一个关键字开头,后接配置参数。resolv.conf的关键字主要有四个,分别是:
nameserver #定义DNS服务器的IP地址
domain #定义本地域名
search #定义域名的搜索列表
sortlist #对返回的域名进行排序

简单验证一下。
1.将/etc/resolv.conf设置为空。
则无法ping通百度域名。

root@chenwr-pc:/home/workspace# ping www.baidu.com
ping: unknown host www.baidu.com

2.设置ubuntu系统resolv.conf默认配置

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1

能够ping通。

root@chenwr-pc:/home/workspace# ping www.baidu.com
PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.
64 bytes from 14.215.177.38: icmp_seq=1 ttl=53 time=15.3 ms
64 bytes from 14.215.177.38: icmp_seq=2 ttl=53 time=15.9 ms
64 bytes from 14.215.177.38: icmp_seq=3 ttl=53 time=14.7 ms
64 bytes from 14.215.177.38: icmp_seq=4 ttl=53 time=14.2 ms

3.4 127.0.1.1与127.0.0.1有啥区别?

回环设置是

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0

我把nameserver 设置为127.0.0.1试试看?
无法ping通,看来127.0.1.1也是一个dns服务器ip。百度了一下,127.0.1.1相当于代理服务器。


root@chenwr-pc:/home/workspace# ps -ef | grep dnsmasq
nobody    1236   816  0 08:27 ?        00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
root     18324 14592  0 19:01 pts/16   00:00:00 grep --color=auto dnsmasq

ubuntu下有一个本地的dns服务叫做dnsmasq,它是由NetworkManager控制的
它监听的本地地址,–listen-address=127.0.1.1 (ubuntu12.04及之前的版本 是 127.0.0.1), 这个地址是一个本地回环地址
而你真实的dns服务器地址,是被这个服务管理维护着的
由一张图表示就是:
local process -> local dnsmasq -> router -> ISP dns

相当于代理。开发板也有这个进程,具体怎么分配流程就不深入了。
可参考linux - ubuntu中 etc resolv.cnf中的127.0.1.1是什么地址 - SegmentFault 思否

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章