gRPC client端請求響應速度慢問題


1.問題描述

1.開發板驗證測試
服務端

/usrdata/grpc_test # ./grpc_server_timestamp
SyncServer listening on localhost:50051
[runtime] Create and run the server cost time 0.37311s
[runtime] The call to the server-side GetInfo interface ends and cost time 0.2816s

客戶端


/usrdata/grpc_test # time ./grpc_client_timestamp 1 1
[runtime] The call to the client-side GetInfo interface ends and cost time 3.154608s
[runtime] The client RPC requests and accepts the server response total cost time 3.161419s
real    0m 3.20s
user    0m 0.08s
sys     0m 0.02s

開發板內部運行server與client程序,client響應速度要3秒。
PC端Ubuntu系統通過以太網通訊驗證,居然更長。

root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client_timestamp 1 1
[runtime] The call to the client-side GetInfo interface ends and cost time 75.134398s
[runtime] The client RPC requests and accepts the server response total cost time 75.161900s

2.問題分析與解決

到底是哪裏阻塞這麼長時間呢?
GRPC_VERBOSITY=INFO的信息不足以分析問題,使用GRPC_VERBOSITY=DEBUG來詳細查看grpc的運行日誌。

/usrdata/grpc_test # GRPC_VERBOSITY=DEBUG ./grpc_client_timestamp 1 1
D0619 14:16:15.642466857    2556 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0619 14:16:15.643895763    2556 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0619 14:16:15.644129930    2556 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0619 14:16:15.644314930    2556 ev_posix.cc:174]            Using polling engine: epoll1
D0619 14:16:15.646215242    2556 dns_resolver_ares.cc:490]   Using ares dns resolver
D0619 14:16:18.679968522    2559 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0619 14:16:18.787043158    2556 subchannel.cc:1055]         New connected subchannel at 0x74b78 for subchannel 0xb5003968

發現dns_resolver_ares.cc:490] Using ares dns resolver這裏居然阻塞了3秒,查看grpc源碼。


大致定位到dns解析的問題。查看開發板/etc/resolv.conf 默認就爲空。原來設計爲撥號成功後配置運營商提供的dns,由於我只是在開發板單純驗證grpc功能,沒插sim卡也沒運行撥號程序。
推測是因爲沒有dns服務器導致解析很慢的。
echo “nameserver 8.8.8.8” >> /etc/resolv.conf 通過該命令配置dns服務器

/usrdata/grpc_test # echo "nameserver 8.8.8.8" >> /etc/resolv.conf
/usrdata/grpc_test #
/usrdata/grpc_test #
/usrdata/grpc_test # GRPC_VERBOSITY=DEBUG ./grpc_client_timestamp 1 1
D0619 14:24:26.808571357    2998 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0619 14:24:26.811212919    2998 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0619 14:24:26.811701253    2998 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0619 14:24:26.811869534    2998 ev_posix.cc:174]            Using polling engine: epoll1
D0619 14:24:26.813298388    2998 dns_resolver_ares.cc:490]   Using ares dns resolver
D0619 14:24:26.842104638    2998 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0619 14:24:26.937539013    2998 subchannel.cc:1055]         New connected subchannel at 0x7d9c8 for subchannel 0x72830

[runtime] The call to the client-side GetInfo interface ends and cost time 0.128943s
[runtime] The client RPC requests and accepts the server response total cost time 0.141080s

果然快很多。

於是我在pc端也驗證了一下。

root@chenwr-VirtualBox:/media/sf_share# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8
root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client 1 1
D0622 10:43:59.845292688    3451 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0622 10:43:59.845550764    3451 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0622 10:43:59.845617206    3451 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0622 10:43:59.845680554    3451 ev_posix.cc:174]            Using polling engine: epoll1
D0622 10:43:59.846227281    3451 dns_resolver_ares.cc:490]   Using ares dns resolver
D0622 10:45:14.870530939    3451 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0622 10:45:14.973018726    3451 subchannel.cc:1055]         New connected subchannel at 0x204b390 for subchannel 0x7fecc4005120

[runtime] The call to the client-side GetInfo interface ends and cost time 75.134907s
[runtime] The client RPC requests and accepts the server response total cost time 75.142769s

但還是很慢。
因爲我開發板和筆記本是通過以太網通訊,是否因爲無法訪問網絡才導致域名解析速度慢?但是同樣沒有撥號爲何開發板上的客戶端就很快?

於是我將pc端的/etc/resolv.conf的域名ip配置成我開發板運行grpc server端的ip.

root@chenwr-VirtualBox:/media/sf_share# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
#nameserver 8.8.8.8
nameserver 192.168.1.3
root@chenwr-VirtualBox:/media/sf_share# GRPC_VERBOSITY=DEBUG ./greeter_client 1 1
D0622 10:48:41.357512939    3460 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
I0622 10:48:41.357584289    3460 ev_epollex_linux.cc:1631]   Skipping epollex because it is not supported.
I0622 10:48:41.357593292    3460 ev_epoll1_linux.cc:116]     grpc epoll fd: 3
D0622 10:48:41.357602656    3460 ev_posix.cc:174]            Using polling engine: epoll1
D0622 10:48:41.358144920    3460 dns_resolver_ares.cc:490]   Using ares dns resolver
D0622 10:48:41.404296216    3460 tcp_posix.cc:1812]          cannot set inq fd=5 errno=92
I0622 10:48:41.511149159    3460 subchannel.cc:1055]         New connected subchannel at 0x1d4d650 for subchannel 0x1d4cc90

[runtime] The call to the client-side GetInfo interface ends and cost time 0.161359s
[runtime] The client RPC requests and accepts the server response total cost time 0.188645s

速度果然就正常了。
我的推測是nameserver 192.168.1.3的作用相當於域名解析的操作在192.168.1.3這臺主機上進行,因爲開發板server端建立的域名與192.168.1.3映射關係。因此pc端client程序請求連接的域名也需要對應到192.168.1.3中。

3.DNS知識點補充

3.1 什麼是dns?

DNS(Domain Name System,域名系統),因特網上作爲域名和IP地址相互映射的一個分佈式數據庫,能夠使用戶更方便的訪問互聯網,而不用去記住能夠被機器直接讀取的IP數串。通過主機名,最終得到該主機名對應的IP地址的過程叫做域名解析(或主機名解析)。

簡單理解爲ip太難記了,比如14.215.177.38和www.baidu.com 很顯然後者更容易記住。
這個轉換過程就由DNS來完成。

3.2 主機名與域名是不是同一個概念?

主機名就是每臺主機的名字,就跟人名一樣。
域名可以認爲是主機在公網環境中的標識,在公網下,對應一個唯一的IP,例如我們訪問百度的主頁:www.baidu.com。
其實兩者的作用都是一樣的,都能夠對應到唯一的IP。只不過應用場景不同叫法不一樣,比如局域網叫主機名,就像你在家裏,家人叫你小名一樣。廣域網(公網)就叫域名,外面的陌生人肯定要叫你大名了呀。

3.3 Linux端如何配置域名解析?

/etc/resolv.conf —DNS域名解析配置文件。/etc/resolv.conf是DNS客戶機配置文件,用於設置DNS服務器的IP地址及DNS域名,還包含了主機的域名搜索順序。

每行以一個關鍵字開頭,後接配置參數。resolv.conf的關鍵字主要有四個,分別是:
nameserver #定義DNS服務器的IP地址
domain #定義本地域名
search #定義域名的搜索列表
sortlist #對返回的域名進行排序

簡單驗證一下。
1.將/etc/resolv.conf設置爲空。
則無法ping通百度域名。

root@chenwr-pc:/home/workspace# ping www.baidu.com
ping: unknown host www.baidu.com

2.設置ubuntu系統resolv.conf默認配置

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1

能夠ping通。

root@chenwr-pc:/home/workspace# ping www.baidu.com
PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.
64 bytes from 14.215.177.38: icmp_seq=1 ttl=53 time=15.3 ms
64 bytes from 14.215.177.38: icmp_seq=2 ttl=53 time=15.9 ms
64 bytes from 14.215.177.38: icmp_seq=3 ttl=53 time=14.7 ms
64 bytes from 14.215.177.38: icmp_seq=4 ttl=53 time=14.2 ms

3.4 127.0.1.1與127.0.0.1有啥區別?

迴環設置是

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0

我把nameserver 設置爲127.0.0.1試試看?
無法ping通,看來127.0.1.1也是一個dns服務器ip。百度了一下,127.0.1.1相當於代理服務器。


root@chenwr-pc:/home/workspace# ps -ef | grep dnsmasq
nobody    1236   816  0 08:27 ?        00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
root     18324 14592  0 19:01 pts/16   00:00:00 grep --color=auto dnsmasq

ubuntu下有一個本地的dns服務叫做dnsmasq,它是由NetworkManager控制的
它監聽的本地地址,–listen-address=127.0.1.1 (ubuntu12.04及之前的版本 是 127.0.0.1), 這個地址是一個本地迴環地址
而你真實的dns服務器地址,是被這個服務管理維護着的
由一張圖表示就是:
local process -> local dnsmasq -> router -> ISP dns

相當於代理。開發板也有這個進程,具體怎麼分配流程就不深入了。
可參考linux - ubuntu中 etc resolv.cnf中的127.0.1.1是什麼地址 - SegmentFault 思否

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章