前段時間在解決celery和rabbitmq之間的TCP連接問題，今天剛好看到有關於TCP keeplive的東西，覺得非常棒，就記錄下來。
摘抄來自http://www.firefoxbug.com/index.php/archives/2805/
理解Keepalive(1)
大家都聽過keepalive，但是其實對於keepalive這個詞還是很晦澀的，至少我一直都只知道一個大概，直到之前排查線上一些問題，發現keepalive還是有很多玄機的。其實keepalive有兩種，一種是TCP層的keepalive，另一種是HTTP層的Keep-Alive。這篇文章先說說tcp層的keepalive
tcp keepalive
設想有一種場景：A和B兩邊通過三次握手建立好TCP連接，然後突然間B就宕機了，之後時間內B再也沒有起來。如果B宕機後A和B一直沒有數據通信的需求，A就永遠都發現不了B已經掛了，那麼A的內核裏還維護着一份關於A&B之間TCP連接的信息，浪費系統資源。於是在TCP層面引入了keepalive的機制，A會定期給B發空的數據包，通俗講就是心跳包，一旦發現到B的網絡不通就關閉連接。這一點在LVS內尤爲明顯，因爲LVS維護着兩邊大量的連接狀態信息，一旦超時就需要釋放連接。
Linux內核對於tcp keepalive的調整主要有以下三個參數
1. tcp_keepalive_time
the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further
tcp_keepalive_intvl
the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
tcp_keepalive_probes
the number of unacknowledged probes to send before considering the connection dead and notifying the application layer
Example
75
$ cat /proc/sys/net/ipv4/tcp_keepalive_probes
9
當tcp發現有tcp_keepalive_time(7200)秒未收到對端數據後，開始以間隔tcp_keepalive_intvl(75)秒的頻率發送的空心跳包，如果連續tcp_keepalive_probes(9)次以上未響應代碼對端已經down了，close連接
在socket編程時候，可以調用setsockopt指定不同的宏來更改上面幾個參數
TCP_KEEPCNT: tcp_keepalive_probes
TCP_KEEPIDLE: tcp_keepalive_time
TCP_KEEPINTVL: tcp_keepalive_intvl
Nginx配置tcp keepalive
Nginx對於keepalive的配置有一大堆，大夥每次看都迷茫了，其實Nginx涉及到tcp層面的keepalive只有一個：so_keepalive。它屬於listen指令的配置參數，具體配置
so_keepalive=on|off|[keepidle]:[keepintvl]:[keepcnt]
this parameter (1.1.11) configures the “TCP keepalive” behavior for the listening socket. If this parameter is omitted then the operating system’s settings will be in effect for the socket. If it is set to the value “on”, the SO_KEEPALIVE option is turned on for the socket. If it is set to the value “off”, the SO_KEEPALIVE option is turned off for the socket. Some operating systems support setting of TCP keepalive parameters on a per-socket basis using the TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPCNT socket options. On such systems (currently, Linux 2.4+, NetBSD 5+, and FreeBSD 9.0-STABLE), they can be configured using the keepidle, keepintvl, and keepcnt parameters. One or two parameters may be omitted, in which case the system default setting for the corresponding socket option will be in effect.
Example
so_keepalive=30m::10
will set the idle timeout (TCP_KEEPIDLE) to 30 minutes, leave the probe interval (TCP_KEEPINTVL) at its system default, and set the probes count (TCP_KEEPCNT) to 10 probes.
在Nginx的代碼裏可以看到
./src/http/ngx_http_core_module.c
static ngx_command_t ngx_http_core_commands[] = {
…
// listen 指令解析 –>> call ngx_http_core_listen()
{ ngx_string(“listen”),
NGX_HTTP_SRV_CONF|NGX_CONF_1MORE,
ngx_http_core_listen,
NGX_HTTP_SRV_CONF_OFFSET,
0,
NULL },
…
}
static char *
ngx_http_core_listen(ngx_conf_t *cf, ngx_command_t *cmd, void *conf){
…
// 下面就是 so_keepalive 後面的參數解析
if (ngx_strncmp(value[n].data, “so_keepalive=”, 13) == 0) {
if (ngx_strcmp(&value[n].data[13], "on") == 0) {
lsopt.so_keepalive = 1;
} else if (ngx_strcmp(&value[n].data[13], "off") == 0) {
lsopt.so_keepalive = 2;
} else {
// 自定義系統keepalive的相關設置
...
}
if (ngx_http_add_listen(cf, cscf, &lsopt) == NGX_OK) {
return NGX_CONF_OK;
}
}
./src/core/ngx_connection.c
if (ls[i].keepidle) {
value = ls[i].keepidle;
// 設置 tcp_keepalive_time
if (setsockopt(ls[i].fd, IPPROTO_TCP, TCP_KEEPIDLE,
(const void *) &value, sizeof(int))
== -1)
{
ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno,
"setsockopt(TCP_KEEPIDLE, %d) %V failed, ignored",
value, &ls[i].addr_text);
}
}
if (ls[i].keepintvl) {
value = ls[i].keepintvl;
// 設置 tcp_keepalive_intvl
if (setsockopt(ls[i].fd, IPPROTO_TCP, TCP_KEEPINTVL,
(const void *) &value, sizeof(int))
== -1)
{
ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno,
"setsockopt(TCP_KEEPINTVL, %d) %V failed, ignored",
value, &ls[i].addr_text);
}
}
if (ls[i].keepcnt) {
// 設置 tcp_keepalive_intvl
if (setsockopt(ls[i].fd, IPPROTO_TCP, TCP_KEEPCNT,
(const void *) &ls[i].keepcnt, sizeof(int))
== -1)
{
ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno,
"setsockopt(TCP_KEEPCNT, %d) %V failed, ignored",
ls[i].keepcnt, &ls[i].addr_text);
}
}
總結
這篇文章說了TCP層面的keepalive相關知識以及Nginx的支持tcp keepalive的配置。tcp層面的keepalive存在更多意義上是爲了檢測兩端連接是否正常，注重點是在於連接的本身！要和HTTP層面的keepaplive區分開來，明白這點很重要。