心跳
心跳，這個詞在不同人的腦袋裏，第一反應會想到不同的東西，比如說醫護人員，想到的是 ‘砰~砰~砰~’;linux網絡編程的人想到可能是客戶端與服務端的探活機制，比如TCP協議的心跳，或者應用層自己構建的心跳機制。那今天說的MySQL的複製心跳其實就屬於應用層自己構建的一種探活機制。
TCP協議心跳
做MySQL相關的工作，無論是哪個方向，都需要多多少少去了解TCP相關的知識，比如下面一大串的配置。
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 262144
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 32768
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 256
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_mem = 52533 70045 105066
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1當然，和TCP心跳相關的有如下幾個參考
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200關於TCP心跳更多細節，請看 TCP心跳設置。
mysqld複製心跳
客戶端使用複製協議與master建立複製鏈接之後，master 會啓用dump線程，發送binlog到客戶端，這裏之所以說客戶端，不說slave，意思是想說明，任何程序，無論是不是mysqld，只要遵守mysqld相關的協議，都可以向master發起binlog dump請求，master會按照客戶端的需求，發送相應的binlog。那複製心跳爲何物呢？主要用來做什麼呢？
啓用複製心跳
搭建過主從複製的小夥伴，對如下的命令應該非常熟悉
change master to master_hosr='',master_user='',master_password='',master_port=13307, ......但是有很多人可能不知道心跳的配置選項，使用change 命令可以指定心跳間隔,如下。當然如果不指定，心跳也是默認開啓的，間隔爲15秒。
MASTER_HEARTBEAT_PERIOD通過tcpdump抓包可以看到如下
16:23:49.625836 IP localhost.13307 > localhost.39414: Flags [P.], seq 2425355868:2425355914, ack 3652228843, win 350, options [nop,nop,TS val 549495480 ecr 549485480], length 46
16:23:49.625870 IP localhost.39414 > localhost.13307: Flags [.], ack 46, win 350, options [nop,nop,TS val 549495480 ecr 549495480], length 0
16:23:59.626298 IP localhost.13307 > localhost.39414: Flags [P.], seq 46:92, ack 1, win 350, options [nop,nop,TS val 549505480 ecr 549495480], length 46
16:23:59.626335 IP localhost.39414 > localhost.13307: Flags [.], ack 92, win 350, options [nop,nop,TS val 549505480 ecr 549505480], length 0
16:24:09.627078 IP localhost.13307 > localhost.39414: Flags [P.], seq 92:138, ack 1, win 350, options [nop,nop,TS val 549515480 ecr 549505480], length 46
16:24:09.627113 IP localhost.39414 > localhost.13307: Flags [.], ack 138, win 350, options [nop,nop,TS val 549515480 ecr 549515480], length 0
16:24:19.627594 IP localhost.13307 > localhost.39414: Flags [P.], seq 138:184, ack 1, win 350, options [nop,nop,TS val 549525480 ecr 549515480], length 46
16:24:19.627628 IP localhost.39414 > localhost.13307: Flags [.], ack 184, win 350, options [nop,nop,TS val 549525480 ecr 549525480], length 0
16:24:29.628152 IP localhost.13307 > localhost.39414: Flags [P.], seq 184:230, ack 1, win 350, options [nop,nop,TS val 549535480 ecr 549525480], length 46
16:24:29.628186 IP localhost.39414 > localhost.13307: Flags [.], ack 230, win 350, options [nop,nop,TS val 549535480 ecr 549535480], length 0
查看心跳間隔
通過查看mysql庫下的slave_master_info表可以查看當前複製通道的心跳間隔
*************************** 4. row ***************************
Number_of_lines: 25
Master_log_name: mysql-bin.001836
Master_log_pos: 298191716
Host: 100.107.14.181
User_name: *********
User_password: ************
Port: **********
Connect_retry: 60
Enabled_ssl: 0
Ssl_ca:
Ssl_capath:
Ssl_cert:
Ssl_cipher:
Ssl_key:
Ssl_verify_server_cert: 0
Heartbeat: 15 /*15秒*/
Bind:
Ignored_server_ids: 0
Uuid:**************
Retry_count: 86400
Ssl_crl:
Ssl_crlpath:
Enabled_auto_position: 1
Channel_name: rrxdebt4
Tls_version:同時也可以通過show 命令查看，如下
mysql> show global status like '%heart%';
+---------------------------+---------------------+
| Variable_name | Value |
+---------------------------+---------------------+
| Slave_heartbeat_period | 15.000 |
| Slave_last_heartbeat | 2018-07-15 14:21:47 |
| Slave_received_heartbeats | 578024 |
+---------------------------+---------------------+
3 rows in set (0.00 sec)mysqld 複製心跳是幹嘛的？
一句話：客戶端使用心跳來確認主機是否存活，網絡是否暢通等。可以這樣理解，客戶端作爲binlog的接收方，如果長時間沒有收到master發送的binlog，無法確認是網絡被隔離，還是master真的沒有寫入，所以在建立起復制關係時，指定複製心跳，以及時間間隔，意思爲，如果在指定的時間間隔內，master如果沒有寫入，則發送心跳到客戶端，來告訴客戶端，master還活着。
slave_net_timeout ？
從機的參數slave_net_timeout，什麼意思？
先看文檔，如下
The number of seconds to wait for more data from a master/slave connection before aborting the read. Setting this variable has no immediate effect. The state of the variable applies on all subsequent START SLAVE commands.大概的意思是，中斷讀取操作之前的等待時間，單位是秒。所以，如果此參數的設置小於heartbeat，會導致什麼問題呢？
問題就是，如果master的寫入不頻繁，可能會頻發發生io線程中斷，重連的問題。
擴展
io線程如何等待數據讀取
來看下io線程是如何等待新的數據的
Thread 74 (Thread 0x7fc2ac271700 (LWP 25290)):
#0 0x00007fc2e823674d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000000001ed8303 in vio_io_wait (vio=0x7fc250010d00, event=VIO_IO_EVENT_READ, timeout=30000) at /data/mysql-server-explain_ddl/vio/viosocket.c:786
#2 0x0000000001ed6f69 in vio_socket_io_wait (vio=0x7fc250010d00, event=VIO_IO_EVENT_READ) at /data/mysql-server-explain_ddl/vio/viosocket.c:77
#3 0x0000000001ed7054 in vio_read (vio=0x7fc250010d00, buf=0x7fc250010f20 "*", size=16384) at /data/mysql-server-explain_ddl/vio/viosocket.c:132
#4 0x0000000001ed7238 in vio_read_buff (vio=0x7fc250010d00, buf=0x7fc250014f50 "", size=4) at /data/mysql-server-explain_ddl/vio/viosocket.c:166
#5 0x000000000152151d in net_read_raw_loop (net=0x7fc25000e1f0, count=4) at /data/mysql-server-explain_ddl/sql/net_serv.cc:672
#6 0x0000000001521721 in net_read_packet_header (net=0x7fc25000e1f0) at /data/mysql-server-explain_ddl/sql/net_serv.cc:762
#7 0x0000000001521803 in net_read_packet (net=0x7fc25000e1f0, complen=0x7fc2ac270c18) at /data/mysql-server-explain_ddl/sql/net_serv.cc:822
#8 0x00000000015219ba in my_net_read (net=0x7fc25000e1f0) at /data/mysql-server-explain_ddl/sql/net_serv.cc:899
#9 0x000000000174650f in cli_safe_read_with_ok (mysql=0x7fc25000e1f0, parse_ok=0 '\000', is_data_packet=0x0) at /data/mysql-server-explain_ddl/sql-common/client.c:1055
#10 0x0000000001746882 in cli_safe_read (mysql=0x7fc25000e1f0, is_data_packet=0x0) at /data/mysql-server-explain_ddl/sql-common/client.c:1188
#11 0x000000000191d4bc in read_event (mysql=0x7fc25000e1f0, mi=0x561ebe0, suppress_warnings=0x7fc2ac270d71) at /data/mysql-server-explain_ddl/sql/rpl_slave.cc:4455
#12 0x0000000001920fdb in handle_slave_io (arg=0x561ebe0) at /data/mysql-server-explain_ddl/sql/rpl_slave.cc:5713
#13 0x0000000001e4a621 in pfs_spawn_thread (arg=0x7fc2340d4d50) at /data/mysql-server-explain_ddl/storage/perfschema/pfs.cc:2188
#14 0x00007fc2e8dad6ba in start_thread (arg=0x7fc2ac271700) at pthread_create.c:333
#15 0x00007fc2e824241d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109可能需要看下poll機制了
如下
POLL(2) Linux Programmer's Manual POLL(2)
NAME
poll, ppoll - wait for some event on a file descriptor
SYNOPSIS
#include <poll.h>
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
#define _GNU_SOURCE /* See feature_test_macros(7) */
#include <signal.h>
#include <poll.h>
int ppoll(struct pollfd *fds, nfds_t nfds,
const struct timespec *tmo_p, const sigset_t *sigmask);
DESCRIPTION
poll() performs a similar task to select(2): it waits for one of a set of file descriptors to become ready to perform I/O.
The set of file descriptors to be monitored is specified in the fds argument, which is an array of structures of the following form:
struct pollfd {
int fd; /* file descriptor */
short events; /* requested events */
short revents; /* returned events */
};
The caller should specify the number of items in the fds array in nfds.
The field fd contains a file descriptor for an open file. If this field is negative, then the corresponding events field is ignored and the revents field returns zero. (This provides an easy
way of ignoring a file descriptor for a single poll() call: simply negate the fd field. Note, however, that this technique can't be used to ignore file descriptor 0.)
The field events is an input parameter, a bit mask specifying the events the application is interested in for the file descriptor fd. This field may be specified as zero, in which case the only
events that can be returned in revents are POLLHUP, POLLERR, and POLLNVAL (see below).
The field revents is an output parameter, filled by the kernel with the events that actually occurred. The bits returned in revents can include any of those specified in events, or one of the
values POLLERR, POLLHUP, or POLLNVAL. (These three bits are meaningless in the events field, and will be set in the revents field whenever the corresponding condition is true.)
If none of the events requested (and no error) has occurred for any of the file descriptors, then poll() blocks until one of the events occurs.
The timeout argument specifies the number of milliseconds that poll() should block waiting for a file descriptor to become ready. The call will block until either:
* a file descriptor becomes ready;
* the call is interrupted by a signal handler; or
* the timeout expires.
Note that the timeout interval will be rounded up to the system clock granularity, and kernel scheduling delays mean that the blocking interval may overrun by a small amount. Specifying a nega‐
tive value in timeout means an infinite timeout. Specifying a timeout of zero causes poll() to return immediately, even if no file descriptors are ready.dump線程如何發送心跳
dump線程等待binlog位點更新，如果設置了心跳間隔，則在等待超時後，發送heartbeat event。其實就是最近的binlog file name+pos。
Thread 24 (Thread 0x7fbd44050700 (LWP 6474)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x0000000001985490 in native_cond_timedwait (cond=0x2db7fa0 <mysql_bin_log+1824>, mutex=0x2db7ee8 <mysql_bin_log+1640>, abstime=0x7fbd4404de50) at /data/mysql-server-explain_ddl/include/thr_cond.h:129
#2 0x00000000019857f6 in safe_cond_timedwait (cond=0x2db7fa0 <mysql_bin_log+1824>, mp=0x2db7ec0 <mysql_bin_log+1600>, abstime=0x7fbd4404de50, file=0x224ec38 "/data/mysql-server-explain_ddl/sql/binlog.cc", line=7683) at /data/mysql-server-explain_ddl/mysys/thr_cond.c:88
#3 0x00000000018d9126 in my_cond_timedwait (cond=0x2db7fa0 <mysql_bin_log+1824>, mp=0x2db7ec0 <mysql_bin_log+1600>, abstime=0x7fbd4404de50, file=0x224ec38 "/data/mysql-server-explain_ddl/sql/binlog.cc", line=7683) at /data/mysql-server-explain_ddl/include/thr_cond.h:180
#4 0x00000000018d96c3 in inline_mysql_cond_timedwait (that=0x2db7fa0 <mysql_bin_log+1824>, mutex=0x2db7ec0 <mysql_bin_log+1600>, abstime=0x7fbd4404de50, src_file=0x224ec38 "/data/mysql-server-explain_ddl/sql/binlog.cc", src_line=7683) at /data/mysql-server-explain_ddl/include/mysql/psi/mysql_thread.h:1229
#5 0x00000000018eb4ba in MYSQL_BIN_LOG::wait_for_update_bin_log (this=0x2db7880 <mysql_bin_log>, thd=0x7fbcfc000950, timeout=0x7fbd4404de50) at /data/mysql-server-explain_ddl/sql/binlog.cc:7683
#6 0x000000000190e4ca in Binlog_sender::wait_with_heartbeat (this=0x7fbd4404e5d0, log_pos=919) at /data/mysql-server-explain_ddl/sql/rpl_binlog_sender.cc:620
#7 0x000000000190e3f7 in Binlog_sender::wait_new_events (this=0x7fbd4404e5d0, log_pos=919) at /data/mysql-server-explain_ddl/sql/rpl_binlog_sender.cc:599
#8 0x000000000190e058 in Binlog_sender::get_binlog_end_pos (this=0x7fbd4404e5d0, log_cache=0x7fbd4404e020) at /data/mysql-server-explain_ddl/sql/rpl_binlog_sender.cc:365
#9 0x000000000190bd00 in Binlog_sender::send_binlog (this=0x7fbd4404e5d0, log_cache=0x7fbd4404e020, start_pos=123) at /data/mysql-server-explain_ddl/sql/rpl_binlog_sender.cc:313
#10 0x000000000190b8d4 in Binlog_sender::run (this=0x7fbd4404e5d0) at /data/mysql-server-explain_ddl/sql/rpl_binlog_sender.cc:225
#11 0x0000000001909470 in mysql_binlog_send (thd=0x7fbcfc000950, log_ident=0x7fbd4404f230 "", pos=4, slave_gtid_executed=0x7fbd4404f000, flags=0) at /data/mysql-server-explain_ddl/sql/rpl_master.cc:412
#12 0x0000000001909307 in com_binlog_dump_gtid (thd=0x7fbcfc000950, packet=0x7fbcfc0098a1 "", packet_length=86) at /data/mysql-server-explain_ddl/sql/rpl_master.cc:396
#13 0x000000000161b15c in dispatch_command (thd=0x7fbcfc000950, com_data=0x7fbd4404fe00, command=COM_BINLOG_DUMP_GTID) at /data/mysql-server-explain_ddl/sql/sql_parse.cc:1677
#14 0x00000000016194aa in do_command (thd=0x7fbcfc000950) at /data/mysql-server-explain_ddl/sql/sql_parse.cc:999
#15 0x000000000175ba57 in handle_connection (arg=0x4d71eb0) at /data/mysql-server-explain_ddl/sql/conn_handler/connection_handler_per_thread.cc:300
#16 0x0000000001e49759 in pfs_spawn_thread (arg=0x5012a50) at /data/mysql-server-explain_ddl/storage/perfschema/pfs.cc:2188
#17 0x00007fbd67c746ba in start_thread (arg=0x7fbd44050700) at pthread_create.c:333
#18 0x00007fbd6710941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109