15、 Heartbeat+DRBD+MySQL高可用架構方案與實施過程細節

原創 20has 2018-09-11 05:21

15、 Heartbeat+DRBD+MySQL高可用架構方案與實施過程細節

參考自：http://oldboy.blog.51cto.com/2561410/1240412


heartbeat和keepalived應用場景及區別

很多網友說爲什麼不使用keepalived而使用長期不更新的heartbeat，下面說一下它們之間的應用場景及區別：

1、對於web，db，負載均衡(lvs,haproxy,nginx)等，heartbeat和keepalived都可以實現

2、lvs最好和keepalived結合，因爲keepalived最初就是爲lvs產生的，(heartbeat沒有對RS的健康檢查功能，heartbeat可以通過ldircetord來進行健康檢查的功能)

3、mysql雙主多從，NFS/MFS存儲，他們的特點是需要數據同步，這樣的業務最好使用heartbeat，因爲heartbeat有自帶的drbd腳本


總結：

無數據同步的應用程序高可用可選擇keepalived， 

有數據同步的應用程序高可用可選擇heartbeat（DRBD） 。

1、

安裝部署準備

（1）架構拓撲

spacer.gif

架構說明：

一主多從最常用的架構，多個從庫可以使用lvs來提供讀的負載均衡。

解決一主單點的問題，當主庫宕機後，可以實現主庫宕機後備節點自動接管，所有的從庫會自動和新的主庫進行同步，實現了mysql主庫的熱備方案


（2）系統環境：

spacer.gif

（3）部署環境

spacer.gif

（4）主庫服務器數據分區信息

spacer.gif

2、heatbeat安裝部署

(1)、配置服務器間心跳連接路由

主節點

[root@master1 ~]# route add -host 172.16.4.3 dev eth2<==到對端的心跳路由

[root@master1 ~]# route add -host 172.168.4.3 dev eth3<==到對端的DRBD數據路由


備節點

[root@master2 ~]# route add -host 172.16.4.2 dev eth2

[root@master2 ~]# route add -host 172.168.4.2 dev eth3


(2)、安裝heartbeat

[root@master1 ~]# yum install heartbeat -y

[root@master1 ~]# yum install heartbeat -y

提示：需要執行兩遍安裝heartbeat操作


(3)、配置heartbeat

主備節點兩端的配置文件(ha.cfauthkeysharesources)完全相同

1)、ha.cf

[root@master1 ~]# vim /etc/ha.d/ha.cf

#log configure

debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility local1

#options configure

keepalive 2

deadtime 30

warntime 10

initdead 120

#bcast eth2

mcast eth2 225.0.0.7 694 1 0

#node configure

auto_failback on

node master1 <==主節點主機名

node master2 <==備節點主機名

crm no

2)、配置authkeys

[root@master1 ~]# vim /etc/ha.d/authkeys

auth 1

1 sha1 47e9336850f1db6fa58bc470bc9b7810eb397f04

3)、配置haresources

[root@master1 ~]# vim /etc/ha.d/haresources

master1 IPaddr::192.168.4.1/16/eth1

#master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld

說明：

drbddisk::data <==啓動drbd data資源，相當於執行/etc/ha.d/resource.d/drbddisk data stop/start操作

Filesystem::/dev/drbd1::/data::ext3 <==drbd分區掛載到/data目錄，相當於執行/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 stop/start <==相當於系統中執行mount /dev/drbd1 /data

mysql <==啓動mysql服務腳本，相當於/etc/init.d/mysql stop/start

(4)、啓動heartbeat

[root@master1 ~]# /etc/init.d/heartbeat start

[root@master1 ~]# chkconfig heartbeat off

說明：關閉開機自啓動，當服務器重啓時，需要人工去啓動

(5)、測試heartbeat

正常狀態

[root@master1 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

[root@master2 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1

說明：master1節點擁有vip地址，master2節點沒有


模擬主節點宕機後的狀態

[root@master1 ~]# /etc/init.d/heartbeat stop

[root@master2 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

說明：master1宕機後，vip地址漂移到master2節點上，master2成爲主節點


模擬主節點故障恢復後的狀態

[root@master1 ~]# /etc/init.d/heartbeat start

[root@master1 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

說明：master1搶佔vip資源

3、DRBD

安裝部署

(1)、新添加硬盤

[root@master1 ~]# fdisk /dev/sdb               #sdb磁盤分兩個分區sdb1和sdb2

[root@master1 ~]# partprobe

[root@master1 ~]# mkfs.ext3 /dev/sdb1          #sdb2分區爲meta data分區，不需要格式化操作

[root@master1 ~]# tune2fs -c -1 /dev/sdb1      #設置最大掛載數爲-1

(2)、安裝DRBD

[root@master1 ~]# yum install kmod-drbd83 drbd83 -y

[root@master1 ~]# modprobe drbd               

#注意：不要設置echo 'modprobe drbd' >>/etc/rc.loca開機自動加載drbd模塊，如果drbd服務是開機自啓動的，會先啓動drbd服務在加載drbd的順序，導致drbd啓動不了出現的問題

(3)、配置DRBD

主備節點兩端配置文件完全一致

[root@master1 ~]# cat /etc/drbd.conf

global {

# minor-count 64;

# dialog-refresh 5; # 5 seconds

# disable-ip-verification;

usage-count no;

}

common {

protocol C;

disk {

on-io-error   detach;

#size 454G;

no-disk-flushes;

no-md-flushes;

}

net {

sndbuf-size 512k;

# timeout       60;    #  6 seconds  (unit = 0.1 seconds)

# connect-int   10;    # 10 seconds  (unit = 1 second)

# ping-int      10;    # 10 seconds  (unit = 1 second)

# ping-timeout   5;    # 500 ms (unit = 0.1 seconds)

max-buffers     8000;

unplug-watermark   1024;

max-epoch-size  8000;

# ko-count 4;

# allow-two-primaries;

cram-hmac-alg "sha1";

shared-secret "hdhwXes23sYEhart8t";

after-sb-0pri disconnect;

after-sb-1pri disconnect;

after-sb-2pri disconnect;

rr-conflict disconnect;

# data-integrity-alg "md5";

# no-tcp-cork;

}

syncer {

rate 120M;

al-extents 517;

}

}

resource data {

on master1 {

device     /dev/drbd1;

disk       /dev/sdb1;

address    192.168.4.2:7788;

meta-disk  /dev/sdb2 [0];

}

on master2 {

device     /dev/drbd1;

disk       /dev/sdb1;

address    192.168.4.3:7788;

meta-disk  /dev/sdb2 [0];

}

}

(4)、初始化meta分區

[root@master1 ~]# drbdadm create-md data

Writing meta data...

initializing activity log

NOT initialized bitmap

New drbd meta data block successfully created.

(5)、初始化設備同步(覆蓋備節點，保持數據一致)

[root@master1 ~]# drbdadm -- --overwrite-data-of-peer primary data

(6)、啓動drbd

[root@master1 ~]# drbdadm up all

[root@master1 ~]# chkconfig drbd off

(7)、掛載drbd分區到data數據目錄

[root@master1 ~]# drbdadm primary all

[root@master1 ~]# mount /dev/drbd1 /data          #說明：/data目錄爲數據庫的數據目錄

(8)、測試DRBD

正常狀態

[root@master1 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

ns:497984 nr:0 dw:1 dr:498116 al:1 bm:31 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

[root@master2 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

ns:0 nr:497984 dw:497984 dr:0 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

#說明：master1爲主節點，master2爲備節點

模擬master1宕機

[root@master1 ~]# umount /dev/drbd1

[root@master1 ~]# drbdadm down all

[root@master2 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----

ns:0 nr:497985 dw:497985 dr:0 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

[root@master2 ~]# drbdadm primary all

[root@master2 ~]# mount /dev/drbd1 /data

[root@master2 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 19G 5.1G 13G 29% /

/dev/sda1 190M 18M 163M 10% /boot

tmpfs 60M 0 60M 0% /dev/shm

/dev/drbd1 471M 11M 437M 3% /data

#說明：master1宕機後，master2可以升級爲主節點，可掛載drbd分區繼續使用

4、MySQL

安裝部署

注意：三臺數據庫都安裝mysql服務，master2只安裝到makeinstall即可，mysqld服務不要設置爲開機自啓動

(1)、解決perl編譯問題

echo 'export LC_ALL=C'>> /etc/profile

source /etc/profile

(2)、安裝CAMKE

cd /home/xu/tools

wget http://www.cmake.org/files/v2.8/cmake-2.8.4.tar.gz

tar zxf cmake-2.8.4.tar.gz

cd cmake-2.8.4

./configure

make & make install

(3)、創建用戶

groupadd mysql

useradd -g mysql mysql

(4)、編譯安裝mysql

wget http://mysql.ntu.edu.tw/Downloads/MySQL-5.5/mysql-5.5.27.tar.gz

tar zxf mysql-5.5.27.tar.gz

cd mysql-5.5.27

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \

-DMYSQL_UNIX_ADDR=/tmp/mysql.sock \

-DDEFAULT_CHARSET=utf8 \

-DDEFAULT_COLLATION=utf8_general_ci \

-DWITH_EXTRA_CHARSETS=complex \

-DWITH_READLINE=1 \

-DENABLED_LOCAL_INFILE=1

make -j 4

make install

(5)、設置mysql環境變量

[root@master1 ~]# echo 'PATH=$PATH:/usr/local/mysql/bin' >>/etc/profile

[root@master1 ~]# source /etc/profile

(6)、初始化數據庫

[root@master1 ~]# mount /dev/drbd1 /data                 #說明：數據庫存放數據的目錄是drbd分區

[root@master1 ~]# cd /usr/local/mysql/

[root@master1 ~]# ./scripts/mysql_install_db --datadir=/data/ --user=mysql

(7)、啓動數據庫

[root@master1 ~]# vim /etc/init.d/mysqld

datadir=/data                                            #說明：修改mysql啓動腳本，指定數據庫的目錄爲/data

[root@master1 ~]# /etc/init.d/mysqld start

[root@master1 ~]# chkconfig mysqld off

(8)、測試數據庫

[root@master1 ~]# mysql -uroot -e "show databases;"

+--------------------+

| Database |

+--------------------+

| information_schema |

| mysql |

| performance_schema |

+--------------------+

5、故障切換測試

(1)、架構正常狀態

master1主節點正常狀態

[root@master1 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

[root@master1 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

ns:39558 nr:12 dw:39570 dr:151 al:16 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

[root@master1 ~]# mysql -uroot -e "create database coral;"

[root@master1 ~]# mysql -uroot -e "show databases like 'coral';"

+------------------+

| Database (coral) |

+------------------+

| coral |

+------------------+

#說明：master1爲主節點，擁有VIP地址，爲drbd的主節點


master2備節點正常狀態

[root@master2 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1

[root@master2 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

ns:0 nr:48 dw:48 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

#說明：master2備節點沒有VIP地址，爲drbd備節點


(2)、模擬master1宕機故障狀態

[root@master1 ~]# /etc/init.d/heartbeat stop <==模擬master1故障宕機

[root@master2 ~]# tailf /var/log/ha-log <==查看備節點接管日誌

heartbeat[13209]: 2013/01/23_04:09:36 info: Received shutdown notice from 'master1'.

heartbeat[13209]: 2013/01/23_04:09:36 info: Resources being acquired from master1.

heartbeat[15293]: 2013/01/23_04:09:36 info: acquire local HA resources (standby).

heartbeat[15294]: 2013/01/23_04:09:37 info: No local resources [/usr/share/heartbeat/ResourceManager listkeys master2] to acquire.

heartbeat[15293]: 2013/01/23_04:09:37 info: local HA resource acquisition completed (standby).

heartbeat[13209]: 2013/01/23_04:09:37 info: Standby resource acquisition done [foreign].

harc[15319]: 2013/01/23_04:09:37 info: Running /etc/ha.d/rc.d/status status

mach_down[15335]: 2013/01/23_04:09:37 info: Taking over resource group IPaddr::192.168.4.1/16/eth1

ResourceManager[15361]: 2013/01/23_04:09:37 info: Acquiring resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld

IPaddr[15388]: 2013/01/23_04:09:37 INFO: Resource is stopped

ResourceManager[15361]: 2013/01/23_04:09:37 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 start

IPaddr[15486]: 2013/01/23_04:09:38 INFO: Using calculated netmask for 192.168.4.1: 255.255.0.0

IPaddr[15486]: 2013/01/23_04:09:38 INFO: eval ifconfig eth1:0 192.168.4.1 netmask 255.255.0.0 broadcast 192.168.255.255

IPaddr[15457]: 2013/01/23_04:09:38 INFO: Success

ResourceManager[15361]: 2013/01/23_04:09:38 info: Running /etc/ha.d/resource.d/drbddisk data start

Filesystem[15636]: 2013/01/23_04:09:39 INFO: Resource is stopped

ResourceManager[15361]: 2013/01/23_04:09:39 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 start

Filesystem[15717]: 2013/01/23_04:09:39 INFO: Running start for /dev/drbd1 on /data

Filesystem[15706]: 2013/01/23_04:09:39 INFO: Success

ResourceManager[15361]: 2013/01/23_04:09:40 info: Running /etc/init.d/mysqld start

mach_down[15335]: 2013/01/23_04:09:44 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired

mach_down[15335]: 2013/01/23_04:09:44 info: mach_down takeover complete for node master1.

heartbeat[13209]: 2013/01/23_04:09:44 info: mach_down takeover complete.

heartbeat[13209]: 2013/01/23_04:10:09 WARN: node master1: is dead

heartbeat[13209]: 2013/01/23_04:10:09 info: Dead node master1 gave up resources.

heartbeat[13209]: 2013/01/23_04:10:09 info: Link master1:eth2 dead.

#說明：當備節點無法檢測到主節點的心跳時，自動接管資源，啓動VIP地址、drbd服務，自動掛載drbd，啓動mysqld服務，備節點接管後，數據依然存在，檢測啓動的服務如下：

[root@master2 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

[root@master2 ~]# cat /proc/drbd

version: 8.3.13 (api:88/proto:86-96)

GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

ns:3 nr:95 dw:98 dr:10 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

[root@master2 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 19G 4.7G 14G 26% /

/dev/sda1 190M 18M 163M 10% /boot

tmpfs 60M 0 60M 0% /dev/shm

/dev/drbd1 471M 40M 408M 9% /data

[root@master2 ~]# mysql -uroot -e "show databases like 'coral';"

+------------------+

| Database (coral) |

+------------------+

| coral |

+------------------+


(3)、模擬master1宕機恢復狀態

啓動的順序是：先啓動VIP--啓動drbd資源--掛載drbd分區--啓動mysqld服務，日誌如下：

[root@master1 ~]# /etc/init.d/heartbeat start

[root@master1 ~]# tailf /var/log/ha-log

heartbeat[27970]: 2013/01/09_17:34:14 info: Version 2 support: no

heartbeat[27970]: 2013/01/09_17:34:14 WARN: Logging daemon is disabled --enabling logging daemon is recommended

heartbeat[27970]: 2013/01/09_17:34:14 info: **************************

heartbeat[27970]: 2013/01/09_17:34:14 info: Configuration validated. Starting heartbeat 2.1.3

heartbeat[27971]: 2013/01/09_17:34:14 info: heartbeat: version 2.1.3

heartbeat[27971]: 2013/01/09_17:34:14 info: Heartbeat generation: 1351554533

heartbeat[27971]: 2013/01/09_17:34:14 info: glib: UDP multicast heartbeat started for group 225.0.0.7 port 694 interface eth2 (ttl=1 loop=0)

heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_TriggerHandler: Added signal manual handler

heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_TriggerHandler: Added signal manual handler

heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_SignalHandler: Added signal handler for signal 17

heartbeat[27971]: 2013/01/09_17:34:14 info: Local status now set to: 'up'

heartbeat[27971]: 2013/01/09_17:34:16 info: Link master2:eth2 up.

heartbeat[27971]: 2013/01/09_17:34:16 info: Status update for node master2: status active

harc[27978]: 2013/01/09_17:34:16 info: Running /etc/ha.d/rc.d/status status

heartbeat[27971]: 2013/01/09_17:34:17 info: Comm_now_up(): updating status to active

heartbeat[27971]: 2013/01/09_17:34:17 info: Local status now set to: 'active'

heartbeat[27971]: 2013/01/09_17:34:17 info: remote resource transition completed.

heartbeat[27971]: 2013/01/09_17:34:17 info: remote resource transition completed.

heartbeat[27971]: 2013/01/09_17:34:17 info: Local Resource acquisition completed. (none)

heartbeat[27971]: 2013/01/09_17:34:18 info: master2 wants to go standby [foreign]

heartbeat[27971]: 2013/01/09_17:34:20 info: standby: acquire [foreign] resources from master2

heartbeat[27997]: 2013/01/09_17:34:20 info: acquire local HA resources (standby).

ResourceManager[28010]: 2013/01/09_17:34:20 info: Acquiring resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld

IPaddr[28037]: 2013/01/09_17:34:21 INFO: Resource is stopped

ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 start

IPaddr[28135]: 2013/01/09_17:34:21 INFO: Using calculated netmask for 192.168.4.1: 255.255.0.0

IPaddr[28135]: 2013/01/09_17:34:21 INFO: eval ifconfig eth1:0 192.168.4.1 netmask 255.255.0.0 broadcast 192.168.255.255

IPaddr[28106]: 2013/01/09_17:34:21 INFO: Success

ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/drbddisk data start

Filesystem[28286]: 2013/01/09_17:34:21 INFO: Resource is stopped

ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 start

Filesystem[28367]: 2013/01/09_17:34:21 INFO: Running start for /dev/drbd1 on /data

Filesystem[28356]: 2013/01/09_17:34:21 INFO: Success

ResourceManager[28010]: 2013/01/09_17:34:22 info: Running /etc/init.d/mysqld start

heartbeat[27997]: 2013/01/09_17:34:25 info: local HA resource acquisition completed (standby).

heartbeat[27971]: 2013/01/09_17:34:25 info: Standby resource acquisition done [foreign].

heartbeat[27971]: 2013/01/09_17:34:25 info: Initial resource acquisition complete (auto_failback)

heartbeat[27971]: 2013/01/09_17:34:25 info: remote resource transition completed.


備節點釋放資源順序：停止mysqld服務--卸載drbd1分區--設置drbd爲備節點--關閉VIP地址，日誌如下：

[root@master2 ~]# tailf /var/log/ha-log

heartbeat[13209]: 2013/01/23_04:26:53 info: Heartbeat restart on node master1

heartbeat[13209]: 2013/01/23_04:26:53 info: Link master1:eth2 up.

heartbeat[13209]: 2013/01/23_04:26:53 info: Status update for node master1: status init

heartbeat[13209]: 2013/01/23_04:26:53 info: Status update for node master1: status up

harc[16151]: 2013/01/23_04:26:53 info: Running /etc/ha.d/rc.d/status status

harc[16167]: 2013/01/23_04:26:53 info: Running /etc/ha.d/rc.d/status status

heartbeat[13209]: 2013/01/23_04:26:53 info: all clients are now paused

heartbeat[13209]: 2013/01/23_04:26:55 info: Status update for node master1: status active

harc[16183]: 2013/01/23_04:26:55 info: Running /etc/ha.d/rc.d/status status

heartbeat[13209]: 2013/01/23_04:26:55 info: all clients are now resumed

heartbeat[13209]: 2013/01/23_04:26:55 info: remote resource transition completed.

heartbeat[13209]: 2013/01/23_04:26:55 info: master2 wants to go standby [foreign]

heartbeat[13209]: 2013/01/23_04:26:55 info: standby: master1 can take our foreign resources

heartbeat[16199]: 2013/01/23_04:26:55 info: give up foreign HA resources (standby).

ResourceManager[16212]: 2013/01/23_04:26:55 info: Releasing resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld

ResourceManager[16212]: 2013/01/23_04:26:55 info: Running /etc/init.d/mysqld stop

ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 stop

Filesystem[16297]: 2013/01/23_04:26:57 INFO: Running stop for /dev/drbd1 on /data

Filesystem[16297]: 2013/01/23_04:26:57 INFO: Trying to unmount /data

Filesystem[16297]: 2013/01/23_04:26:57 INFO: unmounted /data successfully

Filesystem[16286]: 2013/01/23_04:26:57 INFO: Success

ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/drbddisk data stop

ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 stop

IPaddr[16445]: 2013/01/23_04:26:58 INFO: ifconfig eth1:0 down

IPaddr[16416]: 2013/01/23_04:26:58 INFO: Success

heartbeat[16199]: 2013/01/23_04:26:58 info: foreign HA resource release completed (standby).

heartbeat[13209]: 2013/01/23_04:26:58 info: Local standby process completed [foreign].

heartbeat[13209]: 2013/01/23_04:27:02 WARN: 1 lost packet(s) for [master1] [15:17]

heartbeat[13209]: 2013/01/23_04:27:02 info: remote resource transition completed.

heartbeat[13209]: 2013/01/23_04:27:02 info: No pkts missing from master1!

heartbeat[13209]: 2013/01/23_04:27:02 info: Other node completed standby takeover of foreign resources.

6、從庫同VIP同步

(1)、master配置

1)、設置server-id值並開啓Binlog參數

[root@master1 ~]# vim /etc/my.cnf

log-bin=/usr/local/mysql/mysql-bin

server-id = 3

[root@master1 ~]# /etc/init.d/mysqld restart

#注意：只有master1有重啓操作，master2無需重啓操作，因爲備節點的mysql是未啓動狀態，備節點只有heartbeat才能啓動mysql

2)、授權並建立同步賬戶rep

[root@master1 ~]# mysql -uroot -p

mysql> GRANT REPLICATION SLAVE ON *.* TO 'rep'@'192.168.4.%' IDENTIFIED BY 'rep';


(2)、slave配置

1)、設置server-id值並關閉binlog設置

[root@slave1 ~]# vim /etc/my.cnf

#log-bin=mysql-bin

server-id = 4

[root@slave1 ~]# /etc/init.d/mysqld restart          

#說明：從庫無需開啓binlog日誌功能，除非有需求做級聯複製架構或對mysql增量備份操作纔開啓

2)、配置同步參數

[root@Slave ~]# mysql -uroot

CHANGE MASTER TO

MASTER_HOST='192.168.4.1',

MASTER_PORT=3306,

MASTER_USER='rep',

MASTER_PASSWORD='rep',

MASTER_LOG_FILE='mysql-bin.000001',

MASTER_LOG_POS=0;

3)、檢查是否主從同步

[root@Slave ~]# mysql -uroot

mysql> show slave status\G

...

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

...


(3)、模擬高可用宕機切換是否影響從庫同步

1)、主從正常狀態

[root@master1 ~]# mysql -uroot

mysql> create database coral1;

Query OK, 1 row affected (0.02 sec)

[root@slave1 ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running"

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

[root@slave1 ~]# mysql -uroot -e "show databases like 'coral%';"

+-------------------+

| Database (coral%) |

+-------------------+

| coral1 |

+-------------------+

2)、模擬高可用主節點宕機

[root@master1 ~]# /etc/init.d/heartbeat stop             #說明：模擬主節點宕機

[root@master2 ~]# ip addr|grep eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1

inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0

[root@master2 ~]# mysql -uroot

mysql> create database coral2;

Query OK, 1 row affected (0.08 sec)                      #說明：VIP地址已經漂移到master2上面

[root@slave1 ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running"

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

[root@slave1 ~]# mysql -uroot -e "show databases like 'coral%'"

+-------------------+

| Database (coral%) |

+-------------------+

| coral1 |

| coral2 |

+-------------------+

#注意：高可用主備節點切換過程中，會有一段時間從庫才能連接上，大於在60秒內

#說明：此時主從同步是正常的

3)、模擬高可用主節點宕機恢復

[root@master1 ~]# /etc/init.d/heartbeat start

[root@master1 ~]# mysql -uroot

mysql> create database coral3;

[root@slave1 ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running"

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

[root@slave1 ~]# mysql -uroot -e "show databases like 'coral%'"

+-------------------+

| Database (coral%) |

+-------------------+

| coral1 |

| coral2 |

| coral3 |

+-------------------+

#說明：高可用主節點故障恢復後也不影響主從庫的同步

7、高可用腦裂問題及解決方案

(1)、導致裂腦發生的原因

1、高可用服務器之間心跳鏈路故障，導致無法相互檢查心跳

2、高可用服務器上開啓了防火牆，阻擋了心跳檢測

3、高可用服務器上網卡地址等信息配置不正常，導致發送心跳失敗

4、其他服務配置不當等原因，如心跳方式不同，心跳廣播衝突，軟件BUG等


(2)、防止裂腦一些方案

1、加冗餘線路

2、檢測到裂腦時，強行關閉心跳檢測(遠程關閉主節點，控制電源的電路fence)

3、做好腦裂的監控報警

4、報警後，備節點在接管時設置比較長的時間去接管，給運維人員足夠的時間去處理(人爲處理)

5、啓動磁盤鎖，正在服務的一方鎖住磁盤，裂腦發生時，讓對方完全搶不走"共享磁盤資源"


磁盤鎖存在的問題：

使用鎖磁盤會有死鎖的問題，如果佔用共享磁盤的一方不主動"解鎖"另一方就永遠得不到共享磁盤，假如服務器節點突然死機或崩潰，就不可能執行解鎖命令，備節點也就無法接管資源和服務了，有人在HA中設計了智能鎖，正在提供服務的一方只在發現心跳全部斷開時纔會啓用磁盤鎖，平時就不上鎖

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

MySQL基礎

莫渺1996 2019-02-24 13:34:23

6:多源複製的實現

shixiazhoujun 2019-02-24 13:04:18

7:基於語句和行復制的優缺點

shixiazhoujun 2019-02-24 13:04:18

詳解MySQL執行原理、邏輯分層、更改數據庫處理引擎

laozhang 2019-02-24 12:43:05

MySQL中InnoDB和MyISAM的差別

jackieban 2019-02-23 13:57:24

SQL mapper文件的審計

橡皮高 2019-02-23 13:51:10

數據庫管理工具 Navicat使用教程：SQL Server中的磁盤加密

Aote222 2019-02-23 13:34:13

Redhat環境MySQL+Apache+PHP環境配置

znlingdong 2019-02-23 13:29:59

MySQL 常用

loip125 2019-02-23 13:29:07

MySQL字符集選擇

兩袖春風 2019-02-23 13:28:22

MySQL數據類型之Pk10平臺架設字符串(string)

wojiuyong 2019-02-23 13:25:22

MySQL 的並集、交集和差集 交集和差集類似oracle中minus的用法

zhengfj 2019-02-23 13:13:04

在同一臺機器上運行多個MySQL服務器

zhengfj 2019-02-23 13:13:04

14.5 InnoDB Locking and Transaction Model

shixiazhoujun 2019-02-23 13:04:53

14.4.11 Undo Tablespace

shixiazhoujun 2019-02-23 13:04:53