Corosync簡介：
Coreosync在傳遞信息的時候可以通過一個簡單的配置文件來定義信息傳遞的方式和協議等。它是一個新興的軟件，2008年推出，但其實它並不是一個真正意義上的新軟件，在2002年的時候有一個項目Openais , 它由於過大，分裂爲兩個子項目，其中可以實現HA心跳信息傳輸的功能就是Corosync ,它的代碼60%左右來源於Openais. Corosync可以提供一個完整的HA功能，但是要實現更多，更復雜的功能，那就需要使用Openais了。Corosync是未來的發展方向。在以後的新項目裏，一般採用Corosync，而hb_gui可以提供很好的HA管理功能，可以實現圖形化的管理。另外相關的圖形化有RHCS的套件luci+ricci.
案例應用企業需求：
某公司需要通過HA實現服務器的高可用性，即通過corosync+drbd+
pacemaker實現mysql服務器的高可用集羣。
拓撲圖：
1.1配置node1、node2的地址：
重啓服務
[root@node1 ~]# service network restart
[root@node2 ~]# service network restart
1.2調整系統時間:
1.3在node1和node2上配置hosts文件和ssh密鑰信息,有利於以後在一個節點對另一節點直接操作:
1.3.1 在node1上修改/etc/hosts文件
[root@node1 ~]# echo "192.168.145.100 node1.a.com node1" >>/etc/hosts
[root@node1 ~]# echo "192.168.145.99 node2.a.com node2" >>/etc/hosts
1.3.1 在node1上修改/etc/hosts文件
[root@node2 ~]# echo "192.168.145.100 node1.a.com node1" >>/etc/hosts
[root@node2 ~]# echo "192.168.145.99 node2.a.com node2" >>/etc/hosts
1.3.3 在node1上配置ssh密鑰信息
[root@node1 ~]# ssh-keygen -t rsa #一直輸入空格
[root@node1 ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
1.3.4 在node2上配置ssh密鑰信息
[root@node2 ~]# ssh-keygen -t rsa #一直輸入空格
[root@node2 ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
[root@node2 ~]# ssh node1 'ifconfig' #將看到node2上的信息
The authenticity of host 'node1 (192.168.145.100)' can't be established.
RSA key fingerprint is d4:f1:06:3b:a0:81:fd:85:65:20:9e:a1:ee:46:a6:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1' (RSA) to the list of known hosts.
eth0 Link encap:Ethernet HWaddr 00:0C:29:AE:83:D1
inet addr:192.168.145.100 Bcast:192.168.145.255 Mask:255.255.255.0
1.4拷貝源代碼到根目錄下（node1、node2都得拷貝）：
[root@node1 ~]# ll
總計 3792
-rw------- 1 root root 1198 07-15 22:34 anaconda-ks.cfg
-rw-r--r-- 1 root root 271360 10-17 15:55 cluster-glue-1.0.6-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 133254 10-17 15:55 cluster-glue-libs-1.0.6-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 170052 10-17 15:55 corosync-1.2.7-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 158502 10-17 15:55 corosynclib-1.2.7-1.1.el5.i386.rpm
drwxr-xr-x 2 root root 4096 07-15 14:39 Desktop
-rw-r--r-- 1 root root 221868 10-17 12:41 drbd83-8.3.8-1.el5.centos.i386.rpm
-rw-r--r-- 1 root root 165591 10-17 15:55 heartbeat-3.0.3-2.3.el5.i386.rpm
-rw-r--r-- 1 root root 289600 10-17 15:55 heartbeat-libs-3.0.3-2.3.el5.i386.rpm
-rw-r--r-- 1 root root 35236 07-15 22:33 install.log
-rw-r--r-- 1 root root 3985 07-15 22:33 install.log.syslog
-rw-r--r-- 1 root root 125974 10-17 12:41 kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
-rw-r--r-- 1 root root 60458 10-17 15:55 libesmtp-1.0.4-5.el5.i386.rpm
-rw-r--r-- 1 root root 207085 10-17 15:55 openais-1.1.3-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 94614 10-17 15:55 openaislib-1.1.3-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 796813 10-17 15:55 pacemaker-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 207925 10-17 15:55 pacemaker-cts-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 332026 10-17 15:55 pacemaker-libs-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 32818 10-17 15:56 perl-TimeDate-1.16-5.el5.noarch.rpm
-rw-r--r-- 1 root root 388632 10-17 15:56 resource-agents-1.0.4-1.1.el5.i386.rpm
[root@node2 ~]# ll
總計 3728
-rw------- 1 root root 1431 08-11 00:13 anaconda-ks.cfg
-rw-r--r-- 1 root root 271360 10-17 18:03 cluster-glue-1.0.6-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 133254 10-17 18:03 cluster-glue-libs-1.0.6-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 170052 10-17 18:03 corosync-1.2.7-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 158502 10-17 18:03 corosynclib-1.2.7-1.1.el5.i386.rpm
drwxr-xr-x 2 root root 4096 08-10 16:26 Desktop
-rw-r--r-- 1 root root 221868 10-17 16:37 drbd83-8.3.8-1.el5.centos.i386.rpm
-rw-r--r-- 1 root root 165591 10-17 18:03 heartbeat-3.0.3-2.3.el5.i386.rpm
-rw-r--r-- 1 root root 289600 10-17 18:03 heartbeat-libs-3.0.3-2.3.el5.i386.rpm
-rw-r--r-- 1 root root 35768 08-11 00:13 install.log
-rw-r--r-- 1 root root 4705 08-11 00:12 install.log.syslog
-rw-r--r-- 1 root root 125974 10-17 16:37 kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
-rw-r--r-- 1 root root 60458 10-17 18:03 libesmtp-1.0.4-5.el5.i386.rpm
-rw-r--r-- 1 root root 207085 10-17 18:03 openais-1.1.3-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 94614 10-17 18:03 openaislib-1.1.3-1.6.el5.i386.rpm
-rw-r--r-- 1 root root 796813 10-17 18:04 pacemaker-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 207925 10-17 18:04 pacemaker-cts-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 332026 10-17 18:04 pacemaker-libs-1.1.5-1.1.el5.i386.rpm
-rw-r--r-- 1 root root 32818 10-17 18:04 perl-TimeDate-1.16-5.el5.noarch.rpm
-rw-r--r-- 1 root root 388632 10-17 18:04 resource-agents-1.0.4-1.1.el5.i386.rpm
1.5 配置本地yum數據庫：
1.6安裝源代碼：
[root@node1 ~]# yum localinstall -y *.rpm –-nogpgcheck
[root@node2 ~]# yum localinstall -y *.rpm –nogpgcheck
1.7在node1上新增磁盤：
[root@node1 ha]# cd
[root@node1 drbd.d]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1288 10241437+ 83 Linux
/dev/sda3 1289 1353 522112+ 82 Linux swap / Solaris
/dev/sda4 1354 1476 987997+ 5 Extended
/dev/sda5 1354 1476 987966 83 Linux
[root@node1 drbd.d]# partprobe /dev/sda
[root@node1 drbd.d]# cat /proc/partitions
major minor #blocks name
8 0 20971520 sda
8 1 104391 sda1
8 2 10241437 sda2
8 3 522112 sda3
8 4 0 sda4
8 5 987966 sda5
147 0 987896 drbd0
1.8 在node2上新增磁盤
[root@node2 ha]# cd
[root@node2 drbd.d]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2610 20860402+ 8e Linux LVM
Disk /dev/sdb: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 123 987966 5 Extended
/dev/sdb2 124 246 987997+ 83 Linux
/dev/sdb3 247 369 987997+ 83 Linux
/dev/sdb4 370 492 987997+ 83 Linux
[root@node2 drbd.d]# partprobe /dev/sdb
[root@node2 drbd.d]# cat /proc/partitions
major minor #blocks name
8 0 20971520 sda
8 1 104391 sda1
8 2 20860402 sda2
8 16 20971520 sdb
8 17 0 sdb1
8 18 987997 sdb2
8 19 987997 sdb3
8 20 987997 sdb4
253 0 19791872 dm-0
253 1 1048576 dm-1
二、DRBD安裝配置步驟
在node1和node2做以下操作：
我下載的軟件包是：（我放在/root/下了）
drbd83-8.3.8-1.el5.centos.i386.rpm
kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
2.1、安裝DRBD 套件
[root@node1 ~]# cd ha/
[root@node1 ha]# ls
[root@node1 ha]# yum localinstall -y drbd83-8.3.8-1.el5.centos.i386.rpm kmod-drbd83-8.3.8-1.el5.centos.i686.rpm –nogpgcheck
...
Installed:
drbd83.i386 0:8.3.8-1.el5.centos kmod-drbd83.i686 0:8.3.8-1.el5.centos
Complete!
[root@node1 ha]# cd
[root@node2 ~]# cd ha/
[root@node2 ha]# ls
[root@node2 ha]# yum localinstall -y drbd83-8.3.8-1.el5.centos.i386.rpm kmod-drbd83-8.3.8-1.el5.centos.i686.rpm –nogpgcheck
...
Installed:
drbd83.i386 0:8.3.8-1.el5.centos kmod-drbd83.i686 0:8.3.8-1.el5.centos
Complete!
[root@node2 ha]# cd
2.2、加載DRBD 模塊
[root@node1 ~]# modprobe drbd
[root@node1 ~]# lsmod |grep drbd
drbd 228528 0
[root@node1 ~]#
[root@node1 ~]# ssh node2 'modprobe drbd'
[root@node1 ~]# ssh node2 'lsmod |grep drbd'
drbd 228528 0
[root@node1 ~]#
2.3、修改配置文件
drbd.conf配置文件DRBD運行時,會讀取一個配置文件/etc/drbd.conf.這個文件裏描述了DRBD設備與硬盤分區的映射關係
2.3.1 在node1上以下配置
[root@node1 ~]# cp /usr/share/doc/drbd83-8.3.8/drbd.conf /etc/
cp: overwrite `/etc/drbd.conf'? y
[root@node1 ~]# cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
[root@node1 ~]# cd /etc/drbd.d/
[root@node1 drbd.d]# ll
total 4
-rwxr-xr-x 1 root root 1418 Jun 4 2010 global_common.conf
[root@node1 drbd.d]# cp global_common.conf global_common.conf.bak
#修改全局配置文件（說明略）
[root@node1 drbd.d]# vim global_common.conf
global {
usage-count yes;
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
}
startup {
wfc-timeout 120;
degr-wfc-timeout 100;
}
disk {
on-io-error detach;
}
net {
cram-hmac-alg "sha1";
shared-secret "mydrbd123";
}
syncer {
rate 100M;
}
}
#修改資源配置文件（說明略）
[root@node1 drbd.d]# vim /etc/drbd.d/mysql.res
resource mysql {
on node1.a.com {
device /dev/drbd0;
disk /dev/sdb4;
address 192.168.145.100:7789;
meta-disk internal;
}
on node2.a.com {
device /dev/drbd0;
disk /dev/sdb4;
address 192.168.145.99:7789;
meta-disk internal;
}
}
2.3.2 複製配置到node2上：
[root@node1 drbd.d]# scp /etc/drbd.conf node2:/etc/
drbd.conf 100% 133 0.1KB/s 00:00
[root@node1 drbd.d]# scp /etc/drbd.d/* node2:/etc/drbd.d/
global_common.conf 100% 682 0.7KB/s 00:00
global_common.conf.bak 100% 1418 1.4KB/s 00:00
mysql.res 100% 468 0.5KB/s 00:00
2.4、 檢測配置文件, 創建nfs 的資源
//分別在node1和node2上初始化定義的mysql的資源
//檢測配置文件（兩次執行如下命令)
在node1上：
[root@node1 drbd.d]# drbdadm adjust mysql
--== Thank you for participating in the global usage survey ==--
The server's response is:
0: Failure: (119) No valid meta-data signature found.
==> Use 'drbdadm create-md res' to initialize meta-data area. <==
Command 'drbdsetup 0 disk /dev/sda4 /dev/sda4 internal --set-defaults --create-device --on-io-error=detach' terminated with exit code 10
[root@node1 drbd.d]# drbdadm adjust mysql
drbdsetup 0 show:5: delay-probe-volume 0k => 0k out of range [4..1048576]k.
[root@node1 drbd.d]# drbdadm create-md mysql
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
[root@node1 drbd.d]# ll /dev/drbd0
brw-r----- 1 root disk 147, 0 Feb 7 20:44 /dev/drbd0
在node2上：
[root@node2 ~]# drbdadm create-md mysql
--== Thank you for participating in the global usage survey ==--
The server's response is:
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
[root@node2 ~]# ll /dev/drbd0
brw-rw---- 1 root root 147, 0 Feb 7 20:47 /dev/drbd0
2.5 啓動DRBD服務，查看DRBD狀態
[root@node1 drbd.d]# service drbd start
Starting DRBD resources: [
mysql
Found valid meta data in the expected location, 1451703808 bytes into /dev/sda4.
d(mysql) s(mysql) n(mysql) ]......
[root@node2 ~]# service drbd start
Starting DRBD resources: [
mysql
Found valid meta data in the expected location, 1451703808 bytes into /dev/sda4.
d(mysql) s(mysql) n(mysql) ].
[root@node1 drbd.d]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@node1 drbd.d]# drbd-overview
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C r----
[root@node2 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C r----
2.6 設置DRBD的主節點：
從上面的信息中可以看出此時兩個節點均處於Secondary狀態。於是，我們接下來需要將其中一個節點設置爲Primary，這裏將node1設置爲主節點，故要在node1上執行如下命令：可以看到文件同步過程。
[root@node1 drbd.d]# drbdadm -- --overwrite-data-of-peer primary mysql
[root@node1 drbd.d]# drbd-overview
0:mysql SyncSource Primary/Secondary UpToDate/Inconsistent C r----
[===>................] sync'ed: 23.6% (761048/987928)K delay_probe: 20
[root@node1 drbd.d]# drbd-overview
0:mysql SyncSource Primary/Secondary UpToDate/Inconsistent C r----
[========>...........] sync'ed: 47.6% (522296/987928)K delay_probe: 42
[root@node1 drbd.d]#
[root@node1 drbd.d]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 drbd.d]#
[root@node1 drbd.d]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:987928 nr:0 dw:0 dr:987928 al:0 bm:61 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
[root@node1 drbd.d]#
[root@node2 ~]# drbd-overview
0:mysql SyncTarget Secondary/Primary Inconsistent/UpToDate C r----
[=====>..............] sync'ed: 32.7% (670680/987928)K queue_delay: 0.1 ms
[root@node2 ~]# drbd-overview
0:mysql SyncTarget Secondary/Primary Inconsistent/UpToDate C r----
[===========>........] sync'ed: 61.6% (384536/987928)K queue_delay: 11.6 ms
[root@node2 ~]#
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Primary UpToDate/UpToDate C r----
[root@node2 ~]#
[root@node2 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:987928 dw:987928 dr:0 al:0 bm:61 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
注：Primary/Secondary 說明當前節點爲主節點；Secondary/Primary 說明當前節點爲從節點。使用：watch -n 1 'cat /proc/drbd'查看同步過程！
2.7 創建文件系統（只可以在primary節點上進行，這裏在node1上創建）
[root@node1 drbd.d]# mkfs -t ext3 /dev/drbd0
[root@node1 drbd.d]# mkdir -pv /mnt/mysqldata
mkdir: created directory `/mnt/mysqldata'
[root@node1 drbd.d]# ssh node2 'mkdir -pv /mnt/mysqldata'
mkdir: created directory `/mnt/mysqldata'
[root@node1 drbd.d]# mount /dev/drbd0 /mnt/mysqldata/
[root@node1 drbd.d]#
[root@node1 drbd.d]# cd /mnt/mysqldata/
[root@node1 mysqldata]# ll
total 16
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 mysqldata]# echo "123" >f1
[root@node1 mysqldata]# touch f2
[root@node1 mysqldata]# ll
total 20
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 mysqldata]#
2.8將node2設置爲primary節點
[root@node1 mysqldata]# cd
[root@node1 ~]# umount /mnt/mysqldata/
[root@node1 ~]# drbdadm secondary mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
將node2設置爲primary節點
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm primary mysql
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node2 ~]# cd /mnt/mysqldata/
[root@node2 mysqldata]# ll
total 20
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
此處看到在node1上創建的文件在這裏都顯示了，
最後將node2的/mnt/mysqldata/卸載
[root@node2 mysqldata]# cd
[root@node2 ~]# umount /mnt/mysqldata/
至此DRBD配置成功！！！
三、mysql安裝與配置
3.1 將node1設爲主節點，node2設爲備份節點
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm secondary mysql
[root@node2 ~]# drbdadm role mysql
Secondary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm primary mysql
[root@node1 ~]# drbdadm role mysql
Primary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
3.2 在node1.a.com上安裝配置mysql
[root@node1 ~]# groupadd -r mysql
[root@node1 ~]# useradd -g mysql -r mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node1 ~]# mkdir -pv /mnt/mysqldata/data
mkdir: created directory `/mnt/mysqldata/data'
[root@node1 ~]# chown -R mysql.mysql /mnt/mysqldata/data/
[root@node1 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 2 mysql mysql 4096 Feb 7 21:48 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 ~]#cd ha/
[root@node1 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node1 ~]# cd /usr/local/
[root@node1 src]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node1 src]# ll
[root@node1 src]# cd mysql
[root@node1 mysql]# chown -R mysql:mysql .
[root@node1 mysql]# scripts/mysql_install_db --user=mysql --datadir=/mnt/mysqldata/data/
[root@node1 mysql]# chown -R root .
[root@node1 mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@node1 mysql]# vim /etc/my.cnf
39 thread_concurrency = 2
40 datadir = /mnt/mysqldata/data/ #指定mysql數據文件的存放位置(添加)
爲mysql提供sysv服務腳本，使其能使用service命令：
[root@node1 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@node1 mysql]# scp /etc/my.cnf node2:/etc/
my.cnf 100% 4696 4.6KB/s 00:00
[root@node1 mysql]# scp /etc/rc.d/init.d/mysqld node2:/etc/rc.d/init.d/
mysqld 100% 10KB 10.4KB/s 00:00
[root@node1 mysql]#
[root@node1 mysql]# chkconfig --add mysqld
[root@node1 mysql]# chkconfig mysqld off
[root@node1 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node1 mysql]# service mysqld start
Starting MySQL..... [ OK ]
[root@node1 mysql]# l1 /mnt/mysqldata/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.a.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node1 mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@node1 mysql]#
爲了使用mysql的安裝符合系統使用規範，並將其開發組件導出給系統使用，這裏還需要進行如下步驟：
輸出mysql的man手冊至man命令的查找路徑：添加如下行即可：
[root@node1 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
輸出mysql的頭文件至系統頭文件路徑/usr/include，這可以通過簡單的創建鏈接實現：
[root@node1 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node1 mysql]#
輸出mysql的庫文件給系統庫查找路徑：（文件只要是在/etc/ld.so.conf.d/下並且後綴是.conf就可以）而後讓系統重新載入系統庫
[root@node1 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node1 mysql]# ldconfig -v |grep mysql
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH環境變量，讓系統所有用戶可以直接使用mysql的相關命令:
[root@node1 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node1 mysql]# . /etc/profile
[root@node1 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸載drbd設備：
[root@node1 mysql]# umount /mnt/mysqldata/
3.3 將node2設爲主節點，node1設爲備份節點
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm secondary mysql
[root@node1 ~]# drbdadm role mysql
Secondary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm primary mysql
[root@node2 ~]# drbdadm role mysql
Primary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
3.4 在node2.a.com上安裝配置mysql
添加用戶和組：
[root@node2 ~]# groupadd -r mysql
[root@node2 ~]# useradd -g mysql -r mysql
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
掛載drbd設備：
[root@node2 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node2 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 5 mysql mysql 4096 Feb 7 22:00 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node2 ~]# ll /mnt/mysqldata/data/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.a.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node2 ~]#
安裝mysql：
[root@node2 ~]# cd ha/
[root@node2 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node2 ha]# cd /usr/local/
[root@node2 local]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node2 local]# cd mysql
一定不能對數據庫進行初始化，因爲我們在node1上已經初始化了：
[root@node2 mysql]# chown -R root:mysql .
mysql主配置文件和sysc服務腳本已經從node1複製過來了，不用在添加。
管理mysql服務：
[root@node2 mysql]# chkconfig --add mysqld
[root@node2 mysql]# chkconfig mysqld off
[root@node2 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node2 mysql]#
啓動mysql服務：
[root@node2 mysql]# service mysqld start
Starting MySQL....... [ OK ]
[root@node2 mysql]# ll /mnt/mysqldata/data/
total 28756
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:29 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 107 Feb 7 22:29 mysql-bin.000002
-rw-rw---- 1 mysql mysql 38 Feb 7 22:29 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.a.com.err
-rw-rw---- 1 mysql root 941 Feb 7 22:29 node2.a.com.err
-rw-rw---- 1 mysql mysql 5 Feb 7 22:29 node2.a.com.pid
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
測試之後關閉服務：
[root@node2 mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@node2 mysql]#
爲了使用mysql的安裝符合系統使用規範，並將其開發組件導出給系統使用，這裏還需要進行如下步驟：
輸出mysql的man手冊至man命令的查找路徑：添加如下行即可：
[root@node2 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
輸出mysql的頭文件至系統頭文件路徑/usr/include，這可以通過簡單的創建鏈接實現：
[root@node2 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node2 mysql]#
輸出mysql的庫文件給系統庫查找路徑：（文件只要是在/etc/ld.so.conf.d/下並且後綴是.conf就可以）而後讓系統重新載入系統庫
[root@node2 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node2 mysql]# ldconfig -v |grep mysql
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH環境變量，讓系統所有用戶可以直接使用mysql的相關命令:
[root@node2 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node2 mysql]# . /etc/profile
[root@node2 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸載drbd設備：
[root@node2 mysql]# umount /mnt/mysqldata/
四、corosync+pacemaker的安裝和配置
4.1 對node1和node2節點進行相應的配置
1：切換到主配置文件的目錄
[root@node1 ha]# cd /etc/corosync/
[root@node1 corosync]# ll
total 20
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]# cp corosync.conf.example corosync.conf
[root@node1 corosync]# vim corosync.conf
10 bindnetaddr: 192.168.145.0 #修改此行
#添加以下幾行
33 service {
34 ver: 0
35 name: pacemaker
36 use_mgmtd: yes
37 }
38 aisexec {
39 user: root
40 group: root
41 }
2:創建cluster目錄
[root@node1 corosync]# mkdir -pv /var/log/cluster
mkdir: created directory `/var/log/cluster'
3:爲了便面其他主機加入該集羣，需要認證，生成一authkey
[root@node1 corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@node1 corosync]# ll
total 28
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-r-------- 1 root root 128 Feb 7 22:47 authkey
-rw-r--r-- 1 root root 555 Feb 7 22:45 corosync.conf
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]#
4:將node1節點上的文件拷貝到節點node2上面（記住要帶-p)
[root@node1 corosync]# scp -p authkey corosync.conf node2:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00
corosync.conf 100% 561 0.6KB/s 00:00
[root@node1 corosync]# ssh node2 'mkdir -pv /var/log/cluster'
mkdir: created directory `/var/log/cluster'
[root@node1 corosync]#
4.2 在node1和node2節點上檢測
1:在node1和node2節點上面啓動 corosync 的服務
[root@node1 corosync]# service corosync start
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]# ssh node2 'service corosync start'
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]#
2:在node1上驗證corosync引擎是否正常啓動了
[root@node1 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
3: 在node1上查看初始化成員節點通知是否發出
[root@node1 ~]# grep -i totem /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] The network interface [192.168.145.100] is now up.
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
4: 在node1上檢查過程中是否有錯誤產生（避免stonith的錯誤）
[root@node1 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
出現如下1個錯誤：
Feb 7 22:51:43 node1 corosync[5149]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5161, rc=100)此處
解決方法：
仔細看了/var/log/messages日誌，或者使用crm_verify -L檢查一下錯誤，其實沒必要卸載重裝。這個錯誤是由於缺少snoith設備引起的，並不會影響corosync的運行。可以忽略這個錯誤。
[root@node1 corosync]# crm_verify -L
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
-V may provide more details
[root@node1 corosync]#
5: 在node1上檢查pacemaker時候已經啓動了？（如下顯示正常啓動）
[root@node1 ~]# grep -i pcmk_startup /var/log/messages
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Local hostname: node1.a.com
6: 在node2上驗證corosync引擎是否正常啓動了
[root@node2 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
7: 在node2上查看初始化成員節點通知是否發出
[root@node2 ~]# grep -i totem /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] The network interface [192.168.145.99] is now up.
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 7 22:53:03 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
8: 在node2上檢查過程中是否有錯誤產生（避免stonith的錯誤，如下顯示只有stonith錯誤，可忽略）
[root@node2 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
Feb 7 22:53:03 node2 corosync[5246]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5258, rc=100)
9: 在node2上檢查pacemaker時候已經啓動了？（如下顯示正常啓動）
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Local hostname: node2.a.com
10: 在node1和node2上查看羣集的狀態
[root@node1 corosync]# crm status
============
Last updated: Wed Oct 17 21:14:58 2012
Stack: openais
Current DC: node1.a.com - partition WITHOUT quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
1 Nodes configured, 2 expected votes
0 Resources configured.
============
Online: [ node1.a.com ]
[root@node2 corosync]# crm status
============
Last updated: Thu Oct 18 15:56:46 2012
Stack: openais
Current DC: NONE
0 Nodes configured, 2 expected votes
0 Resources configured.
============
五．羣集管理
5.1 配置羣集的工作屬性
corosync默認啓用了stonith，而當前集羣並沒有相應的stonith設備，因此此默認配置目前尚不可用，這可以通過如下命令先禁用stonith：
[root@node1 corosync]# cd
[root@node1 ~]# crm configure property stonith-enabled=false
[root@node1 ~]#
[root@node2 corosync]# cd
[root@node2 ~]# crm configure property stonith-enabled=false
[root@node2 ~]#
對於雙節點的集羣來說，我們要配置此選項來忽略quorum，即這時候票數不起作用，一個節點也能正常運行：
[root@node1 ~]#
[root@node1 ~]# crm configure property no-quorum-policy=ignore
[root@node1 ~]#
[root@node2 ~]#
[root@node2 ~]# crm configure property no-quorum-policy=ignore
[root@node2 ~]#
定義資源的粘性值，使資源不能再節點之間隨意的切換，因爲這樣是非常浪費系統的資源的。
資源黏性值範圍及其作用：
0：這是默認選項。資源放置在系統中的最適合位置。這意味着當負載能力“較好”或較差的節點變得可用時才轉移資源。此選項的作用基本等同於自動故障回覆，只是資源可能會轉移到非之前活動的節點上；
大於0：資源更願意留在當前位置，但是如果有更合適的節點可用時會移動。值越高表示資源越願意留在當前位置；
小於0：資源更願意移離當前位置。絕對值越高表示資源越願意離開當前位置；
INFINITY：如果不是因節點不適合運行資源（節點關機、節點待機、達到migration-threshold 或配置更改）而強制資源轉移，資源總是留在當前位置。此選項的作用幾乎等同於完全禁用自動故障回覆；
-INFINITY：資源總是移離當前位置；
我們這裏可以通過以下方式爲資源指定默認黏性值：
[root@node1 ~]#
[root@node1 ~]# crm configure rsc_defaults resource-stickiness=100
[root@node1 ~]#
[root@node2 ~]#
[root@node2 ~]# crm configure rsc_defaults resource-stickiness=100
[root@node2 ~]#
5.2 定義羣集服務及資源
5.2.1 改變drbd的狀態
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm secondary mysql
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm primary mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
5.2.2 配置drbd爲羣集資源
1、查看當前集羣的配置信息，確保已經配置全局屬性參數爲兩節點集羣所適用
[root@node1 ~]# crm configure show
node node1.a.com
node node2.a.com
property $id="cib-bootstrap-options" \
dc-version="1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
[root@node1 ~]#
2、將已經配置好的DRBD設備/dev/drbd0定義爲集羣服務；
[root@node1 ~]# service drbd stop
Stopping all DRBD resources: .
[root@node1 ~]# chkconfig drbd off
[root@node1 ~]# ssh node2 'service drbd stop'
Stopping all DRBD resources: .
[root@node1 ~]# ssh node2 'chkconfig drbd off'
[root@node1 ~]# drbd-overview
drbd not loaded
[root@node1 ~]# ssh node2 'drbd-overview'
drbd not loaded
[root@node1 ~]#
3、配置drbd爲集羣資源：
提供drbd的RA目前由OCF歸類爲linbit，其路徑爲/usr/lib/ocf/resource.d/linbit/drbd。我們可以使用如下命令來查看此RA及RA的meta信息：
[root@node1 ~]# crm ra classes
heartbeat
lsb
ocf / heartbeat linbit pacemaker
stonith
[root@node1 ~]# crm ra list ocf linbit
drbd
[root@node1 ~]#
4、查看drbd的資源代理的相關信息：
[root@node1 ~]# crm ra info ocf:linbit:drbd
This resource agent manages a DRBD resource
as a master/slave resource. DRBD is a shared-nothing replicated storage
device. (ocf:linbit:drbd)
Master/Slave OCF Resource Agent for DRBD
Parameters (* denotes required, [] the default):
drbd_resource* (string): drbd resource name
The name of the drbd resource from the drbd.conf file.
drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf
Full path to the drbd.conf file.
Operations' defaults (advisory minimum):
start timeout=240
promote timeout=90
demote timeout=90
notify timeout=90
stop timeout=100
monitor_Slave interval=20 timeout=20 start-delay=1m
monitor_Master interval=10 timeout=20 start-delay=1m
[root@node1 ~]#
5、drbd需要同時運行在兩個節點上，但只能有一個節點（primary/secondary模型）是Master，而另一個節點爲Slave；因此，它是一種比較特殊的集羣資源，其資源類型爲多狀態（Multi-state）clone類型，即主機節點有Master和Slave之分，且要求服務剛啓動時兩個節點都處於slave狀態。
[root@node1 ~]# crm
crm(live)# configure
crm(live)configure# primitive mysqldrbd ocf:heartbeat:drbd params drbd_resource="mysql" op monitor role="Master" interval="30s" op monitor role="Slave" interval="31s" op start timeout="240s" op stop timeout="100s"
crm(live)configure# ms MS_mysqldrbd mysqldrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify="true"
crm(live)configure# show mysqldrbd
primitive mysqldrbd ocf:heartbeat:drbd \
params drbd_resource="mysql" \
op monitor interval="30s" role="Master" \
op monitor interval="31s" role="Slave" \
op start interval="0" timeout="240s" \
op stop interval="0" timeout="100s"
crm(live)configure# show MS_mysqldrbd
ms MS_mysqldrbd mysqldrbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
確定無誤後，提交：
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# exit
bye
[root@node1 ~]#
6、查看當前集羣運行狀態：
[root@node1 ~]# crm status
============
Last updated: Tue Feb 7 23:51:09 2012
Stack: openais
Current DC: node1.a.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
1 Resources configured.
============
Online: [ node2.a.com node1.a.com ]
Master/Slave Set: MS_mysqldrbd [mysqldrbd]
Masters: [ node1.a.com ]
Slaves: [ node2.a.com ]
[root@node1 ~]#
由上面的信息可以看出此時的drbd服務的Primary節點爲node1.a.com，Secondary節點爲node2.a.com。當然，也可以在node1上使用如下命令驗正當前主機是否已經成爲mysql資源的Primary節點：
[root@node1 ~]# drbdadm role mysql
Primary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
我們實現將drbd設置自動掛載至/mysqldata目錄。此外，此自動掛載的集羣資源需要運行於drbd服務的Master節點上，並且只能在drbd服務將某節點設置爲Primary以後方可啓動。
確保兩個節點上的設備已經卸載：
[root@node1 ~]# umount /dev/drbd0
[root@node2 ~]# umount /dev/drbd0
以下還在node1上操作：
[root@node1 ~]# crm
crm(live)# configure
crm(live)configure# primitive MysqlFS ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" op start timeout=60s op stop timeout=60s
crm(live)configure#
crm(live)configure# show changed
primitive MysqlFS ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" \
op start interval="0" timeout="60s" \
op stop interval="0" timeout="60s"
crm(live)configure#
crm(live)configure# commit
crm(live)configure# exit
bye
[root@node1 ~]#
7、mysql資源的定義（node1上操作）
先爲mysql集羣創建一個IP地址資源，通過集羣提供服務時使用，這個地址就是客戶端訪問mysql服務器使用的ip地址；
[root@node1 ~]# crm configure primitive myip ocf:heartbeat:IPaddr params ip=192.168.145.103
[root@node1 ~]# crm configure primitive mysqlserver lsb:mysqld
[root@node1 ~]# crm status
============
Last updated: Wed Feb 8 00:01:23 2012
Stack: openais
Current DC: node1.a.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ node2.a.com node1.a.com ]
Master/Slave Set: MS_mysqldrbd [mysqldrbd]
Masters: [ node1.a.com ]
Slaves: [ node2.a.com ]
MysqlFS (ocf::heartbeat:Filesystem): Started node1.a.com
myip (ocf::heartbeat:IPaddr): Started node2.a.com
mysqlserver (lsb:mysqld): Started node1.a.com
[root@node1 ~]#
8、配置資源的各種約束：
集羣擁有所有必需資源，但它可能還無法進行正確處理。資源約束則用以指定在哪些羣集節點上運行資源，以何種順序裝載資源，以及特定資源依賴於哪些其它資源。pacemaker共給我們提供了三種資源約束方法：
1）Resource Location（資源位置）：定義資源可以、不可以或儘可能在哪些節點上運行
2）Resource Collocation（資源排列）：排列約束用以定義集羣資源可以或不可以在某個節點上同時運行
3）Resource Order（資源順序）：順序約束定義集羣資源在節點上啓動的順序。
定義約束時，還需要指定分數。各種分數是集羣工作方式的重要組成部分。其實，從遷移資源到決定在已降級集羣中停止哪些資源的整個過程是通過以某種方式修改分數來實現的。分數按每個資源來計算，資源分數爲負的任何節點都無法運行該資源。在計算出資源分數後，集羣選擇分數最高的節點。INFINITY（無窮大）目前定義爲 1,000,000。加減無窮大遵循以下3個基本規則：
1）任何值 + 無窮大 = 無窮大
2）任何值 - 無窮大 = -無窮大
3）無窮大 - 無窮大 = -無窮大
定義資源約束時，也可以指定每個約束的分數。分數表示指派給此資源約束的值。分數較高的約束先應用，分數較低的約束後應用。通過使用不同的分數爲既定資源創建更多位置約束，可以指定資源要故障轉移至的目標節點的順序。
我們要定義如下的約束：
[root@node1 ~]# crm
crm(live)# configure
crm(live)configure# colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver
crm(live)configure# order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start
crm(live)configure# order myip_after_MysqlFS mandatory: MysqlFS myip
crm(live)configure# order mysqlserver_after_myip mandatory: myip mysqlserver
crm(live)configure# show changed
colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver
order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start
order myip_after_MysqlFS inf: MysqlFS myip
order mysqlserver_after_myip inf: myip mysqlserver
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# exit
bye
[root@node1 ~]#
9.查看配置信息和狀態，並測試：
[root@node1 ~]# crm configure show
node node1.a.com
node node2.a.com
primitive MysqlFS ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" \
op start interval="0" timeout="60s" \
op stop interval="0" timeout="60s"
primitive myip ocf:heartbeat:IPaddr \
params ip="192.168.145.103"
primitive mysqldrbd ocf:heartbeat:drbd \
params drbd_resource="mysql" \
op monitor interval="30s" role="Master" \
op monitor interval="31s" role="Slave" \
op start interval="0" timeout="240s" \
op stop interval="0" timeout="100s"
primitive mysqlserver lsb:mysqld
ms MS_mysqldrbd mysqldrbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver
order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start
order myip_after_MysqlFS inf: MysqlFS myip
order mysqlserver_after_myip inf: myip mysqlserver
property $id="cib-bootstrap-options" \
dc-version="1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
[root@node1 ~]#
[root@node1 ~]# crm status
============
Last updated: Wed Feb 8 00:08:06 2012
Stack: openais
Current DC: node1.a.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ node2.a.com node1.a.com ]
Master/Slave Set: MS_mysqldrbd [mysqldrbd]
Masters: [ node1.a.com ]
Slaves: [ node2.a.com ]
MysqlFS (ocf::heartbeat:Filesystem): Started node1.a.com
myip (ocf::heartbeat:IPaddr): Started node1.a.com
mysqlserver (lsb:mysqld): Started node1.a.com
[root@node1 ~]#
可見，服務現在在node1上正常運行：
在node1上的操作，查看羣集的運行狀態：
[root@node1 ~]# service mysqld status
MySQL running (8720) [ OK ]
[root@node1 ~]# mount |grep drbd
/dev/drbd0 on /mnt/mysqldata type ext3 (rw)
[root@node1 ~]#
[root@node1 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 5 mysql mysql 4096 Feb 8 00:05 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:AE:83:D1
inet addr:192.168.145.103 Bcast:192.168.145.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:67 Base address:0x2000
[root@node1 ~]#
在node2上的操作，查看羣集的運行狀態：
[root@node2 ~]# service mysqld status
MySQL is not running [FAILED]
[root@node2 ~]# mount |grep drbd
[root@node2 ~]# ll /mnt/mysqldata/
total 0
[root@node2 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:D1:D4:32
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:67 Base address:0x2000
[root@node2 ~]#
10.繼續測試羣集：
繼續測試：
在node1上操作，讓node1下線：
[root@node1 ~]# crm status
============
Last updated: Wed Feb 8 00:16:25 2012
Stack: openais
Current DC: node1.a.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Node node1.a.com: standby
Online: [ node2.a.com ]
Master/Slave Set: MS_mysqldrbd [mysqldrbd]
Masters: [ node2.a.com ]
Stopped: [ mysqldrbd:0 ]
MysqlFS (ocf::heartbeat:Filesystem): Started node2.a.com
myip (ocf::heartbeat:IPaddr): Started node2.a.com
mysqlserver (lsb:mysqld): Started node2.a.com
[root@node1 ~]#
在node2上的操作，查看羣集的運行狀態：
可見我們的資源已經都切換到了node2上：
[root@node2 ~]# service mysqld status
MySQL running (9093) [ OK ]
[root@node2 ~]# mount |grep drbd
/dev/drbd0 on /mnt/mysqldata type ext3 (rw)
[root@node2 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 5 mysql mysql 4096 Feb 8 00:16 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node2 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:D1:D4:32
inet addr:192.168.145.103 Bcast:192.168.101.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:67 Base address:0x2000
[root@node2 ~]#
現在一切正常，我們可以驗證mysql服務是否能被正常訪問：
首先，在node2上面建立一個用戶user1，密碼：123456.
我們定義的是通過VIP:192.168.145.103來訪問mysql服務，現在node2上建立一個可以讓某個網段主機能訪問的賬戶(這個內容會同步drbd設備同步到node1上)：
[root@node2 ~]# mysql
...
mysql> grant all on *.* to user1@'192.168.%.%' identified by '123456';
Query OK, 0 rows affected (0.01 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
[root@node2 ~]#
客戶端訪問測試
192.168．145.105客戶Ping測試192.168.145.103
192.168．145.105客戶訪問mysql數據庫192.168.145.103（成功訪問）
（注意，測試客戶端上要安裝mysql，否則無法執行mysql -u root -p -h 192.168.145.103）
[root@client ~]# mysql -u root -p -h 192.168.145.103
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 5.5.15-log MySQL Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
4 rows in set (0.00 sec)
mysql>
至此：使用corosync+drbd+pacemaker實現mysql服務器的高可用集羣成功完成！