mysql服務器的HA集羣之corosync+drbd+pacemaker實現 中

 三、mysql安裝與配置

3.1 將node1設爲主節點，node2設爲備份節點 

 

[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm secondary mysql
[root@node2 ~]# drbdadm role mysql
Secondary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#

[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm primary mysql
[root@node1 ~]# drbdadm role mysql
Primary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#

3.2 在node1.magedu.com上安裝配置mysql 

[root@node1 ~]# groupadd -r mysql
[root@node1 ~]# useradd -g mysql -r mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node1 ~]# mkdir -pv /mnt/mysqldata/data
mkdir: created directory `/mnt/mysqldata/data'
[root@node1 ~]# chown -R mysql.mysql /mnt/mysqldata/data/
[root@node1 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 2 mysql mysql 4096 Feb 7 21:48 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 ~]#cd ha/
[root@node1 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node1 ~]# cd /usr/local/
[root@node1 src]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node1 src]# ll
[root@node1 src]# cd mysql
[root@node1 mysql]# chown -R mysql:mysql .
[root@node1 mysql]# scripts/mysql_install_db --user=mysql --datadir=/mnt/mysqldata/data/
[root@node1 mysql]# chown -R root .
[root@node1 mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@node1 mysql]# vim /etc/my.cnf
39 thread_concurrency = 2
40 datadir = /mnt/mysqldata/data/ #指定mysql數據文件的存放位置(添加)

爲mysql提供sysv服務腳本，使其能使用service命令：

[root@node1 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@node1 mysql]# scp /etc/my.cnf node2:/etc/
my.cnf 100% 4696 4.6KB/s 00:00
[root@node1 mysql]# scp /etc/rc.d/init.d/mysqld node2:/etc/rc.d/init.d/
mysqld 100% 10KB 10.4KB/s 00:00
[root@node1 mysql]#
[root@node1 mysql]# chkconfig --add mysqld
[root@node1 mysql]# chkconfig mysqld off
[root@node1 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node1 mysql]# service mysqld start
Starting MySQL..... [ OK ]
[root@node1 mysql]# l1 /mnt/mysqldata/data/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.magedu.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node1 mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@node1 mysql]#

爲了使用mysql的安裝符合系統使用規範，並將其開發組件導出給系統使用，這裏還需要進行如下步驟：

輸出mysql的man手冊至man命令的查找路徑：添加如下行即可：
[root@node1 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
輸出mysql的頭文件至系統頭文件路徑/usr/include，這可以通過簡單的創建鏈接實現：
[root@node1 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node1 mysql]#
輸出mysql的庫文件給系統庫查找路徑：（文件只要是在/etc/ld.so.conf.d/下並且後綴是.conf就可以）而後讓系統重新載入系統庫
[root@node1 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node1 mysql]# ldconfig -v |grep mysql 
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH環境變量，讓系統所有用戶可以直接使用mysql的相關命令:
[root@node1 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node1 mysql]# . /etc/profile
[root@node1 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸載drbd設備：
[root@node1 mysql]# umount /mnt/mysqldata/
3.3 將node2設爲主節點，node1設爲備份節點 

[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm secondary mysql
[root@node1 ~]# drbdadm role mysql
Secondary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#

[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm primary mysql
[root@node2 ~]# drbdadm role mysql
Primary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#

3.4 在node2.magedu.com上安裝配置mysql 

添加用戶和組：
[root@node2 ~]# groupadd -r mysql
[root@node2 ~]# useradd -g mysql -r mysql
[root@node2 ~]# drbd-overview 
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r---- 
[root@node2 ~]#
掛載drbd設備：
[root@node2 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node2 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 5 mysql mysql 4096 Feb 7 22:00 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node2 ~]# ll /mnt/mysqldata/data/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.magedu.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node2 ~]#
安裝mysql：
[root@node2 ~]# cd ha/
[root@node2 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node2 ha]# cd /usr/local/
[root@node2 local]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node2 local]# cd mysql
一定不能對數據庫進行初始化，因爲我們在node1上已經初始化了：
[root@node2 mysql]# chown -R root:mysql .
mysql主配置文件和sysc服務腳本已經從node1複製過來了，不用在添加。
管理mysql服務：
[root@node2 mysql]# chkconfig --add mysqld
[root@node2 mysql]# chkconfig mysqld off
[root@node2 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node2 mysql]#
啓動mysql服務：
[root@node2 mysql]# service mysqld start
Starting MySQL....... [ OK ]
[root@node2 mysql]# ll /mnt/mysqldata/data/
total 28756
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:29 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 107 Feb 7 22:29 mysql-bin.000002
-rw-rw---- 1 mysql mysql 38 Feb 7 22:29 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.magedu.com.err
-rw-rw---- 1 mysql root 941 Feb 7 22:29 node2.magedu.com.err
-rw-rw---- 1 mysql mysql 5 Feb 7 22:29 node2.magedu.com.pid
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
測試之後關閉服務：
[root@node2 mysql]# service mysqld stop 
Shutting down MySQL. [ OK ]
[root@node2 mysql]#
爲了使用mysql的安裝符合系統使用規範，並將其開發組件導出給系統使用，這裏還需要進行如下步驟：

輸出mysql的man手冊至man命令的查找路徑：添加如下行即可：
[root@node2 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
輸出mysql的頭文件至系統頭文件路徑/usr/include，這可以通過簡單的創建鏈接實現：
[root@node2 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node2 mysql]#
輸出mysql的庫文件給系統庫查找路徑：（文件只要是在/etc/ld.so.conf.d/下並且後綴是.conf就可以）而後讓系統重新載入系統庫
[root@node2 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node2 mysql]# ldconfig -v |grep mysql 
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH環境變量，讓系統所有用戶可以直接使用mysql的相關命令:
[root@node2 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node2 mysql]# . /etc/profile
[root@node2 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸載drbd設備：
[root@node2 mysql]# umount /mnt/mysqldata/

四、corosync+pacemaker的安裝和配置 

4.1安裝有關軟件 

[root@node1 ~]# cd ha/
[root@node1 ha]# yum localinstall -y *.rpm --nogpgcheck
…
Installed:
cluster-glue.i386 0:1.0.6-1.6.el5 cluster-glue-libs.i386 0:1.0.6-1.6.el5
corosync.i386 0:1.2.7-1.1.el5 corosynclib.i386 0:1.2.7-1.1.el5
heartbeat.i386 0:3.0.3-2.3.el5 heartbeat-libs.i386 0:3.0.3-2.3.el5
libesmtp.i386 0:1.0.4-5.el5 openais.i386 0:1.1.3-1.6.el5
openaislib.i386 0:1.1.3-1.6.el5 pacemaker.i386 0:1.1.5-1.1.el5
pacemaker-cts.i386 0:1.1.5-1.1.el5 pacemaker-libs.i386 0:1.1.5-1.1.el5
perl-TimeDate.noarch 1:1.16-5.el5 resource-agents.i386 0:1.0.4-1.1.el5

Dependency Installed:
libibverbs.i386 0:1.1.2-4.el5 librdmacm.i386 0:1.0.8-5.el5
libtool-ltdl.i386 0:1.5.22-6.1 lm_sensors.i386 0:2.10.7-4.el5
openhpi-libs.i386 0:2.14.0-5.el5 openib.noarch 0:1.4.1-3.el5

Complete!
[root@node1 ha]#

[root@node2 ~]# cd ha/
[root@node2 ha]# yum localinstall -y *.rpm --nogpgcheck
…
Installed:
cluster-glue.i386 0:1.0.6-1.6.el5 cluster-glue-libs.i386 0:1.0.6-1.6.el5
corosync.i386 0:1.2.7-1.1.el5 corosynclib.i386 0:1.2.7-1.1.el5
heartbeat.i386 0:3.0.3-2.3.el5 heartbeat-libs.i386 0:3.0.3-2.3.el5
libesmtp.i386 0:1.0.4-5.el5 openais.i386 0:1.1.3-1.6.el5
openaislib.i386 0:1.1.3-1.6.el5 pacemaker.i386 0:1.1.5-1.1.el5
pacemaker-cts.i386 0:1.1.5-1.1.el5 pacemaker-libs.i386 0:1.1.5-1.1.el5
perl-TimeDate.noarch 1:1.16-5.el5 resource-agents.i386 0:1.0.4-1.1.el5

Dependency Installed:
libibverbs.i386 0:1.1.2-4.el5 librdmacm.i386 0:1.0.8-5.el5
libtool-ltdl.i386 0:1.5.22-6.1 lm_sensors.i386 0:2.10.7-4.el5
openhpi-libs.i386 0:2.14.0-5.el5 openib.noarch 0:1.4.1-3.el5

Complete!
[root@node2 ha]#

4.2 對node1和node2節點進行相應的配置 

1：切換到主配置文件的目錄
[root@node1 ha]# cd /etc/corosync/
[root@node1 corosync]# ll 
total 20
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]# cp corosync.conf.example corosync.conf 
[root@node1 corosync]# vim corosync.conf
10 bindnetaddr: 192.168.101.0 #修改此行
#添加以下幾行
33 service {
34 ver: 0
35 name: pacemaker
36 use_mgmtd: yes
37 }
38 aisexec {
39 user: root
40 group: root
41 }
2:創建cluster目錄
[root@node1 corosync]# mkdir -pv /var/log/cluster
mkdir: created directory `/var/log/cluster'
3:爲了便面其他主機加入該集羣，需要認證，生成一authkey
[root@node1 corosync]# corosync-keygen 
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@node1 corosync]# ll
total 28
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-r-------- 1 root root 128 Feb 7 22:47 authkey
-rw-r--r-- 1 root root 555 Feb 7 22:45 corosync.conf
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]#
4:將node1節點上的文件拷貝到節點node2上面（記住要帶-p)
[root@node1 corosync]# scp -p authkey corosync.conf node2:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00 
corosync.conf 100% 561 0.6KB/s 00:00 
[root@node1 corosync]# ssh node2 'mkdir -pv /var/log/cluster'
mkdir: created directory `/var/log/cluster'
[root@node1 corosync]#

4.3 在node1和node2節點上檢測

 

1:在node1和node2節點上面啓動 corosync 的服務
[root@node1 corosync]# service corosync start
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]# ssh node2 'service corosync start'
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]# 
2:在node1上驗證corosync引擎是否正常啓動了
[root@node1 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'. 
3: 在node1上查看初始化成員節點通知是否發出
[root@node1 ~]# grep -i totem /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] The network interface [192.168.101.81] is now up.
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] A processor joined or left the membership and a new membership was formed. 
4: 在node1上檢查過程中是否有錯誤產生（避免stonith的錯誤）
[root@node1 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
出現如下1個錯誤：
Feb 7 22:51:43 node1 corosync[5149]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5161, rc=100)此處
解決方法：
仔細看了/var/log/messages日誌，或者使用crm_verify -L檢查一下錯誤，其實沒必要卸載重裝。這個錯誤是由於缺少snoith設備引起的，並不會影響corosync的運行。可以忽略這個錯誤。
[root@node1 corosync]# crm_verify -L
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
-V may provide more details
[root@node1 corosync]#
5: 在node1上檢查pacemaker時候已經啓動了？（如下顯示正常啓動）
[root@node1 ~]# grep -i pcmk_startup /var/log/messages 
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Local hostname: node1.magedu.com
6: 在node2上驗證corosync引擎是否正常啓動了
[root@node2 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
7: 在node2上查看初始化成員節點通知是否發出
[root@node2 ~]# grep -i totem /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] The network interface [192.168.101.82] is now up.
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 7 22:53:03 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
8: 在node2上檢查過程中是否有錯誤產生（避免stonith的錯誤，如下顯示只有stonith錯誤，可忽略）
[root@node2 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
Feb 7 22:53:03 node2 corosync[5246]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5258, rc=100)
9: 在node2上檢查pacemaker時候已經啓動了？（如下顯示正常啓動）
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Local hostname: node2.magedu.com
10: 在node1和node2上查看羣集的狀態
[root@node1 corosync]# crm status
============
Last updated: Tue Feb 7 23:18:39 2012
Stack: openais
Current DC: node1.magedu.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node2.magedu.com node1.magedu.com ]

[root@node1 corosync]#

[root@node2 corosync]# crm status
============
Last updated: Tue Feb 7 23:20:06 2012
Stack: openais
Current DC: node1.magedu.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node2.magedu.com node1.magedu.com ]

[root@node2 corosync]#