Corosync與pacemaker安裝:
1.環境說明
(1).操作系統
CentOS 6.5 X86_64位系統
(2).軟件環境
**corosync-1.4.1-17.el6.x86_64
**pacemaker-1.1.10-14.el6.x86_64
**crmsh-1.2.6-4.el6.x86_64.rpm
**pssh-2.3.1-2.el6.x86_64.rpm
**drbd-8.4.3-33.el6.x86_64.rpm
**drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
(3).拓撲環境
節點數:2 分別爲:node1 node2
node1:172.16.100.6 node2:172.16.100.7 9 TestHost:172.16.100.88
拓撲結構如下圖所示:
實現配置過程如下:
1、準備工作
爲了配置一臺Linux主機成爲HA的節點，通常需要做出如下的準備工作：
1)所有節點的主機名稱和對應的IP地址解析服務可以正常工作，且每個節點的主機名稱需要跟"uname -n“命令的結果保持一致；因此，需要保證兩個節點上的/etc/hosts文件均爲下面的內容：
# vim /etc/hosts 172.16.100.6 node1.samlee.com node1 172.16.100.7 node2.samlee.com node2
爲了使得重新啓動系統後仍能保持如上的主機名稱，還分別需要在各節點執行類似如下的命令：
Node1配置:
# sed -i 's@\(HOSTNAME=\).*@\1node1.samlee.com@g' /etc/sysconfig/network # hostname node1.samlee.com
Node2配置：
# sed -i 's@\(HOSTNAME=\).*@\1node2.samlee.com@g' /etc/sysconfig/network # hostname node2.samlee.com
2)設定兩個節點可以基於密鑰進行ssh通信，這可以通過如下的命令實現：
Node1配置:
# ssh-keygen -t rsa -P '' # ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2 # ssh node2.samlee.com 'date';date
Node2配置:
# ssh-keygen -t rsa -P '' # ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1 # ssh node1.samlee.com 'date';date
3)設置5分鐘自動同步時間(node1、node2都需要配置)
# crontab -e */5 * * * * /sbin/ntpdata 172.16.100.10 &> /dev/null
2、安裝配置Corosync集羣管理工具
1)安裝Corosync工具及pacemaker(yum方式)
# yum -y install corosync # yum -y install pacemaker
安裝crmsh(rpm方式)
RHEL自6.4起不再提供集羣的命令行配置工具crmsh，轉而使用pcs；如果你習慣了使用crm命令，可下載相關的程序包自行安裝即可。crmsh依賴於pssh，因此需要一併下載。
# cd /root/corosync_packages/ # yum -y --nogpgcheck localinstall crmsh*.rpm pssh*.rpm
2)配置corosync(操作在node1.samlee.com上執行)
# cd /etc/corosync/
# cp corosync.conf.example corosync.conf
# vim corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: on --開啓認證功能
threads: 0 --CPU個數
interface {
ringnumber: 0
bindnetaddr: 172.16.0.0 --集羣節點運行所在的網絡地址
mcastaddr: 226.96.6.17 --組播傳輸地址
mcastport: 5405 --心跳信息檢測端口
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
##設置隨corosync啓動的服務
service {
ver: 0
name: pacemaker
}
##ais運行身份設定
aisexec {
user: root
group: root
}
並設定此配置文件中 bindnetaddr後面的IP地址爲你的網卡所在網絡的網絡地址，我們這裏的兩個節點在172.16.0.0網絡，因此這裏將其設定爲172.16.0.0；如下
bindnetaddr: 172.16.0.03)生成節點間通信時所用到的認證密鑰文件:
# corosync-keygen 如果隨機數不夠的話需要需要登錄狀態狂敲鍵盤
如發現corosync生成/etc/corosync/autokey需要敲隨機數，可以使用以下方式解決(僅測試環境參考使用):
# mv /dev/random /dev/h # ln /dev/urandom /dev/random # corosync-keygen # rm -rf /dev/random # mv /dev/h /dev/random
4)將corosync.conf和authkey複製至node2：
# scp -p corosync.conf authkey node2:/etc/corosync/
5)分別在node1、node2兩個節點中創建corosync生成的日誌所在的目錄
# mkdir /var/log/cluster # ssh node2 'mkdir /var/log/cluster'
6)啓動corosync服務
# service corosync start # ssh node2 '/etc/init.d/corosync start'
7)查詢Corosync集羣狀態
# crm status Last updated: Thu Aug 18 23:22:20 2016 Last change: Thu Aug 18 22:49:11 2016 via crmd on node2.samlee.com Stack: classic openais (with plugin) Current DC: node2.samlee.com - partition with quorum Version: 1.1.10-14.el6-368c726 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ node1.samlee.com node2.samlee.com ]
以上爲corosync集羣配置所有過程。
3、安裝配置drbd共享存儲管理工具
(1)下載rpm安裝包直接安裝即可:
# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
(2)配置/etc/drbd.d/global-common.conf
# vim /etc/drbd.d/global_common.conf
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
#wfc-timeout 120;
#degr-wfc-timeout 120;
}
disk {
on-io-error detach;
#fencing resource-only;
}
net {
cram-hmac-alg "sha1";
shared-secret "mydrbdlab";
}
syncer {
rate 1000M;
}
}(3)創建共享存儲分區大小爲5G(node1,node2都必須執行分區)。
# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (7675-15665, default 7675):
Using default value 7675
Last cylinder, +cylinders or +size{K,M,G} (7675-15665, default 15665): +5G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
#重新讀取內核分區
# partx -a /dev/sda
# cat /proc/partitions
major minor #blocks name
8 0 125829120 sda
8 1 204800 sda1
8 2 61440000 sda2
8 3 5248836 sda3(4)定義資源文件/etc/drbd.d/mystore.res,內容如下:
# vim /etc/drbd.d/mystore.res
resource mystore {
on node1.samlee.com {
device /dev/drbd0;
disk /dev/sda3;
address 172.16.100.6:7789;
meta-disk internal;
}
on node2.samlee.com {
device /dev/drbd0;
disk /dev/sda3;
address 172.16.100.7:7789;
meta-disk internal;
}
}以上文件在兩個節點上必須相同，因此，可以基於ssh將剛纔配置的文件全部同步至另外一個節點。
# scp /etc/drbd.d/* node2:/etc/drbd.d/
(5)在兩個節點上初始化已定義的資源並啓動服務：
1)初始化資源，在Node1和Node2上分別執行：
# drbdadm create-md mystore
2)啓動服務，在Node1和Node2上分別執行：
# service drbd start
3)查看啓動狀態：
# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5248636
也可以使用drbd-overview命令來查看：
# drbd-overview 0:mystore/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
從上面的信息中可以看出此時兩個節點均處於Secondary狀態。於是，我們接下來需要將其中一個節點設置爲Primary。在要設置爲Primary的節點上執行如下命令：
# drbdadm primary --force mystore
注： 也可以在要設置爲Primary的節點上使用如下命令來設置主節點：
# drbdadm -- --overwrite-data-of-peer primary mystore
(5)創建文件系統
文件系統的掛載只能在Primary節點進行，因此，也只有在設置了主節點後才能對drbd設備進行格式化,下面的操作在node1.samlee.com上完成：
# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate C r----- # mke2fs -t ext4 /dev/drbd0 # mkdir /mydata # mount /dev/drbd0 /mydata
(6)創建mysql用戶及mysql數據存儲目錄規劃
# groupadd -r -g 3306 mysql # useradd -r -g 3306 -d /mydata/data/ -s /sbin/nologin mysql --以上mysql用戶node1和node2節點都要創建 # mkdir /mydata/data # chown mysql.mysql /mydata/data/
考慮生產環境數據安全,可使用ACL權限控制:
# setfacl -m u:mysql:rwx /mydata/data/ # getfacl /mydata/data/ getfacl: Removing leading '/' from absolute path names # file: mydata/data/ # owner: mysql # group: mysql user::rwx user:mysql:rwx group::r-x mask::rwx other::r-x
掛載好drbd資源後及mysql用戶目錄環境準備後下面開始安裝配置mysql：
(1).安裝開發環境及開發包組:
# yum -y groupinstall "Development Tools" # yum -y groupinstall "Development Libraries" # yum -y install pcre-devel # yum -y install ncurses-devel # yum -y install openssl-devel
(2).安裝誇平臺編譯器cmake
# tar xf cmake-2.8.8.tar.gz # cd cmake-2.8.8 # ./configure # make && make install
(3).編譯安裝mysql-5.5.33
# tar xf mysql-5.5.33.tar.gz # cd mysql-5.5.33 # cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \ > -DMYSQL_DATADIR=/mydata/data \ > -DSYSCONFDIR=/etc \ > -DWITH_INNOBASE_STORAGE_ENGINE=1 \ > -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ > -DWITH_BLACKHOLE_STORAGE_ENGINE=1 \ > -DWITH_READLINE=1 \ > -DWITH_SSL=system \ > -DWITH_ZLIB=system \ > -DWITH_LIBWRAP=0 \ > -DMYSQL_UNIX_ADDR=/tmp/mysql.sock \ > -DDEFAULT_CHARSET=utf8 \ > -DDEFAULT_COLLATION=utf8_general_ci # make && make install
(4).數據庫服務器初始化(數據初始化創建系統數據庫，在主節點上創建一次即可)
# cd /usr/local/mysql/ # chown -R :mysql /usr/local/mysql/ #賦予目錄用戶組爲mysql #運行./script/mysql_install_db 創建系統庫 # scripts/mysql_install_db --user=mysql --datadir=/mydata/data/ # cp support-files/my-large.cnf /etc/my.cnf #複製配置文件 # vim/etc/my.cnf #編輯配置文件 在[mysqld]下添加此項: datadir = /mydata/data innodb_file_per_table = on # cp support-files/mysql.server /etc/init.d/mysqld #複製啓動腳本 # chkconfig --add mysqld #加入服務列表 # service mysqld start
導出頭文件,庫文件和man文檔:
# ln -sv /usr/local/mysql/include/ /usr/include/mysql # vim /etc/ld.so.conf.d/mysql.conf 添加此行: /usr/local/mysql/lib/ # ldconfig # ldconfig -p | grep mysql # vim /etc/man.config 添加此行: MANPATH /usr/local/mysql/man # man mysqld
(5)配置PATH環境變量
# vim /etc/profile.d/mysql.sh export PATH=$PATH:/usr/local/mysql/bin # chmod +x /etc/profile.d/mysql.sh # . /etc/profile.d/mysql.sh
(6)測試成功如下:
node1:
[root@node1 ~]# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate C r----- /mydata ext4 5.0G 168M 4.6G 4% [root@node1 ~]# ll /mydata/data/ total 28700 -rw-rw---- 1 mysql mysql 18874368 Aug 19 01:03 ibdata1 -rw-rw---- 1 mysql mysql 5242880 Aug 19 01:03 ib_logfile0 -rw-rw---- 1 mysql mysql 5242880 Aug 19 01:03 ib_logfile1 drwx------ 2 mysql root 4096 Aug 19 01:00 mysql -rw-rw---- 1 mysql mysql 107 Aug 19 01:03 mysql-bin.000001 -rw-rw---- 1 mysql mysql 19 Aug 19 01:03 mysql-bin.index -rw-r----- 1 mysql root 1900 Aug 19 01:03 node1.samlee.com.err -rw-rw---- 1 mysql mysql 6 Aug 19 01:03 node1.samlee.com.pid drwx------ 2 mysql mysql 4096 Aug 19 01:00 performance_schema drwx------ 2 mysql root 4096 Aug 19 01:00 test 連接登陸mysql #mysql 創建一個遠程用戶： mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'172.16.%.%' IDENTIFIED BY 'redhat'; 重讀授權表 mysql> FLUSH PRIVILEGES;
將node1停止mysql服務降級爲備節點:
# service mysqld stop # umount /mydata/ # drbdadm secondary mystore # drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
提升node2爲主節點啓動mysql服務:
# drbdadm primary mystore # mount /dev/drbd0 /mydata/ # service mysqld start
測試完成後，停止所有服務，將所有節點降級爲備節點狀態:
# service mysqld stop # umount /mydata/ # drbdadm secondary mystore # drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
停止所有節點drbd服務，drbd服務必須加入服務列表中：
# chkconfig --add drbd # chkconfig --list drbd drbd 0:off 1:off 2:on 3:on 4:on 5:on 6:off # chkconfig drbd off # service drbd stop
配置drbd+mysql資源高可用配置如下：
crm(live)configure# property stonith-enabled=false crm(live)configure# property no-quorum-policy=ignore crm(live)configure# rsc_defaults resource-stickiness=100 crm(live)configure# verify crm(live)configure# commit crm(live)configure# primitive mysqlvip ocf:heartbeat:IPaddr params ip=172.16.100.66 nic=eth0 cidr_netmask=16 crm(live)configure# verify crm(live)configure# primitive mysqlstore ocf:linbit:drbd params drbd_resource=mystore op monitor role=Master interval=30s timeout=30s op monitor role=Slave interval=60s timeout=20s op start timeout=240s op stop timeout=100s crm(live)configure# verify crm(live)configure# master ms_mysqlstore mysqlstore meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify="true" crm(live)configure# verify crm(live)configure# primitive mysqlfs ocf:heartbeat:Filesystem params device=/dev/drbd0 directory=/mydata fstype=ext4 op monitor interval=30s timeout=40s op start timeout=60s op stop timeout=60s on-fail=restart crm(live)configure# verify crm(live)configure# primitive mysqlserver lsb:mysqld crm(live)configure# verify crm(live)configure# colocation mysqlfs_with_ms_mysqlstore_master inf: mysqlfs ms_mysqlstore:Master crm(live)configure# verify crm(live)configure# colocation mysqlserver_with_mysqlfs inf: mysqlfs mysqlserver crm(live)configure# verify crm(live)configure# colocation mysqlvip_with_mysqlserver inf: mysqlvip mysqlserver crm(live)configure# verify crm(live)configure# order mysqlfs_after_ms_mysqlstore_master mandatory: ms_mysqlstore:promote mysqlfs:start crm(live)configure# verify crm(live)configure# order mysqlserver_after_ms_mysqlstore inf: mysqlfs mysqlserver crm(live)configure# verify crm(live)configure# commit # crm configure show node node1.samlee.com node node2.samlee.com primitive mysqlfs ocf:heartbeat:Filesystem \ params device="/dev/drbd0" directory="/mydata" fstype="ext4" \ op monitor interval="30s" timeout="40s" \ op start timeout="60s" interval="0" \ op stop timeout="60s" on-fail="restart" interval="0" primitive mysqlserver lsb:mysqld primitive mysqlstore ocf:linbit:drbd \ params drbd_resource="mystore" \ op monitor role="Master" interval="30s" timeout="30s" \ op monitor role="Slave" interval="60s" timeout="20s" \ op start timeout="240s" interval="0" \ op stop timeout="100s" interval="0" primitive mysqlvip ocf:heartbeat:IPaddr \ params ip="172.16.100.66" nic="eth0" cidr_netmask="16" ms ms_mysqlstore mysqlstore \ meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" colocation mysqlfs_with_ms_mysqlstore_master inf: mysqlfs ms_mysqlstore:Master colocation mysqlserver_with_mysqlfs inf: mysqlfs mysqlserver colocation mysqlvip_with_mysqlserver inf: mysqlvip mysqlserver order mysqlfs_after_ms_mysqlstore_master inf: ms_mysqlstore:promote mysqlfs:start order mysqlserver_after_ms_mysqlstore inf: mysqlfs mysqlserver property $id="cib-bootstrap-options" \ dc-version="1.1.10-14.el6-368c726" \ cluster-infrastructure="classic openais (with plugin)" \ expected-quorum-votes="2" \ stonith-enabled="false" \ no-quorum-policy="ignore" rsc_defaults $id="rsc-options" \ resource-stickiness="100"
查詢集羣狀態信息:
# crm status Last updated: Fri Aug 19 02:03:41 2016 Last change: Fri Aug 19 02:00:03 2016 via cibadmin on node1.samlee.com Stack: classic openais (with plugin) Current DC: node2.samlee.com - partition with quorum Version: 1.1.10-14.el6-368c726 2 Nodes configured, 2 expected votes 5 Resources configured Online: [ node1.samlee.com node2.samlee.com ] mysqlvip (ocf::heartbeat:IPaddr): Started node1.samlee.com Master/Slave Set: ms_mysqlstore [mysqlstore] Masters: [ node1.samlee.com ] Slaves: [ node2.samlee.com ] mysqlfs (ocf::heartbeat:Filesystem): Started node1.samlee.com mysqlserver (lsb:mysqld): Started node1.samlee.com
通過客戶連接測試如下:
# mysql -h172.16.100.66 -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.5.33-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema| | mysql | | performance_schema| | test | +--------------------+ 4 rows in set (0.00 sec)