一、DRBD簡介
DRBD的全稱爲：Distributed ReplicatedBlock Device(DRBD)分佈式塊設備複製,DRBD是由內核模塊和相關腳本而構成，用以構建高可用性的集羣。其實現方式是通過網絡來鏡像整個設備。你可以把它看作是一種網絡RAID。它允許用戶在遠程機器上建立一個本地塊設備的實時鏡像。
1.下載rpm包
目前適用CentOS 5的drbd版本主要有8.0、8.2、8.3三個版本，其對應的rpm包的名字分別爲drbd, drbd82和drbd83，對應的內核模塊的名字分別爲kmod-drbd, kmod-drbd82和kmod-drbd83。而適用於CentOS 6的版本爲8.4，其對應的rpm包爲drbd和drbd-kmdl，但在實際選用時，要切記兩點：drbd和drbd-kmdl的版本要對應；另一個是drbd-kmdl的版本要與當前系統的內容版本相對應。各版本的功能和配置等略有差異；我們實驗所用的平臺爲x86_64且系統爲CentOS 6.5，因此需要同時安裝內核模塊和管理工具。我們這裏選用最新的8.4的版本(drbd-8.4.3-33.el6.x86_64.rpm和drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
2.準備工作
兩個節點的節點的名稱和對應的ip地址解析服務應該能正常工作
3.安裝，由於drbd包沒有依賴關係，可以直接使用rpm安裝
[root@node1 ~]# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm[root@node2 ~]# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
重新啓動rsyslog服務
4.配置說明
drbd的主配置文件爲/etc/drbd.conf；爲了管理的便捷性，目前通常會將些配置文件分成多個部分，且都保存至/etc/drbd.d目錄中，主配置文件中僅使用"include"指令將這些配置文件片斷整合起來。通常，/etc/drbd.d目錄中的配置文件爲global_common.conf和所有以.res結尾的文件。其中global_common.conf中主要定義global段和common段，而每一個.res的文件用於定義一個資源。
在配置文件中，global段僅能出現一次，且如果所有的配置信息都保存至同一個配置文件中而不分開爲多個文件的話，global段必須位於配置文件的最開始處。目前global段中可以定義的參數僅有minor-count, dialog-refresh, disable-ip-verification和usage-count。
common段則用於定義被每一個資源默認繼承的參數，可以在資源定義中使用的參數都可以在common段中定義。實際應用中，common段並非必須，但建議將多個資源共享的參數定義爲common段中的參數以降低配置文件的複雜度。
resource段則用於定義drbd資源，每個資源通常定義在一個單獨的位於/etc/drbd.d目錄中的以.res結尾的文件中。資源在定義時必須爲其命名，名字可以由非空白的ASCII字符組成。每一個資源段的定義中至少要包含兩個host子段，以定義此資源關聯至的節點，其它參數均可以從common段或drbd的默認中進行繼承而無須定義。
5.配置文件/etc/drbd.d/global_common.conf
global {usage-count no;//drbd官網用來統計drbd的使用數據的# minor-count dialog-refresh disable-ip-verification}common {//提供共享配置handlers {//處理器，在特定的環境下執行的命令# These are EXAMPLE handlers only.# They may have severe implications,# like hard resetting the node under certain circumstances.# Be careful when chosing your poison.pri-on-incon-degr"/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";//在主節點降級的情況下要執行的命令pri-lost-after-sb"/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";//在備節點接替主節點前，對主節點的操作local-io-error"/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f"; 當本地發送io錯誤時的操作# fence-peer"/usr/lib/drbd/crm-fence-peer.sh";# split-brain"/usr/lib/drbd/notify-split-brain.sh root";# out-of-sync"/usr/lib/drbd/notify-out-of-sync.sh root";# before-resync-target"/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;}startup {//可以不配置，使用默認即可# wfc-timeout 等待對端上線的超時時間# degr-wfc-timeout 降級的超時時間#outdated-wfc-timeout 過期的等待超時時間#wait-after-sb 腦裂的等待時間}options {//可以不配置 使用默認即可# cpu-mask on-no-data-accessible}disk {# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes# disk-drain md-flushes resync-rate resync-after al-extents# c-plan-ahead c-delay-target c-fill-target c-max-rate# c-min-rate disk-timeouton-io-error 當發生io錯誤是，應該要做的操作，有三個選項，pass_on：降級當前節點；call-local-io-error：執行本機的io-error操作；detach：將磁盤拆掉}net {protocol C 協議版本cram-hmac-alg"sha1"shared-secret"kjasdbiu2178uwhbj"# protocol timeout max-epoch-size max-buffers unplug-watermark# connect-intping-intsndbuf-size rcvbuf-size ko-count# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri# after-sb-1pri after-sb-2pri always-asbp rr-conflict# ping-timeout data-integrity-alg tcp-cork on-congestionsyncer{rate 1000M 同步的速率}}
6.準備磁盤設備，雙方節點都需要準備，最好是能等同大小，編號最好也能一樣.
[root@node2 ~]# fdisk /dev/sdaWARNING: DOS-compatible modeisdeprecated. It's strongly recommended toswitchoff the mode (command'c') and change display units tosectors (command'u').Command (mforhelp): nCommand actione extendedp primary partition (1-4)pPartition number (1-4):3First cylinder (7859-15665,default7859):Usingdefaultvalue7859Last cylinder, +cylinders or +size{K,M,G} (7859-15665,default15665): +10GCommand (mforhelp): wThe partition table has been altered!Calling ioctl() to re-read partition table.WARNING: Re-reading the partition table failedwitherror16: Device or resource busy.The kernel still uses the old table. Thenewtable will be used atthe next reboot or after you run partprobe(8) or kpartx(8)Syncing disks.[root@node2 ~]# kpartx -af /dev/sdadevice-mapper: reload ioctl on sda1 failed: Invalid argumentcreate/reload failed on sda1device-mapper: reload ioctl on sda2 failed: Invalid argumentcreate/reload failed on sda2device-mapper: reload ioctl on sda3 failed: Invalid argumentcreate/reload failed on sda3[root@node2 ~]# partx -a /dev/sdaBLKPG: Device or resource busyerror adding partition1BLKPG: Device or resource busyerror adding partition2BLKPG: Device or resource busyerror adding partition3
7.準備資源配置文件，如drbd.re
resource drbd {on node1.zero1.com { 用on來標誌節點名稱device /dev/drbd0； 指定drbd的編號disk /dev/sda3； 磁盤分區編號address192.168.1.200:7789； 監聽的套接字meta-diskinternal； 對於原始數據的處理，此處爲保持在本磁盤}on node2.zero1.com {device /dev/drbd0；disk /dev/sda3；address192.168.1.201:7789；meta-diskinternal；}}
8.將數據複製到節點
[root@node1 drbd.d]# scp * node2:/etc/drbd.d/
七、初始化測試
1.創建資源
[root@node1 drbd.d]# drbdadm create-md drbdWriting meta data...initializing activity logNOT initializing bitmaplk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directoryNew drbd meta data block successfully created.//創建成功lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
2.啓動服務
[root@node2 ~]# service drbd startStarting DRBD resources: [create res: drbdprepare disk: drbdadjust disk: drbdadjust net: drbd]
兩邊要同時啓動
3.查看啓動狀態
[root@node2 ~]# cat /proc/drbdversion:8.4.3(api:1/proto:86-101)GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@,2013-11-2912:28:000: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----ns:0nr:0dw:0dr:0al:0bm:0lo:0pe:0ua:0ap:0ep:1wo:f oos:10489084也可以通過drbd-overview命令來查看[root@node2 ~]# drbd-overview0:drbd/0Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
從上面的信息中可以看出此時兩個節點均處於Secondary狀態。
4.提升節點
[root@node1 drbd.d]# drbdadm primary --force drbd[root@node1 drbd.d]# drbd-overview0:drbd/0SyncSource Primary/Secondary UpToDate/Inconsistent C r-----[>....................] sync'ed:4.8% (9760/10240)M
可以看到正在同步數據，等數據同步完再查看
[root@node1 ~]# cat /proc/drbdversion:8.4.3(api:1/proto:86-101)GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@,2013-11-2912:28:000: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----ns:10490880nr:0dw:0dr:10496952al:0bm:641lo:0pe:0ua:0ap:0ep:1wo:f oos:0
5.創建文件系統
文件系統的掛載只能在Primary節點進行，因此，也只有在設置了主節點後才能對drbd設備進行格式化：
[root@node1 ~]# mke2fs -t ext4 /dev/drbd0[root@node1 ~]# mkdir /drbd[root@node1 ~]# mount /dev/drbd0 /drbd/
6.切換Primary和Secondary節點
對主Primary/Secondary模型的drbd服務來講，在某個時刻只能有一個節點爲Primary，因此，要切換兩個節點的角色，只能在先將原有的Primary節點設置爲Secondary後，才能原來的Secondary節點設置爲Primary：
主節點
[root@node1 ~]# cp /etc/fstab /drbd/[root@node1 ~]# umount /drbd/[root@node1 ~]# drbdadm secondary drbd
備節點
[root@node2 ~]# drbdadm primary drbd[root@node2 ~]# mkdir /drbd[root@node2 ~]# mount /dev/drbd0 /drbd/[root@node2 ~]# ll /drbd/total20-rw-r--r--1root root921Apr192014fstabdrwx------2root root16384Apr192014lost+found
可以看到我們的數據在節點2上可以看到了
[root@node2 ~]# cat /proc/drbdversion:8.4.3(api:1/proto:86-101)GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@,2013-11-2912:28:000: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----ns:24nr:10788112dw:10788136dr:1029al:2bm:642lo:0pe:0ua:0ap:0ep:1wo:f oos:0
可以看到現在node2爲主節點
八、corosync的安裝與配置
安裝corosync。
2.crm的安裝
[root@node1 ~]# yum install -y crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm[root@node2 ~]# yum install -y crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm
3.定義drbd資源
[root@node1 ~]# crmcrm(live)# configurecrm(live)configure# primitive mariadbdrbd ocf:linbit:drbd params drbd_resource=drbd op monitor role=Master interval=10s timeout=20s op monitor role=Slave interval=20s timeout=20s op start timeout=240s op stop timeout=120s定於drbd的主從資源crm(live)configure# ms ms_mariadbdrbd mariadbdrbd meta master-max=1master-node-max=1clone-max=2clone-node-max=1notify=true
4.定義文件系統資源和約束關係
crm(live)configure# primitive mariadbstore ocf:heartbeat:Filesystem params device="/dev/drbd0"directory="/mydata"fstype="ext4"op monitor interval=40s timeout=40s op start timeout=60s op stop timeout=60scrm(live)configure# colocation mariadbstore_with_ms_mariadbdrbd inf: mariadbstore ms_mariadbdrbd:Mastercrm(live)configure# order ms_mariadbrbd_before_mariadbstore mandatory: ms_mariadbdrbd:promote mariadbstore:start
5.增加vip和MariaDB的資源及約束關係
crm(live)configure# primitive madbvip ocf:heartbeat:IPaddr2 params ip="192.168.1.240"op monitor interval=20s timeout=20s on-fail=restartcrm(live)configure# primitive maserver lsb:mysqld op monitor interval=20s timeout=20s on-fail=restartcrm(live)configure# verify
定義約束關係
crm(live)configure# colocation maserver_with_mariadbstore inf: maserver mariadbstorecrm(live)configure# order mariadbstore_before_maserver mandatory: mariadbstore:start maserver:startcrm(live)configure# verifycrm(live)configure# colocation madbvip_with_maserver inf: madbvip maservercrm(live)configure# order madbvip_before_masever mandatory: madbvip maservercrm(live)configure# verifycrm(live)configure# commit
6.查看所有定義的資源
node node1.zero1.comnode node2.zero1.comprimitive madbvip ocf:heartbeat:IPaddr2 \params ip="192.168.1.240"\op monitor interval="20s"timeout="20s"on-fail="restart"primitive mariadbdrbd ocf:linbit:drbd \params drbd_resource="drbd"\op monitor role="Master"interval="30s"timeout="20s"\op monitor role="Slave"interval="60s"timeout="20s"\op start timeout="240s"interval="0"\op stop interval="0s"timeout="100s"primitive mariadbstore ocf:heartbeat:Filesystem \params device="/dev/drbd0"directory="/mydata"fstype="ext4"\op monitor interval="40s"timeout="40s"\op start timeout="60s"interval="0"\op stop timeout="60s"interval="0"primitive maserver lsb:mysqld \op monitor interval="20s"timeout="20s"on-fail="restart"ms ms_mariadbdrbd mariadbdrbd \meta master-max="1"master-node-max="1"clone-max="2"clone-node-max="1"notify="true"colocation madbvip_with_maserver inf: madbvip maservercolocation mariadbstore_with_ms_mariadbdrbd inf: mariadbstore ms_mariadbdrbd:Mastercolocation maserver_with_mariadbstore inf: maserver mariadbstoreorder madbvip_before_masever inf: madbvip maserverorder mariadbstore_before_maserver inf: mariadbstore:start maserver:startorder ms_mariadbrbd_before_mairadbstore inf: ms_mariadbdrbd:promote mariadbstore:startproperty $id="cib-bootstrap-options"\dc-version="1.1.10-14.el6_5.2-368c726"\cluster-infrastructure="classic openais (with plugin)"\expected-quorum-votes="2"\stonith-enabled="false"\no-quorum-policy="ignore"
九、測試
1.查看運行狀態
[root@node1 ~]# crm statusLast updated: Wed Apr2316:24:112014Last change: Wed Apr2316:21:502014via cibadmin on node1.zero1.comStack: classic openais (withplugin)Current DC: node1.zero1.com - partitionwithquorumVersion:1.1.10-14.el6_5.2-368c7262Nodes configured,2expected votes5Resources configuredOnline: [ node1.zero1.com node2.zero1.com ]Master/Slave Set: ms_mariadbdrbd [mariadbdrbd]Masters: [ node1.zero1.com ]Slaves: [ node2.zero1.com ]mariadbstore (ocf::heartbeat:Filesystem): Started node1.zero1.commadbvip (ocf::heartbeat:IPaddr2): Started node1.zero1.commaserver (lsb:mysqld): Started node1.zero1.com
2.手動切換節點
[root@node1 ~]# crm node standby node1.zero1.com[root@node1 ~]# crm statusLast updated: Wed Apr2316:26:052014Last change: Wed Apr2316:25:342014via crm_attribute on node1.zero1.comStack: classic openais (withplugin)Current DC: node1.zero1.com - partitionwithquorumVersion:1.1.10-14.el6_5.2-368c7262Nodes configured,2expected votes5Resources configuredNode node1.zero1.com: standbyOnline: [ node2.zero1.com ]Master/Slave Set: ms_mariadbdrbd [mariadbdrbd]Masters: [ node2.zero1.com ]Stopped: [ node1.zero1.com ]mariadbstore (ocf::heartbeat:Filesystem): Started node2.zero1.commadbvip (ocf::heartbeat:IPaddr2): Started node2.zero1.commaserver (lsb:mysqld): Started node2.zero1.com
這樣所有的資源都在node2上運行了。