四、測試
調試工具：
cman_tool、clustat、clusvadm、init 0
測試目的：
1.實現資源轉移
2.探究quorum機制和fence機制
測試思路：
不開啓qdiskd服務，查看quorum票數，逐個節點關機/模擬宕機，看日誌、看quorum；
開啓qdiskd服務，查看quorum票數，逐個節點關機/模擬宕機，看日誌、看quorum；
階段一：（移除非資源節點）
所有節點處於正常狀態，資源運行在web3節點，沒有開啓qdisk
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 21:44:07 2014
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
web1.rocker.com 1 Online, Local,rgmanager
web2.rocker.com 2 Online, rgmanager
web3.rocker.com 3 Online, rgmanager
web4.rocker.com 4 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web3.rocker.com started
[root@web1 ~]# cman_tool status
……
Nodes: 4
Expected votes: 8
Total votes: 8
Node votes: 2
Quorum: 5
#可見Quorum=(Total votes/Nodes)+1，資料說一旦小於這個值，cluster就會掛起，我們試一下是不是這樣
……
=====Step1：對眀攀戀2節點進行關機操作=====
[root@web1 ~]# ssh root@web2 'halt'
查看日誌
[root@web1 ~]# tail /var/log/message
Sep 22 21:49:59 web1 clurgmgrd[4807]: <notice> Member 2 shuttingdown
Sep 22 21:50:17 web1 openais[4555]: [TOTEM] The token was lost in theOPERATIONAL state.
Sep 22 21:50:17 web1 openais[4555]: [TOTEM] Receive multicast socketrecv buffer size (320000 bytes).
Sep 22 21:50:17 web1 openais[4555]: [TOTEM] Transmit multicast socketsend buffer size (221184 bytes).
Sep 22 21:50:17 web1 openais[4555]: [TOTEM] entering GATHER state from2.
Sep 22 21:50:29 web1 openais[4555]: [TOTEM] entering GATHER state from0.
Sep 22 21:50:29 web1 openais[4555]: [TOTEM] Creating commit tokenbecause I am the rep.
Sep 22 21:50:29 web1 openais[4555]: [TOTEM] Storing new sequence id forring 14
Sep 22 21:50:29 web1 openais[4555]: [TOTEM] entering COMMIT state.
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] entering RECOVERY state.
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] position [0] member192.168.1.201:
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] previous ring seq 16 rep192.168.1.201
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] aru bc high delivered bcreceived flag 1
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] position [1] member192.168.1.203:
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] previous ring seq 16 rep192.168.1.201
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] aru bc high delivered bcreceived flag 1
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] position [2] member192.168.1.204:
Sep 22 21:50:30 web1 openais[4555]: [TOTEM] previous ring seq 16 rep192.168.1.201
Sep 22 21:50:31 web1 openais[4555]: [TOTEM] aru bc high delivered bcreceived flag 1
Sep 22 21:50:31 web1 openais[4555]: [TOTEM] Did not need to originateany messages in recovery.
Sep 22 21:50:31 web1 openais[4555]: [TOTEM] Sending initial ORF token
Sep 22 21:50:31 web1 last message repeated 2 times
Sep 22 21:50:32 web1 openais[4555]: [CLM ] CLM CONFIGURATION CHANGE
Sep 22 21:50:32 web1 openais[4555]: [CLM ] New Configuration:
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.201)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.203)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.204)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] Members Left:
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.202)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] Members Joined:
Sep 22 21:50:32 web1 openais[4555]: [CLM ] CLM CONFIGURATION CHANGE
Sep 22 21:50:32 web1 openais[4555]: [CLM ] New Configuration:
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.201)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.203)
Sep 22 21:50:32 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.204)
Sep 22 21:50:32 web1 kernel: dlm: closing connection to node 2
Sep 22 21:50:32 web1 openais[4555]: [CLM ] Members Left:
Sep 22 21:50:32 web1 openais[4555]: [CLM ] Members Joined:
Sep 22 21:50:32 web1 openais[4555]: [SYNC ] This node is within theprimary component and will provide service.
Sep 22 21:50:32 web1 openais[4555]: [TOTEM] entering OPERATIONAL state.
Sep 22 21:50:32 web1openais[4555]: [CLM ] got nodejoinmessage 192.168.1.201
Sep 22 21:50:32 web1openais[4555]: [CLM ] got nodejoinmessage 192.168.1.203
Sep 22 21:50:32 web1openais[4555]: [CLM ] got nodejoinmessage 192.168.1.204
Sep 22 21:50:32 web1openais[4555]: [CPG ] got joinlistmessage from node 4
Sep 22 21:50:32 web1openais[4555]: [CPG ] got joinlistmessage from node 1
Sep 22 21:50:32 web1openais[4555]: [CPG ] got joinlistmessage from node 3
#表示重新定義成員
查看成員
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 21:50:54 2014
Member Status: Quorate #表示集羣可用狀態
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Online,Local, rgmanager
web2.rocker.com 2 Offline
web3.rocker.com 3 Online,rgmanager
web4.rocker.com 4 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web3.rocker.com started
查看quorum
[root@web1 ~]# cman_tool status
……
Nodes: 3
Expected votes: 8
Total votes: 6
Node votes: 2
Quorum: 4
……
#對比關機前的quorum，Total votes=8，Quorum=5；可見，現在Total vote=6，少了web2的2票，但是quorum是重新計算的，就是說quorum是浮動的。
=====Step2：對web4節點進行關機操作=====
[root@web1 ~]# ssh root@web4 'halt'
查看節點狀態
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 21:55:26 2014
Member Status: Quorate #表示集羣可用狀態
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Online,Local, rgmanager
web2.rocker.com 2 Offline
web3.rocker.com 3 Online, rgmanager
web4.rocker.com 4 Offline
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web3.rocker.com started
查看quorum
[root@web1 ~]# cman_tool status
……
Nodes: 2
Expected votes: 8
Total votes: 4
Node votes: 2
Quorum: 3
……
#看，quorum在浮動，也就是說，只剩下一臺也不會讓整個集羣掛掉。注意，前提是節點正常關機的情況下。
*****************************************************************
階段二：這次要關機佔用資源的節點
先恢復web2和web4，開機然後開啓服務即可
日誌：
Sep 22 22:03:17 web1 kernel: dlm: got connection from 2
Sep 22 22:03:20 web1 kernel: dlm: got connection from 4
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 22:03:50 2014
Member Status: Quorate
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Online, Local, rgmanager
web2.rocker.com 2 Online,rgmanager
web3.rocker.com 3 Online,rgmanager
web4.rocker.com 4 Online,rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web3.rocker.com started
[root@web1 ~]# cman_tool status
……
Nodes: 4
Expected votes: 8
Total votes: 8
Node votes: 2
Quorum: 5
……
===Step1：現在把佔用資源的web3關機，看看集羣是怎麼樣運作的===
[root@web1 ~]# ssh root@web3 'halt'
[root@web1 ~]# tail /var/log/message
Sep 22 22:05:31 web1 clurgmgrd[4807]: <notice> Member 3 shuttingdown
Sep 22 22:06:04 web1 openais[4555]: [TOTEM] The token was lost in theOPERATIONAL state.
Sep 22 22:06:04 web1 openais[4555]: [TOTEM] Receive multicast socketrecv buffer size (320000 bytes).
Sep 22 22:06:04 web1 openais[4555]: [TOTEM] Transmit multicast socketsend buffer size (221184 bytes).
Sep 22 22:06:04 web1 openais[4555]: [TOTEM] entering GATHER state from2.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] entering GATHER state from11.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] Creating commit tokenbecause I am the rep.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] Storing new sequence id forring 24
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] entering COMMIT state.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] entering RECOVERY state.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] position [0] member192.168.1.201:
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] previous ring seq 32 rep192.168.1.201
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] aru 8d high delivered 8dreceived flag 1
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] position [1] member192.168.1.202:
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] previous ring seq 32 rep192.168.1.201
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] aru 8d high delivered 8dreceived flag 1
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] position [2] member192.168.1.204:
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] previous ring seq 32 rep192.168.1.201
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] aru 8d high delivered 8dreceived flag 1
Sep 22 22:06:16 web1 kernel: dlm: closing connection to node 3
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] Did not need to originateany messages in recovery.
Sep 22 22:06:16 web1 openais[4555]: [TOTEM] Sending initial ORF token
Sep 22 22:06:17 web1 openais[4555]: [CLM ] CLM CONFIGURATION CHANGE
Sep 22 22:06:17 web1 openais[4555]: [CLM ] New Configuration:
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.201)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.202)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.204)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] Members Left:
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.203)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] Members Joined:
Sep 22 22:06:17 web1 openais[4555]: [CLM ] CLM CONFIGURATION CHANGE
Sep 22 22:06:17 web1 openais[4555]: [CLM ] New Configuration:
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.201)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.202)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] r(0)ip(192.168.1.204)
Sep 22 22:06:17 web1 openais[4555]: [CLM ] Members Left:
Sep 22 22:06:17 web1 openais[4555]: [CLM ] Members Joined:
Sep 22 22:06:17 web1 openais[4555]: [SYNC ] This node is within theprimary component and will provide service.
Sep 22 22:06:17 web1 openais[4555]: [TOTEM] entering OPERATIONAL state.
Sep 22 22:06:17 web1 openais[4555]: [CLM ] got nodejoin message 192.168.1.201
Sep 22 22:06:17 web1 openais[4555]: [CLM ] got nodejoin message 192.168.1.202
Sep 22 22:06:17 web1 openais[4555]: [CLM ] got nodejoin message 192.168.1.204
Sep 22 22:06:17 web1 openais[4555]: [CPG ] got joinlist message from node 4
Sep 22 22:06:18 web1 openais[4555]: [CPG ] got joinlist message from node 1
Sep 22 22:06:18 web1 openais[4555]: [CPG ] got joinlist message from node 2
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 22:06:37 2014
Member Status: Quorate #集羣可用
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Online,Local, rgmanager
web2.rocker.com 2 Online, rgmanager
web3.rocker.com 3 Offline
web4.rocker.com 4 Online,rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web2.rocker.com started
#集羣資源轉移到了web2節點上了，測試一下。
查看quorum
[root@web1 ~]# cman_tool status
……
Nodes: 3
Expected votes: 8
Total votes: 6
Node votes: 2
Quorum: 4
……
#對比之前的quorum，這裏的quorum也發生了浮動。
=====step2：把web2關機======
[root@web1 ~]# ssh root@web2 'halt'
[root@web1 ~]# tail /var/log/message
Sep 22 22:09:30 web1 clurgmgrd[4807]: <notice> Member 2 shutting down
Sep 22 22:09:30 web1 clurgmgrd[4807]: <notice> Starting stopped service service:myservice
Sep 22 22:09:33 web1 avahi-daemon[3919]: Registering new address record for 192.168.1.200 on eth0. #資源轉到web1節點了
Sep 22 22:09:34 web1clurgmgrd[4807]: <notice> Service service:myservice started
……
Sep 22 22:10:15 web1 openais[4555]: [CLM ] got nodejoin message 192.168.1.201
Sep 22 22:10:15 web1 openais[4555]: [CLM ] got nodejoin message 192.168.1.204
Sep 22 22:10:15 web1 openais[4555]: [CPG ] got joinlist message from node 4
Sep 22 22:10:15 web1 openais[4555]: [CPG ] got joinlist message from node 1
查看節點狀態
[root@web1 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 22:11:25 2014
Member Status: Quorate
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Online,Local, rgmanager
web2.rocker.com 2 Offline
web3.rocker.com 3 Offline
web4.rocker.com 4 Online,rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web1.rocker.com started
測試
=====step3：把web1節點關機=====
[root@web4 ~]# ssh root@web1 'halt'
[root@web4 ~]# clustat
Cluster Status for mycluster @ Mon Sep 22 22:31:02 2014
Member Status: Quorate #集羣依然可用
Member Name ID Status
------ ---- ----------
web1.rocker.com 1 Offline
web2.rocker.com 2 Offline
web3.rocker.com 3 Offline
web4.rocker.com 4 Online,Local, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:myservice web4.rocker.com started
測試
###字數限制####
###更多見下文###