針對LVS的Director的高可用集羣測試
實驗目的:
1. 測試 Director的高可用集羣
2. 觀察heartbeat-ldirectord 對後端 Real Server 健康狀況的檢測
實驗環境:
Redhat 5.8
VIP 192.168.0.2
Real Server:
RS1 192.168.0.5
RS2 192.168.0.6
Director:
node1 192.168.0.11
node2 192.168.0.12
需要用到的rpm包:
heartbeat-2.1.4-9.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
heartbeat-gui-2.1.4-9.el5.i386.rpm
libnet-1.1.4-3.el5.i386.rpm
heartbeat-ldirectord-2.1.4-9.el5.i386.rpm
perl-MailTools-1.77-1.el5.noarch.rpm
heartbeat-pils-2.1.4-10.el5.i386.rpm
另外還要準備好系統光盤,作爲yum源
下面來說具體實驗過程:
一. 先配置 Real Server
1. 同步兩臺Real Server的時間
# hwclock -s
2. 安裝 apache
# yum -y install httpd
爲兩臺Real Server提供網頁文件
- [root@RS1 ~]# echo "<h1>Real Server 1<h1>" > /var/www/html/index.html
- [root@RS2 ~]# echo "<h1>Real Server 2<h1>" > /var/www/html/index.html
- [root@RS1 ~]# vi /etc/httpd/conf/httpd.conf
- 更改:ServerName RS1.yue.com
- [root@RS2 ~]# vi /etc/httpd/conf/httpd.conf
- 更改:ServerName RS2.yue.com
# /etc/init.d/httpd start
3. 在RS1上編輯內核的相關參數
- [root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
- [root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
- [root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
- [root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
- [root@RS1 ~]# ifconfig lo:0 192.168.0.2 broadcast 192.168.0.255 netmask 255.255.255.255 up
- [root@RS1 ~]# ifconfig
- eth0 Link encap:Ethernet HWaddr 00:0C:29:7E:8B:C6
- inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0
- UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
- RX packets:2719 errors:0 dropped:0 overruns:0 frame:0
- TX packets:3628 errors:0 dropped:0 overruns:0 carrier:0
- collisions:0 txqueuelen:1000
- RX bytes:200533 (195.8 KiB) TX bytes:644821 (629.7 KiB)
- Interrupt:67 Base address:0x2000
- lo Link encap:Local Loopback
- inet addr:127.0.0.1 Mask:255.0.0.0
- UP LOOPBACK RUNNING MTU:16436 Metric:1
- RX packets:71 errors:0 dropped:0 overruns:0 frame:0
- TX packets:71 errors:0 dropped:0 overruns:0 carrier:0
- collisions:0 txqueuelen:0
- RX bytes:5699 (5.5 KiB) TX bytes:5699 (5.5 KiB)
- lo:0 Link encap:Local Loopback
- inet addr:192.168.0.2 Mask:255.255.255.255
- UP LOOPBACK RUNNING MTU:16436 Metric:1
- [root@RS1 ~]# elinks -dump http://192.168.0.5
- Real Server 1
- [root@RS1 ~]# elinks -dump http://192.168.0.2
- Real Server 1
- [root@RS1 ~]# route add -host 192.168.0.2 dev lo:0
- [root@RS1 ~]# route -n
- Kernel IP routing table
- Destination Gateway Genmask Flags Metric Ref Use Iface
- 192.168.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 lo
- 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
- 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
- 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
設定服務開機自動啓動
- [root@RS1 ~]# chkconfig --add httpd
- [root@RS1 ~]# chkconfig httpd on
- [root@RS1 ~]# chkconfig --list httpd
- httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
4. 在RS2 上做同樣的設置
二、配置Director
節點的主機名要與 "uname -n "的結果一致
1. 先在 node1 上配置
同步時間:
# hwclock -s
主機名解析:
# vi /etc/hosts 添加如下內容
192.168.0.11 node1.yue.com node1
192.168.0.12 node2.yue.com node2
主機名:
# hostname RS1
- [root@node1 ~]# vi /etc/sysconfig/network
- NETWORKING=yes
- NETWORKING_IPV6=no
- HOSTNAME=node1.yue.com
IP地址:
- [root@node1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
- # Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
- DEVICE=eth0
- BOOTPROTO=none
- ONBOOT=yes
- HWADDR=00:0c:29:e7:3d:5c
- IPADDR=192.168.0.11
- GATEWAY=192.168.0.1
- NETAMASK=255.255.255.0
雙機互信:
- [root@node1 ~]# ssh-keygen -t rsa
- Generating public/private rsa key pair.
- Enter file in which to save the key (/root/.ssh/id_rsa): 密碼爲空(直接回車)
- Created directory '/root/.ssh'.
- Enter passphrase (empty for no passphrase): 再次輸入密碼
- Enter same passphrase again:
- Your identification has been saved in /root/.ssh/id_rsa.
- Your public key has been saved in /root/.ssh/id_rsa.pub.
- The key fingerprint is:
- 0f:c8:62:6b:2e:68:4c:8b:ce:0f:25:52:23:93:c7:0a [email protected]
- [root@node1 ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected] 將公鑰傳送到node2(默認在root用戶的家目錄下的.ssh目錄下)
- 15
- The authenticity of host 'node2.yue.com (192.168.0.12)' can't be established.
- RSA key fingerprint is 9d:d9:14:94:81:c2:7b:d5:7b:af:2c:64:58:8f:e3:49.
- Are you sure you want to continue connecting (yes/no)? yes 提示是否接受連接
- [email protected]'s password: 輸入node2的密碼
- Now try logging into the machine, with "ssh '[email protected]'", and check in:
- .ssh/authorized_keys
- to make sure we haven't added extra keys that you weren't expecting.
測試一下效果:
- [root@node1 ~]# ssh node2 'ifconfig' 遠程在node2上執行命令令
- The authenticity of host 'node2 (192.168.0.12)' can't be established.
- RSA key fingerprint is 9d:d9:14:94:81:c2:7b:d5:7b:af:2c:64:58:8f:e3:49.
- Are you sure you want to continue connecting (yes/no)? yes
- Warning: Permanently added 'node2' (RSA) to the list of known hosts.
- eth0 Link encap:Ethernet HWaddr 00:0C:29:D9:75:DF
- inet addr:192.168.0.12 Bcast:192.168.0.255 Mask:255.255.255.0
- UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
- RX packets:629 errors:0 dropped:0 overruns:0 frame:0
- TX packets:528 errors:0 dropped:0 overruns:0 carrier:0
- collisions:0 txqueuelen:1000
- RX bytes:60497 (59.0 KiB) TX bytes:56572 (55.2 KiB)
- Interrupt:67 Base address:0x2000
- lo Link encap:Local Loopback
- inet addr:127.0.0.1 Mask:255.0.0.0
- UP LOOPBACK RUNNING MTU:16436 Metric:1
- RX packets:10 errors:0 dropped:0 overruns:0 frame:0
- TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
- collisions:0 txqueuelen:0
- RX bytes:692 (692.0 b) TX bytes:692 (692.0 b)
2. 在 node2 上做相應的配置
3. 安裝相關軟件包:
- [root@node1 tmp]# ls
- heartbeat-2.1.4-9.el5.i386.rpm
- heartbeat-stonith-2.1.4-10.el5.i386.rpm
- heartbeat-gui-2.1.4-9.el5.i386.rpm
- libnet-1.1.4-3.el5.i386.rpm
- heartbeat-ldirectord-2.1.4-9.el5.i386.rpm
- perl-MailTools-1.77-1.el5.noarch.rpm
- heartbeat-pils-2.1.4-10.el5.i386.rpm
- [root@node1 tmp]# yum --nogpgcheck localinstall *.rpm
- [root@node1 tmp]# chkconfig --list ldirectord
- ldirectord 0:off 1:off 2:off 3:on 4:off 5:on 6:off
- [root@node1 tmp]# chkconfig ldirectord off
- [root@node2 tmp]# yum --nogpgcheck localinstall *.rpm
- [root@node2 tmp]# chkconfig ldirectord off
- [root@node2 tmp]# chkconfig --list ldirectord
- ldirectord 0:off 1:off 2:off 3:off 4:off 5:off 6:off
配置文件:
- [root@node1 tmp]# cp /usr/share/doc/heartbeat-ldirectord-2.1.4/ldirectord.cf /etc/ha.d/
- [root@node1 ~]# cd /usr/share/doc/heartbeat-2.1.4/
- [root@node1 heartbeat-2.1.4]# cp ha.cf authkeys haresources /etc/ha.d/
- [root@node1 heartbeat-2.1.4]# cd /etc/ha.d/
- [root@node1 ha.d]# chmod 600 authkeys 一定要改權限,否則啓動的時候會報錯
(1)
# vi /etc/ha.d/ha.cf
- logfile /var/log/ha-log
- logfacility local0
- keepalive 2 多長時間傳送一次心跳信息
- deadtime 30 多長時間收不到心跳信息,就認爲死亡
- warntime 10 警告時間
- initdead 120 第一次啓動時等待多長時間,認爲死亡,通常就爲deadtime 的2倍
- udpport 694
- bcast eth0 # Linux 廣播方式傳送心跳信息
- auto_failback on 是否自動回收資源
- node node1.yue.com 節點列表.要與/etc/hosts文件中的定義相同
- node node2.yue.com
- ping 192.168.0.1 指向一個IP(通常是離我們最近的網關),檢查自身的網絡連接,以確定對方是否已經死亡
- compression bz2 傳送的信息要壓縮
- compression_threshold 2 壓縮的下限
- 並添加: crm respawn 這一行
(2)
- [root@node1 ha.d]# dd if=/dev/urandom count=1 bs=512 | md5sum
- 1+0 records in
- 1+0 records out
- 512 bytes (512 B) copied, 0.00025866 seconds, 2.0 MB/s
- 4faf8724bc49da78b21fc04ceb7b5bc3 -
# vi /etc/ha.d/authkeys
- #auth 1 使用哪種加密機制,並指定其編號
- #1 crc
- #2 sha1 HI!
- #3 md5 Hello!
- auth 1 使用1這個編號的算法
- 1 sha1 1daea09a52368d9fe65a37163d4ae3ea 1 號算法爲sha1
(3).
# vi /etc/ha.d/ldirectord.cf
- virtual=192.168.0.2:80 Vip
- real=192.168.0.5:80 gate gate: dr模型
- real=192.168.0.6:80 gate
- fallback=127.0.0.1:80 gate 若兩個Real Server都掛掉,是否通過本機給客戶端一個提示信息
- service=http 基於什麼協議檢測後端的Real Server
- request="test.html" 檢測哪個網頁
- receive="Real Server OK" 期望從檢測網頁得到什麼樣內容
- scheduler=rr
- # persistent=600 持久性
- netmask=255.255.255.255 定義廣播域
- protocol=tcp
- checktype=negotiate 檢測的方式,協商
- checkport=80
提供檢測頁面:
- [root@RS1 ~]# vi /var/www/html/test.html
- <h1>Real Server OK<h1>
- [root@RS2 ~]# vi /var/www/html/test.html
- <h1>Real Server OK<h1>
傳送配置文件:
- [root@node1 ha.d]# scp authkeys ha.cf haresources ldirectord.cf node2:/etc/ha.d/
- authkeys 100% 692 0.7KB/s 00:00
- ha.cf 100% 10KB 10.4KB/s 00:00
- haresources 100% 5905 5.8KB/s 00:00
- ldirectord.cf 100% 7689 7.5KB/s 00:00
啓動heartbeat:
啓動是有順序的:必須先在node 1 上啓動,然後在node1 上遠程啓動node2 上的heartbeat;關閉node2的時候必須是在node1遠程進行
- [root@node1 ha.d]# /etc/init.d/heartbeat start 先在node 1 上啓動
- Starting High-Availability services: [ OK ]
- [root@node1 ha.d]# ssh node2 '/etc/init.d/heartbeat start' 在node 1 上遠程啓動node 2 上的heartbeat
- Starting High-Availability services:
- [ OK ]
查看當前集羣節點的工作狀況:
- [root@node1 ~]# crm_mon -1 顯示當前集羣的工作狀況 ,只顯示一次
- ============
- Last updated: Sun Aug 5 09:12:40 2012
- Current DC: node1.yue.com (5d29dca8-514e-441d-8619-8c395db8cb70)
- 2 Nodes configured. 2個節點
- 0 Resources configured. 0個資源
- ============
- Node: node1.yue.com (5d29dca8-514e-441d-8619-8c395db8cb70): online
- Node: node2.yue.com (8f2d3cdb-f19e-493d-85be-f5214f7615ff): online
- [root@node1 ~]# netstat -tnlp 查看5560端口是否已經開啓
- tcp 0 0 0.0.0.0:5560 0.0.0.0:* LISTEN 31039/mgmtd
- [root@node1 ~]# crmadmin --status node1.yue.com 查看狀態
- Status of [email protected]: S_IDLE (ok) 主節點DC
- [root@node1 ~]# crmadmin --status node2.yue.com
- Status of [email protected]: S_NOT_DC (ok)
- [root@node1 ~]# tail -1 /etc/passwd 給hacuster用戶添加密碼
- hacluster:x:101:157:heartbeat user:/var/lib/heartbeat/cores/hacluster:/sbin/nologin
- [root@node1 ~]# passwd hacluster
- Changing password for user hacluster.
- New UNIX password: 輸入密碼
- BAD PASSWORD: it is based on a dictionary word
- Retype new UNIX password: 再輸入一次
- passwd: all authentication tokens updated successfully.
配置資源:
web_ip
web_ldirectord
[root@node1 ~]# hb_gui &
[1] 3535
新建資源組:web_server
在組中新建資源:
創建資源web_ip
創建資源web_ldirectord
啓動資源:
- [root@node1 ~]# crm_mon -1
- ============
- Last updated: Sun Aug 5 10:03:17 2012
- Current DC: node2.yue.com (8f2d3cdb-f19e-493d-85be-f5214f7615ff)
- 2 Nodes configured.
- 1 Resources configured.
- ============
- Node: node1.yue.com (5d29dca8-514e-441d-8619-8c395db8cb70): online
- Node: node2.yue.com (8f2d3cdb-f19e-493d-85be-f5214f7615ff): online
- Resource Group: Web_server
- web_ldirectord (ocf::heartbeat:ldirectord): Started node1.yue.com
- web_ip (o cf::heartbeat:IPaddr2): Started node1.yue.com
- [root@node1 ~]# ip addr show
- 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
- link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
- inet 127.0.0.1/8 scope host lo
- 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
- link/ether 00:0c:29:e7:3d:5c brd ff:ff:ff:ff:ff:ff
- inet 192.168.0.11/24 brd 192.168.0.255 scope global eth0
- inet 192.168.0.2/32 brd 192.168.0.255 scope global eth0
- [root@node1 ~]# ipvsadm -Ln
- IP Virtual Server version 1.2.1 (size=4096)
- Prot LocalAddress:Port Scheduler Flags
- -> RemoteAddress:Port Forward Weight ActiveConn InActConn
- TCP 192.168.0.2:80 rr
- -> 192.168.0.5:80 Route 1 0 3
- -> 192.168.0.6:80 Route 1 0 2
此時可以用瀏覽器測試:http://192.168.0.2 查看頁面是否正常,是否可以負載均衡
讓node1處於Standby狀態,查看資源是否會切換:
- [root@node2 ~]# ipvsadm -Ln
- IP Virtual Server version 1.2.1 (size=4096)
- Prot LocalAddress:Port Scheduler Flags
- -> RemoteAddress:Port Forward Weight ActiveConn InActConn
- TCP 192.168.0.2:80 rr
- -> 192.168.0.5:80 Route 1 0 0
- -> 192.168.0.6:80 Route 1 0 0 vip已經啓用
- [root@node2 ~]# ip addr show
- 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
- link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
- inet 127.0.0.1/8 scope host lo
- 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
- link/ether 00:0c:29:d9:75:df brd ff:ff:ff:ff:ff:ff
- inet 192.168.0.12/24 brd 192.168.0.255 scope global eth0
- inet 192.168.0.2/32 brd 192.168.0.255 scope global eth0 vip啓用
- [root@node2 ~]# crm_mon -1
- ============
- Last updated: Sun Aug 5 10:34:17 2012
- Current DC: node2.yue.com (8f2d3cdb-f19e-493d-85be-f5214f7615ff)
- 2 Nodes configured.
- 1 Resources configured.
- ============
- Node: node1.yue.com (5d29dca8-514e-441d-8619-8c395db8cb70): standby
- Node: node2.yue.com (8f2d3cdb-f19e-493d-85be-f5214f7615ff): online
- Resource Group: Web_server
- web_ldirectord (ocf::heartbeat:ldirectord): Started node2.yue.com
- web_ip (ocf::heartbeat:IPaddr2): Started node2.yue.com
三、觀察heartbeat-ldirectord 對後端 Real Server 健康狀況的檢測
可以停掉一臺Real Server ,然後通過瀏覽器訪問 http://192.168.0.2
通過刷新頁面來觀察heartbeat-ldirectord是否可以檢測到後端Real Server的健康狀況