author：skate
tiome：2013/12/06
lvs+ldirectord+pacemaker+corosync+mysql實現高可用負載均衡
環境說明
vip：192.168.213.97
rip1：192.168.213.91
rip2：192.168.213.92
client：192.168.213.93
os：centos6.3
kernel：2.6.32-279.el6.x86_64
mysql version：5.5.28
1. ldirectord安裝與配置
ldirectord下載
下載地址：http://horms.net/projects/ldirectord/download.shtml
# unzip resource-agents-master.zip
# cd resource-agents-master
# ./autogen.sh
# ./configure
# make && make install
此時ldirectord還是不能使用，比如查看狀態，如下：
[root@localhost resource-agents-master]# service ldirectord status
Can't locate Socket6.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/sbin/ldirectord line 838.
BEGIN failed--compilation aborted at /usr/sbin/ldirectord line 838.
[root@localhost resource-agents-master]#
需要安裝一些perl的包
[root@localhost resource-agents-master]# yum install perl-Socket6 perl-IO-Socket-INET6 perl-Email-Date-Format perl-TimeDate perl-Pod-Escapes perl-Pod-Simple perl-Test-Pod perl-MailTools perl-libwww-perl
....
....
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
perl-IO-Socket-INET6 noarch 2.56-4.el6 base 17 k
perl-Socket6 x86_64 0.23-4.el6 base 27 k
perl-Test-Pod noarch 1.40-1.el6 base 14 k
Updating:
perl-Pod-Escapes x86_64 1:1.04-136.el6 base 32 k
perl-Pod-Simple x86_64 1:3.13-136.el6 base 212 k
Installing for dependencies:
db4-cxx x86_64 4.7.25-18.el6_4 base 588 k
db4-devel x86_64 4.7.25-18.el6_4 base 6.6 M
gdbm-devel x86_64 1.8.0-36.el6 base 25 k
perl-CGI x86_64 3.51-136.el6 base 209 k
Updating for dependencies:
db4 x86_64 4.7.25-18.el6_4 base 563 k
db4-utils x86_64 4.7.25-18.el6_4 base 130 k
perl x86_64 4:5.10.1-136.el6 base 10 M
perl-Archive-Extract x86_64 1:0.38-136.el6 base 40 k
perl-Archive-Tar x86_64 1.58-136.el6 base 73 k
perl-CPAN x86_64 1.9402-136.el6 base 246 k
perl-CPANPLUS x86_64 0.88-136.el6 base 307 k
perl-Compress-Raw-Bzip2 x86_64 2.021-136.el6 base 45 k
perl-Compress-Raw-Zlib x86_64 1:2.021-136.el6 base 69 k
perl-Compress-Zlib x86_64 2.021-136.el6 base 45 k
perl-Digest-SHA x86_64 1:5.47-136.el6 base 64 k
perl-ExtUtils-CBuilder x86_64 1:0.27-136.el6 base 48 k
perl-ExtUtils-Embed x86_64 1.28-136.el6 base 31 k
perl-ExtUtils-MakeMaker x86_64 6.55-136.el6 base 293 k
perl-ExtUtils-ParseXS x86_64 1:2.2003.0-136.el6 base 45 k
perl-File-Fetch x86_64 0.26-136.el6 base 41 k
perl-IO-Compress-Base x86_64 2.021-136.el6 base 69 k
perl-IO-Compress-Bzip2 x86_64 2.021-136.el6 base 48 k
perl-IO-Compress-Zlib x86_64 2.021-136.el6 base 135 k
perl-IO-Zlib x86_64 1:1.09-136.el6 base 33 k
perl-IPC-Cmd x86_64 1:0.56-136.el6 base 46 k
perl-Locale-Maketext-Simple x86_64 1:0.18-136.el6 base 31 k
perl-Log-Message x86_64 1:0.02-136.el6 base 46 k
perl-Log-Message-Simple x86_64 0.04-136.el6 base 28 k
perl-Module-Build x86_64 1:0.3500-136.el6 base 229 k
perl-Module-CoreList x86_64 2.18-136.el6 base 65 k
perl-Module-Load x86_64 1:0.16-136.el6 base 28 k
perl-Module-Load-Conditional x86_64 0.30-136.el6 base 34 k
perl-Module-Loaded x86_64 1:0.02-136.el6 base 27 k
perl-Module-Pluggable x86_64 1:3.90-136.el6 base 40 k
perl-Object-Accessor x86_64 1:0.34-136.el6 base 37 k
perl-Package-Constants x86_64 1:0.02-136.el6 base 26 k
perl-Params-Check x86_64 1:0.26-136.el6 base 35 k
perl-Parse-CPAN-Meta x86_64 1:1.40-136.el6 base 29 k
perl-Term-UI x86_64 0.20-136.el6 base 39 k
perl-Test-Harness x86_64 3.17-136.el6 base 231 k
perl-Test-Simple x86_64 0.92-136.el6 base 112 k
perl-Time-HiRes x86_64 4:1.9721-136.el6 base 48 k
perl-Time-Piece x86_64 1.15-136.el6 base 46 k
perl-core x86_64 5.10.1-136.el6 base 23 k
perl-devel x86_64 4:5.10.1-136.el6 base 423 k
perl-libs x86_64 4:5.10.1-136.el6 base 578 k
perl-parent x86_64 1:0.221-136.el6 base 27 k
perl-suidperl x86_64 4:5.10.1-136.el6 base 50 k
perl-version x86_64 3:0.77-136.el6 base 51 k
zlib x86_64 1.2.3-29.el6 base 73 k
zlib-devel x86_64 1.2.3-29.el6 base 44 k
Transaction Summary
====================================================================================================================================
Install 7 Package(s)
Upgrade 49 Package(s)
Total download size: 23 M
....
....
繼續看看ldirectord是否可用
[root@localhost resource-agents-master]# service ldirectord status
Can not find ipvsadm at /usr/sbin/ldirectord line 887.
[root@localhost resource-agents-master]#
發現沒有安裝ipvsadm，安裝如下：
# yum install ipvsadm
....
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
ipvsadm x86_64 1.26-2.el6 base 41 k
Transaction Summary
====================================================================================================================================
Install 1 Package(s)
Total download size: 41 k
....
....
[root@localhost resource-agents-master]# service ldirectord status
Config file ldirectord.cf not found
想查看配置文件放在何處，可以查看文件“/etc/init.d/ldirectord”和“/usr/sbin/ldirectord”；默認是放在“/etc/ha.d/ldirectord.cf”。把在安裝目錄的樣本配置複製到“/etc/ha.d/”下，然後再修改。這時ldirectord服務纔可以正常工作
這回可用了
[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord start
Starting ldirectord... success
[root@localhost resource-agents-master]# service ldirectord status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 28911
[root@localhost resource-agents-master]#
安裝參考：
http://blog.chinaunix.net/uid-17268883-id-3145428.html
2. lvs安裝
LVS軟件包括二部分：
A.IPVS模塊，LVS已經是Linux標準內核的一部分,直接被編譯在內核中!
B.IPVS管理工具ipvsadm
檢測IPVS模塊是否真的編譯到內核中去了
[root@master mysqlinstall]# modprobe -l |grep ipvs
kernel/net/netfilter/ipvs/ip_vs.ko
kernel/net/netfilter/ipvs/ip_vs_rr.ko
kernel/net/netfilter/ipvs/ip_vs_wrr.ko
kernel/net/netfilter/ipvs/ip_vs_lc.ko
kernel/net/netfilter/ipvs/ip_vs_wlc.ko
kernel/net/netfilter/ipvs/ip_vs_lblc.ko
kernel/net/netfilter/ipvs/ip_vs_lblcr.ko
kernel/net/netfilter/ipvs/ip_vs_dh.ko
kernel/net/netfilter/ipvs/ip_vs_sh.ko
kernel/net/netfilter/ipvs/ip_vs_sed.ko
kernel/net/netfilter/ipvs/ip_vs_nq.ko
kernel/net/netfilter/ipvs/ip_vs_ftp.ko
ipvsadm管理工具安裝上面介紹了，通過yum安裝比較方便，可以解決很多依賴關係，如下：
# yum install ipvsadm
安裝參考：
http://beyondhdf.blog.51cto.com/229452/1331874
http://soulboy.blog.51cto.com/4007306/1299896
http://zh.linuxvirtualserver.org/node/98
http://itnihao.blog.51cto.com/1741976/744237
Directorer的lvsdr腳本
[root@localhost mysqlinstall]# more lvsdr.sh
###############################################################
#Director script:
################################################################
#!/bin/bash
#
# LVS script for VS/DR
#
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise director of the Linux Virtual Server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator:
# zxg: repair bug
#
#
# Source function library
. /etc/rc.d/init.d/functions
#
VIP=192.168.213.97
RIP1=192.168.213.91
RIP2=192.168.213.92
ETHx=eth0:3
PORT=3306
#
start() {
/sbin/ifconfig $ETHx $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev $ETHx
# Since this is the Director we must be able to forward packets
echo 1 > /proc/sys/net/ipv4/ip_forward
# Clear all iptables rules.
/sbin/iptables -F
# Reset iptables counters.
/sbin/iptables -Z
# Clear all ipvsadm rules/services.
/sbin/ipvsadm -C
# Add an IP virtual service for VIP 192.168.0.219 port $PORT
# In this recipe, we will use the round-robin scheduling method.
# In production, however, you should use a weighted, dynamic scheduling method.
/sbin/ipvsadm -A -t $VIP:$PORT -s wrr
# Now direct packets for this VIP to
# the real server IP (RIP) inside the cluster
/sbin/ipvsadm -a -t $VIP:$PORT -r $RIP1 -g -w 1
/sbin/ipvsadm -a -t $VIP:$PORT -r $RIP2 -g -w 3
/bin/touch /var/lock/subsys/ipvsadm &> /dev/null
}
stop() {
# Stop forwarding packets
echo 0 > /proc/sys/net/ipv4/ip_forward
# Reset ipvsadm
/sbin/ipvsadm -C
# Bring down the VIP interface
/sbin/ifconfig $ETHx down
/sbin/route del $VIP
/bin/rm -f /var/lock/subsys/ipvsadm
echo "ipvs stopped..."
}
status() {
if [ ! -e /var/lock/subsys/ipvsadm ]; then
echo "ipvsadm stopped ..."
else
echo "ipvs is running ..."
ipvsadm -L -n
fi
}
restart() {
stop
start
}
case "$1" in
start)
[ -f "/var/lock/subsys/ipvsadm" ] && exit 0
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage: $0 {start|stop|status|restart}"
;;
esac
realserver的運行腳本
[root@master mysqlinstall]# more lvsrl.sh
#############################################################
#RealServer腳本:
##############################################################
#!/bin/bash
#
# Script to start LVS DR real server.
# description: LVS DR real server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator:
# zxg: repair bug
#
#
. /etc/rc.d/init.d/functions
VIP=192.168.213.97
host=`/bin/hostname`
case "$1" in
start)
# Start LVS-DR real server on this machine.
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
;;
stop)
# Stop LVS-DR real server loopback device(s).
/sbin/ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
# Status of LVS-DR real server.
islothere=`/sbin/ifconfig lo:0 | grep $VIP`
isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
if [ ! "$islothere" -o ! "isrothere" ];then
# Either the route or the lo:0 device
# not found.
echo "LVS-DR real server Stopped."
else
echo "LVS-DR real server Running."
fi
;;
*)
# Invalid entry.
echo "$0: Usage: $0 {start|status|stop}"
exit 1
;;
esac
先把腳本lvsrl.sh部署到所有的realserver上，並啓動
然後把腳本lvsdr.sh部署到director上，並啓動
可以用如下兩個命令查看lvs的運行狀態
# ip addr show dev eth0 //查看ip的漂移情況
# ipvsadm -ln //查看轉發配置表
例如：
[root@localhost mysqlinstall]# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:b8:7a:78 brd ff:ff:ff:ff:ff:ff
inet 192.168.213.90/24 brd 192.168.213.255 scope global eth0
inet 192.168.213.97/32 brd 192.168.213.97 scope global eth0:3
inet6 fe80::250:56ff:feb8:7a78/64 scope link
valid_lft forever preferred_lft forever
[root@localhost mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr persistent 2
-> 192.168.213.91:3306 Route 1 0 0
-> 192.168.213.92:3306 Route 1 0 40
[root@localhost mysqlinstall]#
在客戶端寫個循環連接的腳本
[root@slave93 mysqlinstall]# more load_mysql.sh
while (true)
do
mysql -uroot -proot -h192.168.213.97 -e "select count(*) from test.a;"
sleep 3
done
在客戶端執行腳本
[root@slave93 mysqlinstall]# sh load_mysql.sh
在director上查看連接情況
# ipvsadm -lcn
上面的lvs是採用的持久連接的方式，輪詢的均衡算法；對於使用數據庫大都會使用持久連接（300s），可以採用wrr的均衡算法
上面的director是獨立的主機，爲了節省成本，可以把director和某一臺realserver部署同一臺上，比如部署到91上
[root@master mysqlinstall]# sh lvsdr.sh status
ipvs is running ...
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 wrr
-> 192.168.213.91:3306 Local 1 0 0
-> 192.168.213.92:3306 Route 3 0 0
到現在，lvs可以正常工作，可以實現負載均衡，但後端某個realserver有問題時，lvs把請求分配器上，就會有問題，這時就需要配置
前面我們安裝的ldirectord服務，其可以自動監控後端realserver的健康狀態，發現有問題時就更新director的虛擬服務器和RS鏈表。即自動從集羣中摘除，待以後故障恢復，ldirectord服務會自動把其加入集羣中
按如下修改ldirectord的配置文件
[root@master mysqlinstall]# more /etc/ha.d/ldirectord.cf
#
# Sample ldirectord configuration file to configure various virtual services.
#
# Ldirectord will connect to each real server once per second and request
# /index.html. If the data returned by the server does not contain the
# string "Test Message" then the test fails and the real server will be
# taken out of the available pool. The real server will be added back into
# the pool once the test succeeds. If all real servers are removed from the
# pool then localhost:80 is added to the pool as a fallback measure.
# Global Directives
checktimeout=3 //超時
checkinterval=1 //check的間隔時間
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes //實時讀取配置文件
#logfile="/var/log/ldirectord.log" //定義日誌文件位置
#logfile="local0"
#emailalert="[email protected]"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no //真正的把故障realserver從lvs映射表中刪除，以保證集羣對外的不間斷服務
#Sample configuration for a MySQL virtual service.
virtual = 192.168.213.97:3306
#real=master->slave92:3306 gate 10
real=192.168.213.91 gate 10
real=192.168.213.92 gate 10
fallback=127.0.0.1:3306
service=mysql
scheduler=wrr
#persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
login="root"
passwd="root"
database="test"
request="SELECT * FROM a"
然後重啓ldirectord服務
[root@master mysqlinstall]# service ldirectord restart
[root@master mysqlinstall]# service ldirectord status
然後查看log看ldirectord的工作狀況
[root@master mysqlinstall]# tail -f /var/log/ldirectord.log
測試ldirectord是否可以對realserver做健康檢查，爲了便於測試，我把均衡算法改爲rr，並去掉持久連接
測試前：
[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr
-> 192.168.213.91:3306 Local 10 0 59
-> 192.168.213.92:3306 Route 10 0 59
[root@master mysqlinstall]#
啓動客戶端測試腳本，每秒執行一次數據庫查詢操作
# sh load_mysql.sh
停止92的mysql
查看結果
[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr
-> 192.168.213.91:3306 Local 10 0 62
[root@master mysqlinstall]#
查看ldirectord的log
[root@master mysqlinstall]# tail -f /var/log/ldirectord.log
[Sat Dec 7 01:52:22 2013|ldirectord.cf|7445] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)
[Sat Dec 7 01:52:22 2013|ldirectord|26081] system(/sbin/ipvsadm -a -t 192.168.213.97:3306 -r 192.168.213.92:0 -g -w 10) failed:
[Sat Dec 7 01:52:22 2013|ldirectord|26081] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)
[Sat Dec 7 02:02:53 2013|ldirectord.cf|7445] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)
[Sat Dec 7 02:02:53 2013|ldirectord|26081] system(/sbin/ipvsadm -d -t 192.168.213.97:3306 -r 192.168.213.92:3306) failed:
[Sat Dec 7 02:02:53 2013|ldirectord|26081] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)
前端的服務有短暫1s的影響
.....
+----------+
+----------+
| count(*) |
+----------+
| 2 |
+----------+
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.213.97' (111)
+----------+
| count(*) |
+----------+
| 2 |
+----------+
......
到目前爲止，lvs可以負載均衡前端的請求，並可以通過ldirectord服務實現對後端的realserver的健康檢查，但是如過lvs和ldirectord所在的機器故障怎麼辦呢，就沒辦法高可用了，這時就需要corosync和pacemaker來做HA。
會繼續下面會繼續........
-------續---------
