LVS+Keepalived實現負載均衡試驗筆記

 試驗環境

192.168.108.108    Master-LVS-Director
192.168.108.109    Backup-LVS-Director
192.168.108.180     VIP
192.168.108.161    RealServer1
192.168.108.162    RealServer2

拓撲圖

安裝ipvsadm
# yum install -y ipvsadm

安裝keepalived

確認當前運行的內核
# name -r
2.6.18-128.4.1.el5xen
# ls -1 /usr/src/kernels
2.6.18-128.4.1.el5-x86_64
2.6.18-128.el5-x86_64

# wget http://www.keepalived.org/software/keepalived-1.1.17.tar.gz
# tar -xvzf keepalived-1.1.17.tar.gz
# cd keepalived-1.1.17
# ./configure --sysconfdir=/etc/ --sbindir=/usr/sbin/ --with-kernel-dir=/usr/src/kernels/2.6.18-128.4.1.el5-x86_64
Keepalived configuration
------------------------
Keepalived version: 1.1.17
Compiler: gcc
Compiler flags: -g -O2
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework : Yes
Use LinkWatch: No
Use Debug flags: No
# make && make install

配置keepalived  
# vi /etc/keepalived/keepalived.conf
下載 keepalived.conf

! Configuration File for keepalived

# 全局定義
global_defs {
notification_email {
[e-mail]
}
notification_email_from [e-mail]

#smtp主機地址
smtp_server 127.0.0.1
smtp_connect_timeout 30

#運行Keepalived服務器的一個標識。發郵件時顯示在郵件標題中的信息
router_id LVS_MASTER
}

#VIP
vrrp_instance VI_1 {

#指定實例的初始狀態(角色)。在兩臺router都啓動時馬上會根據priority的高低開始競選
#高priority爲Master
state MASTER

#VT_1 實例綁定的網卡
interface eth0

#VRID 標記(0-255)
virtual_router_id 51

#優先級,BACKUP的值一定要低於MASTER
priority 100

#檢查間隔
advert_int 1

#設置認證
authentication {
#認證類型
auth_type PASS
#認證密碼
auth_pass 1111
}

#VIP 這個IP在發生MASTER 到 BACKUP切換時會隨之add或del，所以每臺服務器上可以不綁定
#虛擬地址，而都放入virtual_ipaddress塊中(可以多個)，keepalived會自動使用ip地址進
#行綁定(不需要依賴ifcfg-eth0)，利用ip add show eth0可以看到加入的VIP
virtual_ipaddress {
192.168.108.180
}
}

#定義virtual_server (HTTP | 80)
virtual_server 192.168.108.180 80 {
delay_loop 6            # service polling的delay時間
lb_algo wlc             # 調度算法
lb_kind DR              # LVS工作方式
persistence_timeout 50  # 會話保持時間
protocol TCP            # 協議類型(TCP|UDP)

#定義rs1，每一個rs都需要下面的一個配置段
real_server 192.168.108.161 80 {
weight 1            # 權值 默認1，0爲失效
# inhibit_on_failure    # 在服務器健康檢查失敗後不從IPVS中刪除而將其權值標記爲0

# TCP方式的健康檢查
TCP_CHECK {
connect_timeout 10      # 連接超時時間
nb_get_retry 3          # 重試次數
delay_before_retry 3    # 重試間隔
connect_port 80         # 健康檢查端口
}
}

# 定義rs2
real_server 192.168.108.162 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}

# 定義virtual_server (HTTPS | 443)
virtual_server 192.168.108.180 443 {
delay_loop 6
lb_algo wlc
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP

real_server 192.168.108.161 443 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}

real_server 192.168.108.162 443 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
}

配置RS
爲了方便起見我自己編寫了一個啓動腳本，如下：
下載 lvsRealServer.sh

^?View Code BASH

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

#!/bin/bash #Description : RealServer Start! #Write by:Cooper #Last Modefiy:2009.08.21   VIP=192.168.108.180 LVS_TYPE=DR   startrs() { echo "start LVS of REALServer"   if [ "$LVS_TYPE" == "DR" ];then /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up /sbin/route add -host $VIP dev lo:0 else /sbin/ifconfig tunl0 $VIP netmask 255.255.255.255 broadcast $VIP up /sbin/route add -host $VIP dev tunl0 fi echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce }   stoprs() { if [ "$LVS_TYPE" == "DR" ];then /sbin/ifconfig lo:0 down echo "close LVS Directorserver" else /sbin/ifconfig tunl0 down echo "close LVS Tunnel server" fi echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce }   # ============ Main ===========   case $1 in "start") startrs;; "stop") stoprs;; "*") echo "Usage $0 {start|stop}" exit 1 esac

該腳本默認啓動LVS/DR模式，通過修改腳本變量可以實現LVS/Tunnel模式的切換。

運行腳本進行RS設置後執行相關的服務

[root@rs-1]# sh lvsRealServer.sh
[root@rs-1]# service httpd restart
rs-2執行相同的操作

Master/Backup LVS啓動keepalived 服務

[root@Master-LVS]# service keepalived start
Backup-LVS 同樣執行如上命令啓動keepalived

查看Master-LVS上eth0接口在啓動keepalived前後變化

啓動keepalived之前
# ip add show eth0
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
link/ether 00:1d:7d:3d:1c:63 brd ff:ff:ff:ff:ff:ff
inet 192.168.108.108/24 brd 192.168.108.255 scope global eth0
inet6 fe80::21d:7dff:fe3d:1c63/64 scope link
valid_lft forever preferred_lft forever

啓動之後
# ip add show eth0
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
link/ether 00:1d:7d:3d:1c:63 brd ff:ff:ff:ff:ff:ff
inet 192.168.108.108/24 brd 192.168.108.255 scope global eth0
inet 192.168.108.180/32 scope global eth0
inet6 fe80::21d:7dff:fe3d:1c63/64 scope link
valid_lft forever preferred_lft forever

查看LVS運行情況

[root@Master-LVS]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.108.180:443 wlc persistent 50
-> 192.168.108.161:443          Route   1      0          0
-> 192.168.108.162:443          Route   1      0          0
TCP  192.168.108.180:80 wlc persistent 50
-> 192.168.108.161:80           Route   1      0          0
-> 192.168.108.162:80           Route   1      0          0

故障測試

RS故障

切換到其中的一臺rs上，如192.168.108.161
# service httpd stop

這時查看Master/Backup LVS上的的日誌輸出
[root@Master-LVS]# tail -f /var/log/message

^?View Code LOG

 

1 2 3 4 5 6 7

Sep  3 11:08:01 experiment Keepalived_healthcheckers: TCP connection to [192.168.108.161:80] failed !!! Sep  3 11:08:01 experiment Keepalived_healthcheckers: Removing service [192.168.108.161:80] from VS [192.168.108.180:80] Sep  3 11:08:01 experiment Keepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected. Sep  3 11:08:01 experiment Keepalived_healthcheckers: TCP connection to [192.168.108.161:443] failed !!! Sep  3 11:08:01 experiment Keepalived_healthcheckers: Removing service [192.168.108.161:443] from VS [192.168.108.180:443] Sep  3 11:08:01 experiment Keepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected. Sep  3 11:08:01 experiment Keepalived_healthcheckers: SMTP alert successfully sent.

[root@Backup-LVS]# tail -f /var/log/message

^?View Code LOG

 

1 2 3 4 5 6 7

Sep  3 11:08:02 localhost Keepalived_healthcheckers: TCP connection to [192.168.108.161:443] failed !!! Sep  3 11:08:02 localhost Keepalived_healthcheckers: Removing service [192.168.108.161:443] from VS [192.168.108.180:443] Sep  3 11:08:02 localhost Keepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected. Sep  3 11:08:02 localhost Keepalived_healthcheckers: TCP connection to [192.168.108.161:80] failed !!! Sep  3 11:08:02 localhost Keepalived_healthcheckers: Removing service [192.168.108.161:80] from VS [192.168.108.180:80] Sep  3 11:08:02 localhost Keepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected. Sep  3 11:08:03 localhost Keepalived_healthcheckers: SMTP alert successfully sent.

通過日誌可以看出Master與Backup幾乎同時感知了RS1服務器已經故障，並且從IPVS中移除故障rs（或者將其權值標記爲0也就是不可用）。並且向指定的郵箱發送郵件，Master和Backup都會發送郵件，其郵件標題會根據router_id的值區分出Master和Backup

Master LVS-Router故障

停止Master-LVS的keepalived服務，人爲造成故障
[root@Master-LVS]# service keepalived stop

這時查看Backup-LVS的log信息
[root@Backup-LVS]# tail -f /var/log/message

^?View Code LOG

 

1 2 3 4 5 6 7

Sep  3 11:23:28 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Sep  3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Sep  3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Sep  3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.108.180 Sep  3 11:23:29 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.108.180 added Sep  3 11:23:29 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.108.180 added Sep  3 11:23:34 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.108.180

由日誌可以看出，Backup-LVS監測到Master-LVS故障後立即將自己的身份切換爲Master然後將VIP設置到自己的eth0端口上併發送ARP廣播。

現在我手動將Master-LVS的keepalived的服務起來，然後再查看Backup-LVS的log信息
[root@Master-LVS]# service keepalived start

[root@Backup-LVS]# tail -f /var/log/message

^?View Code LOG

 

1 2 3 4 5

Sep  3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert Sep  3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE Sep  3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs. Sep  3 11:30:44 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.108.180 removed Sep  3 11:30:44 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.108.180 removed

由Backup-LVS的日誌可以看到，其檢測到比自己優先級高的實例後將自己的身份切換成了Backup，然後將VIP從eth0端口移除，併發送心跳給Master-LVS。

piranha與keepalived比較

經過比較得知，piranha的主-備的地位是相同的，也就是說主故障後備就會代替主，經其地位從備切換爲主，而當先前的主恢復正常後則先前的主便成了備，其不會主動切換自己的身份爲主，當前的備可以檢測到先前的主已經恢復但並不會主動將自己的身份修改爲備。

而keepalived則是主備分明的，其利用優先級的設置可以嚴格的制定主備身份。

參考文章

LVS-HOWTO
http://www.keepalived.org/documentation.html
http://bbs.linuxtone.org/thread-1077-1-1.html

資源下載
Keepalived-UserGuide    CN EN

 

 

出自：salogs.com