聲明：本測試文檔只針對未曾使用過lvs+keepalived的技術愛好者，對於文中錯誤希望讀過此文檔的朋友斧正。文檔冗長的原因是包含了很多細節，可能有些並無必要說明的我也做了詳細的解釋。基於以上原因，敬請高人略過此文檔。
未完，待續........
一、測試名稱
lvs/dr模式下keepalived應用的測試
二、測試目的
目前有1臺lvs/dr轉發用戶請求到後端web服務器，轉發通過腳本控制。有兩個問題需要解決，⑴、後端web服務器故障不能提供web服務時lvs/dr仍然向其轉發請求；⑵、當lvs/dr故障時，將完全中斷web服務。故需要使用軟件實現自動增加刪除故障web結點，同時實現lvs/dr的雙機主備機制。
三、測試環境
1、本測試中使用4臺服務器，包括2臺lvs/dr和2臺web/server。具體情況見下表，
|
服務器名稱 |
服務器網絡配置 |
|
|
真實IP地址(RIP) |
|
lvs-dr1 (dr1) |
192.168.1.210 |
|
lvs-dr2 (dr2) |
192.168.1.211 |
|
lvs-web1 (web1) |
192.168.1.216 |
|
lvs-web2 (web2) |
192.168.1.217 |
|
|
虛擬IP地址(VIP) |
|
web VIP1 |
192.168.1.215 |
|
web VIP2 |
192.168.1.220 |
2、當前狀況如下圖所示，
⑴、用戶訪問網站的請求首先到達lvs-dr1並通過腳本設置判斷如何轉發用戶請求。
⑵、通過轉發的用戶請求被lvs-web1和lvs-web2響應後直接返回給客戶端。
但是這樣的方式可能出現如下問題，
⑴、當lvs-dr1故障時，無法再接受用戶請求並將請求轉發給真實的web服務器（即便真實web服務器正常）從而導致整個web服務的癱瘓，也就是lvs控制器存在單點故障問題。
⑵、當lvs-dr1正常時，真實地web服務器如lvs-web1故障。此時lvs-dr1並不知道真實服務器是否在正常提供web服務，所以仍然在向故障的lvs-web1轉發用戶請求。這樣的結果是用戶請求無法被故障web服務器相應，某些用戶可以訪問網站有些則無法訪問。
注：服務器故障包括：服務器宕機、web服務終止、網線鬆動等等。
3、基於以上的問題，我們需要想辦法實現對lvs控制器和web服務器的健康監測,一旦服務出現問題能保證服務不中斷的情況下排除故障。即增加lvs控制器實現主備模式避免單點故障以及自動刪除故障web服務結點並當它恢復後再自動添加到羣集中這樣的功能。
預期狀況如下圖所示，
通過上圖所示實現以下功能
⑴、lvs-dr1和lvs-dr2採用主從的方式配置，從控制器通過心跳監測主控制器是否存在。當監測到主控制器不存在時接管虛擬IP實現向web服務器轉發用戶請求。
⑵、lvs控制器監控web服務器是否存在，當服務器出現故障（宕機、服務終止、網線鬆動）時自動刪除故障結點，當服務器上線恢復提供服務後又能夠自動將該結點添加到羣集中。
4、根據以上對現狀和預期的分析後，爲了測試我們做了如下準備。
|
服務器名稱 |
安裝軟件 |
|
lvs-dr1 |
ipvsadm-1.24
keepalived-1.1.15 |
|
lvs-dr2 | |
|
lvs-web1 |
cronolog-1.6.2
httpd-2.2.9 |
|
lvs-web2 |
注：
⑴、由於整個測試實施文檔在非生產環境下完成，所以使用虛擬機軟件VMWare Workstation 5.5.1作爲以上所有結點的運行平臺。
⑵、測試環境中所有結點的操作系統是CentOS 5.1 /CentOS release 5 (Final)，內核版本爲2.6.18-53.el5。
如何查看系統和內核版本？
①、/etc/issue
②、uname -a
③、/proc/version
⑶、實現lvs羣集的兩個重要部件是ipvs內核模塊和ipvsadm工具包。
當前內核版本的系統已經包含ipvs內核模塊，但默認並沒有加載到內核中，可以手工加載或安裝ipvsadm之後會被加載。
使用modprobe命令手工加載ipvs模塊並查詢模塊是否加載。
⑷、keepalived是一個監測lvs轉發器和web服務器狀態的軟件，下面是官方網站對keepalived的一段解釋，官方網站地址[url]http://www.keepalived.org[/url]
What is Keepalived ?
The main goal of the keepalived project is to add a strong & robust keepalive facility to the Linux Virtual Server project. This project is written in C with multilayer TCP/IP stack checks. Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5/7. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entrie from the LVS topology. In addition keepalived implements an independent VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover.
The main goal of the keepalived project is to add a strong & robust keepalive facility to the Linux Virtual Server project. This project is written in C with multilayer TCP/IP stack checks. Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5/7. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entrie from the LVS topology. In addition keepalived implements an independent VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover.
Why using Keepalived ?
If your are using a LVS director to loadbalance a server pool in a production environnement, you may want to have a robust solution for healthcheck & failover.
If your are using a LVS director to loadbalance a server pool in a production environnement, you may want to have a robust solution for healthcheck & failover.
⑸、web服務器上需要安裝apache2，cronolog是選擇安裝軟件，它可以格式化日誌文件的格式，易於對apache日誌的管理和分析。
四、測試步驟
1、配置各個結點的主機名、IP地址。配置過程暫省略，配置如下
|
服務器名稱 |
服務器網絡配置 |
|
lvs-dr1 (dr1) |
192.168.1.210 |
|
lvs-dr2 (dr2) |
192.168.1.211 |
|
lvs-web1 (web1) |
192.168.1.216 |
|
lvs-web2 (web2) |
192.168.1.217 |
注：修改主機名需要/etc/sysconfig/network和/etc/hosts兩個文件，然後重新啓動生效。
2、在lvs-dr1和lvs-dr2上安裝ipvsadm和keepalived，過程如下：
⑴、安裝ipvsadm，從官方網站下載ipvsadm-1.24.tar.gz後
[root@lvs-dr1 ~]# tar zxvf ipvsadm-1.24.tar.gz # 解壓縮ipvsadm包 #
[root@lvs-dr1 ~]# ln -s /usr/src/kernels/2.6.18-53.el5-i686 /usr/src/linux # 做一個目錄的鏈接 #
[root@lvs-dr1 ~]# ls -l /usr/src/ # 查看ln過的鏈接狀態#
total 16
drwxr-xr-x 3 root root 4096 Jun 24 17:28 kernels
lrwxrwxrwx 1 root root 35 Aug 3 18:35 linux -> /usr/src/kernels/2.6.18-53.el5-i686
drwxr-xr-x 7 root root 4096 Jun 24 17:27 redhat
[root@lvs-dr1 ~]# cd ipvsadm-1.24
[root@lvs-dr1 ipvsadm-1.24]make && make install # 編譯並安裝 #
[root@lvs-dr1 ipvsadm-1.24]# find / -name ipvsadm # 查看ipvsadm的位置 #
/etc/rc.d/init.d/ipvsadm
/root/ipvsadm-1.24/ipvsadm
/sbin/ipvsadm
⑵、安裝keepalived，從官方網站下載keepalived-1.1.15.tar.gz後
[root@lvs-dr1 ~]# tar zxvf keepalived-1.1.15.tar.gz # 解壓縮keepalived包 #
[root@lvs-dr1 ~]# cd keepalived-1.1.15
[root@lvs-dr1 keepalived-1.1.15]# ./configure
# 最好按照默認配置參數不要指定prefix，可能會出現問題 #
……………省略……………………………………
Keepalived configuration
------------------------
Keepalived version : 1.1.15
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework : Yes
Use LinkWatch : No
Use Debug flags : No
……………結束……………………………………
[root@lvs-dr1 keepalived-1.1.15]# make && make install # 編譯並安裝 #
[root@lvs-dr1 keepalived-1.1.15]# find / -name keepalived # 查看keepalived位置 #
/usr/local/etc/sysconfig/keepalived
/usr/local/etc/keepalived
/usr/local/etc/rc.d/init.d/keepalived
/usr/local/sbin/keepalived
/root/keepalived-1.1.15/bin/keepalived
/root/keepalived-1.1.15/keepalived
/root/keepalived-1.1.15/keepalived/etc/keepalived
3、在lvs-web1和lvs-web2上安裝並配置apache和cronolog。
⑴、安裝cronolog，從官方網站下載cronolog-1.6.2.tar.gz後
[root@lvs-web1 ~]# tar zxvf cronolog-1.6.2.tar.gz
[root@lvs-web1 ~]# cd cronolog-1.6.2
[root@lvs- web1 cronolog-1.6.2]# ./configure
[root@lvs- web1 cronolog-1.6.2]# make
[root@lvs- web1 r cronolog-1.6.2]# make install
[root@lvs- web1 cronolog-1.6.2]# find / -name cronolog
/usr/local/sbin/cronolog
⑵、安裝apache，從官方網站下載httpd-2.2.9.tar.gz後
①、添加用於啓動httpd服務的用戶和組
[root@lvs-web1 ~]# groupadd –g 500 hjw
[root@lvs-web1 ~]# useradd –g 500 –u 500 hjw
[root@lvs-web1 ~]# grep hjw /etc/passwd # 查看用戶文件 #
hjw:x:500:500::/home/hjw:/bin/bash
# 雖然沒有設置爲/sbin/nologin，但沒有爲賬戶設置口令用戶無法使用該賬戶登陸，hjw只能作爲服務帳戶使用 #
[root@lvs-web1 ~]# grep hjw /etc/group # 查看組文件 #
hjw:x:500:
②、安裝apache
安裝：
[root@lvs-web1 ~]# tar zxvf httpd-2.2.9.tar.gz
[root@lvs-web1 ~]# cd httpd-2.2.9
[root@lvs-web1 httpd-2.2.9]# ./configure --prefix=/usr/local/apache2 --disable-option-checking --enable-cache --enable-disk-cache --enable-mem-cache --enable-rewrite=shared
#配置簡單說明
# --prefix=/usr/local/apache2 指定apache安裝路徑
# --disable-option-checking 不返回錯誤信息
# --enable-cache 啓用緩存
# --enable-disk-cache 啓用磁盤緩存
# --enable-mem-cache 啓用內存緩存
# --enable-rewrite=shared 實時重寫URL請求
[root@lvs-web1 httpd-2.2.9]# make && make install
[root@lvs-web1 ~]# apachectl -t -D DUMP_MODULES # 查看apache加載的模塊 #
Loaded Modules:
……………省略……………………………………
cache_module (static) # static-靜態被編譯到代碼中，而非動態調用 #
disk_cache_module (static)
mem_cache_module (static)
……………省略……………………………………
so_module (static) # 允許在Apache啓動和重啓時加載DSO模塊，而不用重新編譯。Apache 2.2.9默認將so模塊編譯到httpd代碼中 #
rewrite_module (shared) # shared-DSO(Dynamic Shared Objects) 使該模塊用於動態調用。關於DSO相關文檔請參閱[url]http://beacon.blog.51cto.com/442731/94711[/url] #
Syntax OK
……………結束……………………………………
[root@lvs-web1 ~]# vi /usr/local/apache2/conf/httpd.conf # 修改配置文件 #
……………省略……………………………………
User daemon # 改成hjw
Group daemon # 改成hjw
……………省略……………………………………
#ServerName [url]www.example.com:80[/url]
ServerName 127.0.0.1 #添加這一行
……………省略……………………………………
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all # 改成 Allow from all
</Directory>
……………省略……………………………………
# Virtual hosts
#Include conf/extra/httpd-vhosts.conf #將這一行註釋去掉
……………結束……………………………………
[root@lvs-web1 ~]# vi /usr/local/apache2/conf/extra/httpd-vhosts.conf #修改虛擬目錄文件#
……………省略……………………………………
# 添加虛擬目錄如下
<VirtualHost *:80>
# ServerAdmin [email][email protected][/email]
DocumentRoot "/usr/local/apache2/webapps"
DirectoryIndex index.html
# ServerName dummy-host2.example.com
# ErrorLog "logs/dummy-host2.example.com-error_log"
# CustomLog "logs/dummy-host2.example.com-access_log" common
</VirtualHost>
<VirtualHost *:8090>
# ServerAdmin [email][email protected][/email]
DocumentRoot "/usr/local/apache2/webapps"
DirectoryIndex index1.html
# ServerName dummy-host2.example.com
# ErrorLog "logs/dummy-host2.example.com-error_log"
# CustomLog "logs/dummy-host2.example.com-access_log" common
……………結束……………………………………
直接訪問客戶端
在/etc/rc.local中添加，使得apache在開機時啓動
[root@lvs-web1 extra]# more /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/usr/local/apache2/bin/apachectl start
# shell end
4、使用ipvs腳本實現單臺lvs控制器轉發請求到兩臺web服務器
⑴、lvs-dr1/lvs控制器
①、ipvs腳本
[root@lvs-dr1 bin]# pwd
/usr/local/bin # 控制器上ipvs腳本的路徑 #
[root@lvs-dr1 bin]# ll lvsdr
-rwxr-xr-x 1 root root 1163 Aug 15 22:37 lvsdr
# 腳本需要可執行權限，使用chmod 755 lvsdr給與腳本文件執行執行權限 #
[root@lvs-dr1 bin]# more lvsdr
#!/bin/bash
# 2008-08-19 by hjw
RIP1=192.168.1.216
RIP2=192.168.1.217
VIP1=192.168.1.215
VIP2=192.168.1.220
/etc/rc.d/init.d/functions
case "$1" in
start)
echo " start LVS of DirectorServer"
# set the Virtual IP Address and sysctl parameter
/sbin/ifconfig eth0:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up
/sbin/ifconfig eth0:1 $VIP2 broadcast $VIP2 netmask 255.255.255.255 up
/sbin/route add -host $VIP1 dev eth0:0
/sbin/route add -host $VIP2 dev eth0:1
echo "1" >/proc/sys/net/ipv4/ip_forward
#Clear IPVS table
/sbin/ipvsadm -C
#set LVS
#Web Apache
/sbin/ipvsadm -A -t $VIP1:80 -s wlc -p 800
/sbin/ipvsadm -a -t $VIP1:80 -r $RIP1:80 -g
/sbin/ipvsadm -a -t $VIP1:80 -r $RIP2:80 -w 3 -g
#/sbin/ipvsadm -a -t $VIP1:80 -r $RIP2:80 -w 3 -g
#/sbin/ipvsadm -a -t $VIP1:80 -r $RIP3:80 -g
/sbin/ipvsadm -A -t $VIP2:8090 -s wlc -p 1800
/sbin/ipvsadm -a -t $VIP2:8090 -r $RIP1:8090 -g
/sbin/ipvsadm -a -t $VIP2:8090 -r $RIP2:8090 -w 3 -g
#Run LVS
/sbin/ipvsadm
;;
stop)
echo "close LVS Directorserver"
echo "0" >/proc/sys/net/ipv4/ip_forward
/sbin/ipvsadm -C
/sbin/ifconfig eth0:0 down
#/sbin/ifconfig eth0:1 down
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
# shell end
②、lvs-dr1的網絡配置
[root@lvs-dr1 bin]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:F4:5B:1C
inet addr:192.168.1.210 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fef4:5b1c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3148 errors:0 dropped:0 overruns:0 frame:0
TX packets:430 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:257945 (251.8 KiB) TX bytes:59872 (58.4 KiB)
Interrupt:177 Base address:0x1080
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:F4:5B:1C
inet addr:192.168.1.215 Bcast:192.168.1.215 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:177 Base address:0x1080
eth0:1 Link encap:Ethernet HWaddr 00:0C:29:F4:5B:1C
inet addr:192.168.1.220 Bcast:192.168.1.220 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:177 Base address:0x1080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:48 errors:0 dropped:0 overruns:0 frame:0
TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4000 (3.9 KiB) TX bytes:4000 (3.9 KiB)
③、在/etc/rc.local中添加，腳本開機啓動
[root@lvs-dr1 ~]# more /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/usr/local/bin/lvsdr start
# shell end
⑵、web服務器/ lvs-web1和lvs-web2
①、ipvs腳本
[root@lvs-web1 bin]# pwd
/usr/local/bin #web服務器上ipvs腳本的路徑 #
[root@lvs-web1 bin]# ll lvs
-rwxr-xr-x 1 root root 874 Aug 15 05:18 lvs
# 腳本需要可執行權限，使用chmod 755 lvsdr給與腳本文件執行執行權限 #
[root@lvs-web1 bin]# more lvs
#!/bin/bash
#description:start realserver
#chkconfig 235 26 26
# 2008-08-19 by hjw
VIP1=192.168.1.215
VIP2=192.168.1.220
/etc/rc.d/init.d/functions
case "$1" in
start)
echo " start LVS of REALServer"
/sbin/ifconfig lo:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up
/sbin/ifconfig lo:1 $VIP2 broadcast $VIP2 netmask 255.255.255.255 up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
/sbin/ifconfig lo:0 down
/sbin/ifconfig lo:1 down
echo "close LVS Directorserver"
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
# shell end
②、lvs-web1的網絡配置（lvs-web2略）
[root@lvs-web1 bin]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:8C:52:7E
inet addr:192.168.1.216 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8c:527e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3671 errors:0 dropped:0 overruns:0 frame:0
TX packets:475 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:301398 (294.3 KiB) TX bytes:65170 (63.6 KiB)
Interrupt:177 Base address:0x1080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:904 (904.0 b) TX bytes:904 (904.0 b)
lo:0 Link encap:Local Loopback
inet addr:192.168.1.215 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
lo:1 Link encap:Local Loopback
inet addr:192.168.1.220 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
③、在/etc/rc.local中添加，腳本開機啓動
[root@lvs-web1 ~]# more /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/usr/local/apache2/bin/apachectl start
/usr/local/bin/lvs start&
# shell end
⑶、測試轉發
使用ipvsadm -l查看控制器轉發狀態。
[root@lvs-dr1 bin]# watch ipvsadm -l
查看lvs-web1狀態
[root@lvs-web1 bin]# netstat -an|grep 80
[root@lvs-web1 bin]# netstat -an|grep 80|wc -l
查看lvs-web2狀態
客戶端返回的頁面
由於lvs-web2的權重值大，客戶端從lvs-web2得到請求的頁面，說明整個系統運行正常。
②、模擬web服務器故障，將lvs-web2網絡中斷。
用ipvsadm –l查看狀態，有4個等待轉發的不活動連接
最終因爲lvs-web2的web服務不可用無法相應客戶端請求，返回無法顯示的頁面
5、使用keepalived實現單臺lvs控制器轉發請求到兩臺web服務器，並且lvs控制器監控web服務器的狀態並自動添加刪除web結點。
⑴、在lvs控制器上安裝keepalived後需要停止原來的lvsdr腳本，然後根據需要對keepalived配置文件進行調整，而web服務器的腳本和網絡配置不需要進行任何改動。
①、keepalived腳本
[root@lvs-dr1 keepalived]# pwd
/usr/local/etc/keepalived # keepalived配置文件的路徑 #
[root@lvs-dr1 keepalived]# ll keepalived.conf
-rw-r--r-- 1 root root 1627 Aug 16 09:13 keepalived.conf # keepalived配置文件名 #
[root@lvs-dr1 keepalived]# more keepalived.conf
! Configuration File for keepalived
global_defs {
# notification_email {
# [email][email protected][/email]
# [email][email protected][/email]
# [email][email protected][/email]
# }
# notification_email_from [email][email protected][/email]
# smtp_server 192.168.200.1
# smtp_connect_timeout 30
router_id LVS_DEVEL
}
# 2008-08-19 by hjw
# VIP1
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.215
192.168.1.220
}
}
virtual_server 192.168.1.215 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.216 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.1.217 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 192.168.1.220 8090 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.216 8090 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8090
}
}
real_server 192.168.1.217 8090 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8090
}
}
}
# shell end
②、在/etc/rc.local中添加，腳本開機啓動
[root@lvs-dr1 ~]# more /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
#/usr/local/bin/lvsdr start
/usr/local/sbin/keepalived -D -f /usr/local/etc/keepalived/keepalived.conf
# shell end
⑵、用戶請求轉發功能測試
由於轉發功能測試在步驟4中已經進行，這裏省略。
⑶、自動刪除添加結點功能測試
測試刪除/添加結點需要模擬web服務器故障，其中包括手動終止服務和中斷網絡（相當於宕機），測試以lvs-web2爲例。
①、手動停止lvs-web2上的httpd服務
控制器lvs-dr1上監控轉發狀態。當監測到lvs-web2上的服務停止後大概2-3秒，keepalived自動將這個結點從轉發列表中刪除。
此時，用戶請求將只被轉發到lvs-web1上。
啓動httpd服務之後5秒之內lvs-web2又重新被加到轉發列表中，此時用戶請求可以被轉發到lvs-web1和lvs-web2但由於web2的權重值大所以用戶再次請求[url]http://192.168.1.215[/url] 時請求被轉發到lvs-web2。
②、中斷網絡，
控制器通過發送ICMP包判斷lvs-web2已經不存在，所以將它從轉發列表中刪除。lvs-dr1會根據配置文件不斷監測real_server，當lvs-web2重新上線並能夠提供web服務之後dr1又會重新將web2添加到轉發列表中。
這個過程刪除和添加web結點的過程與上面停止httpd服務的完全一致的，在此不再贅述。這些過程只是在服務器端進行，對於用戶來說是完全透明的，他們並不會感受到服務器端正在發生的變化。
6、避免lvs控制器單點故障，添加lvs-dr2到羣集中使用keepalived實現控制器的主備。
⑴、keepalived配置文件
[root@lvs-dr1 keepalived]# pwd
/usr/local/etc/keepalived # keepalived配置文件位置 #
[root@lvs-dr1 keepalived]# ll keepalived.conf
-rw-r--r-- 1 root root 1624 Aug 16 23:19 keepalived.conf
[root@lvs-dr1 keepalived]# more keepalived.conf
! Configuration File for keepalived
global_defs {
# notification_email {
# [email][email protected][/email]
# [email][email protected][/email]
# [email][email protected][/email]
# }
# notification_email_from [email][email protected][/email]
# smtp_server 192.168.200.1
# smtp_connect_timeout 30
router_id LVS_DEVEL
}
# 2008-08-19 by hjw
vrrp_instance VI_1 {
state MASTER # lvs-dr2爲BACKUP #
interface eth0
virtual_router_id 51
priority 200 # lvs-dr2爲100 #
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.215
192.168.1.220
}
}
virtual_server 192.168.1.215 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.216 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.1.217 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 192.168.1.220 8090 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.216 8090 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8090
}
}
real_server 192.168.1.217 8090 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8090
}
}
}
# shell end
⑵、lvs-dr1啓動，lvs-dr2關機
|
服務器 |
狀態 |
日誌 |
|
lvs-dr1 |
啓動 |
√ |
|
lvs-dr2 |
關機 |
|
lvs-dr1啓動後日志
……………省略……………………………………
Aug 17 00:53:40 lvs-dr1 Keepalived: Starting Keepalived v1.1.15 (08/04,2008)
Aug 17 00:53:40 lvs-dr1 Keepalived: Starting Healthcheck child process, pid=1671 #啓動Healthcheck進程#
Aug 17 00:53:40 lvs-dr1 Keepalived: Starting VRRP child process, pid=1681 #啓動VRRP child進程#
Aug 17 00:53:40 lvs-dr1 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Aug 17 00:53:40 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.210 added
Aug 17 00:53:40 lvs-dr1 Keepalived_vrrp: Registering Kernel netlink reflector
Aug 17 00:53:40 lvs-dr1 Keepalived_vrrp: Registering Kernel netlink command channel
Aug 17 00:53:40 lvs-dr1 Keepalived_vrrp: Registering gratutious ARP shared channel
Aug 17 00:53:41 lvs-dr1 kernel: IPVS: Registered protocols (TCP, UDP, AH, ESP)
Aug 17 00:53:41 lvs-dr1 kernel: IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
Aug 17 00:53:41 lvs-dr1 kernel: IPVS: ipvs loaded.
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.210 added
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Registering Kernel netlink reflector
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Registering Kernel netlink command channel
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 00:53:41 lvs-dr1 Keepalived_vrrp: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Configuration is using : 16607 Bytes
Aug 17 00:53:41 lvs-dr1 Keepalived_vrrp: Configuration is using : 35534 Bytes
Aug 17 00:53:41 lvs-dr1 kernel: IPVS: [wlc] scheduler registered.
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:80]
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:80]
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:8090]
Aug 17 00:53:41 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:8090]
Aug 17 00:53:41 lvs-dr1 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Aug 17 00:53:42 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE #dr1置成master狀態#
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 17 00:53:43 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.215 added
Aug 17 00:53:43 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.220 added
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215 #arp綁定ip 192.168.1.215到eth0#
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.215 added
Aug 17 00:53:43 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.220 added
Aug 17 00:53:48 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215
Aug 17 00:53:48 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
……………結束……………………………………
⑶、lvs-dr1開機，lvs-dr2啓動 lvs-dr2日誌
|
服務器 |
狀態 |
日誌 |
|
lvs-dr1 |
開機 |
|
|
lvs-dr2 |
啓動 |
√ |
lvs-dr2日誌啓動日誌
……………省略……………………………………
Aug 17 03:49:21 lvs-dr2 Keepalived: Starting Keepalived v1.1.15 (08/04,2008)
Aug 17 03:49:21 lvs-dr2 Keepalived: Starting Healthcheck child process, pid=1656
Aug 17 03:49:21 lvs-dr2 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Aug 17 03:49:21 lvs-dr2 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.211 added
Aug 17 03:49:21 lvs-dr2 Keepalived_vrrp: Registering Kernel netlink reflector
Aug 17 03:49:21 lvs-dr2 Keepalived_vrrp: Registering Kernel netlink command channel
Aug 17 03:49:21 lvs-dr2 Keepalived_vrrp: Registering gratutious ARP shared channel
Aug 17 03:49:21 lvs-dr2 Keepalived: Starting VRRP child process, pid=1658
Aug 17 03:49:22 lvs-dr2 kernel: IPVS: Registered protocols (TCP, UDP, AH, ESP)
Aug 17 03:49:22 lvs-dr2 kernel: IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
Aug 17 03:49:22 lvs-dr2 kernel: IPVS: ipvs loaded.
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.211 added
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Registering Kernel netlink reflector
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Registering Kernel netlink command channel
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 03:49:22 lvs-dr2 Keepalived_vrrp: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 03:49:22 lvs-dr2 Keepalived_healthcheckers: Configuration is using : 16607 Bytes
Aug 17 03:49:22 lvs-dr2 Keepalived_vrrp: Configuration is using : 35534 Bytes
Aug 17 03:49:23 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE # dr2置成backup狀態#
Aug 17 03:49:23 lvs-dr2 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Aug 17 03:49:23 lvs-dr2 kernel: IPVS: [wlc] scheduler registered.
Aug 17 03:49:23 lvs-dr2 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:80]
Aug 17 03:49:23 lvs-dr2 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:80]
Aug 17 03:49:23 lvs-dr2 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:8090]
Aug 17 03:49:23 lvs-dr2 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:8090]
……………結束……………………………………
⑷、lvs-dr1關機，lvs-dr2開機 lvs-dr2日誌
|
服務器 |
狀態 |
日誌 |
|
lvs-dr1 |
關閉 |
|
|
lvs-dr2 |
開機 |
√ |
lvs-dr2日誌
……………省略……………………………………
Aug 17 03:52:11 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE # dr2置成master狀態#
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.215 added
Aug 17 03:52:12 lvs-dr2 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.220 added
Aug 17 03:52:12 lvs-dr2 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.215 added
Aug 17 03:52:12 lvs-dr2 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.220 added
Aug 17 03:52:17 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215
Aug 17 03:52:17 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
……………結束……………………………………
⑸、lvs-dr1啓動，lvs-dr2開機 lvs-dr1和lvs-dr2日誌
|
服務器 |
狀態 |
日誌 |
|
lvs-dr1 |
啓動 |
√ |
|
lvs-dr2 |
開機 |
√ |
lvs-dr1啓動日誌
……………結束……………………………………
Aug 17 01:01:32 lvs-dr1 Keepalived: Starting Keepalived v1.1.15 (08/04,2008)
Aug 17 01:01:32 lvs-dr1 Keepalived: Starting Healthcheck child process, pid=1688
Aug 17 01:01:32 lvs-dr1 Keepalived: Starting VRRP child process, pid=1690
Aug 17 01:01:32 lvs-dr1 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Aug 17 01:01:32 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.210 added
Aug 17 01:01:32 lvs-dr1 Keepalived_vrrp: Registering Kernel netlink reflector
Aug 17 01:01:32 lvs-dr1 Keepalived_vrrp: Registering Kernel netlink command channel
Aug 17 01:01:32 lvs-dr1 Keepalived_vrrp: Registering gratutious ARP shared channel
Aug 17 01:01:33 lvs-dr1 kernel: IPVS: Registered protocols (TCP, UDP, AH, ESP)
Aug 17 01:01:33 lvs-dr1 kernel: IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
Aug 17 01:01:33 lvs-dr1 kernel: IPVS: ipvs loaded.
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.210 added
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Registering Kernel netlink reflector
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Registering Kernel netlink command channel
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 01:01:33 lvs-dr1 Keepalived_vrrp: Opening file '/usr/local/etc/keepalived/keepalived.conf'.
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Configuration is using : 16607 Bytes
Aug 17 01:01:33 lvs-dr1 Keepalived_vrrp: Configuration is using : 35534 Bytes
Aug 17 01:01:33 lvs-dr1 kernel: IPVS: [wlc] scheduler registered.
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:80]
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:80]
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.216:8090]
Aug 17 01:01:33 lvs-dr1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.217:8090]
Aug 17 01:01:33 lvs-dr1 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Aug 17 01:01:34 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE # dr1置成master狀態#
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.215 added
Aug 17 01:01:35 lvs-dr1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.220 added
Aug 17 01:01:35 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.215 added
Aug 17 01:01:35 lvs-dr1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.220 added
Aug 17 01:01:40 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.215
Aug 17 01:01:40 lvs-dr1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.220
……………結束……………………………………
lvs-dr2日誌
……………省略……………………………………
Aug 17 03:54:11 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
# 收到優先級別更高的通告 #
Aug 17 03:54:11 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE # dr2置成backup狀態#
Aug 17 03:54:11 lvs-dr2 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs. # 移除VIP綁定 #
Aug 17 03:54:11 lvs-dr2 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.215 removed
Aug 17 03:54:11 lvs-dr2 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.220 removed
Aug 17 03:54:11 lvs-dr2 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.215 removed
Aug 17 03:54:11 lvs-dr2 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.220 removed
……………結束……………………………………
討論、結論和參考資料整體方案實施後補充
待續……..
五、測試討論
六、測試結論
七、參考資料