【運維小實驗】基於VMware的虛擬Linux集羣搭建-lvs+keepalived

通過keepalived實現lvs服務器的的雙機熱備和真實服務器之間的負載均衡的blog挺多的，在搭建的時候也參考了相關博文，不同人搭建的實驗環境不一樣，本文是基於VM搭建的虛擬集羣環境，全部虛擬機網卡模式爲NAT方式相連，處於同一網段。

搭建環境：

使用redhead 2.6.32-431.el6.x86_64版本的linux，創建四臺，配置相關網絡信息，確保同一網段下。

配置好各自ip即可，虛擬ip在配置lvs時設定。

安裝配置LVS

1.在主備Lvs上安裝ipvsadm和keepalived

LVS是通過IPVS模塊實現的，檢查kernel是否支持LVS的IPVS模塊，再安裝IPVS管理軟件ipvsadm

[root@rex ~]# modprobe -l | grep ipvs

kernel/net/netfilter/ipvs/ip_vs.ko

kernel/net/netfilter/ipvs/ip_vs_rr.ko

kernel/net/netfilter/ipvs/ip_vs_wrr.ko

kernel/net/netfilter/ipvs/ip_vs_lc.ko

kernel/net/netfilter/ipvs/ip_vs_wlc.ko

kernel/net/netfilter/ipvs/ip_vs_lblc.ko

kernel/net/netfilter/ipvs/ip_vs_lblcr.ko

kernel/net/netfilter/ipvs/ip_vs_dh.ko

kernel/net/netfilter/ipvs/ip_vs_sh.ko

kernel/net/netfilter/ipvs/ip_vs_sed.ko

kernel/net/netfilter/ipvs/ip_vs_nq.ko

kernel/net/netfilter/ipvs/ip_vs_ftp.ko

kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko

安裝ipvsadm

[root@rex ~]# yum install ipvsadm

能通過ipvsadm --help查看相關信息即安裝成功。

安裝keepalived

安裝依賴庫：yum install -y openssl openssl-devel（我是缺這兩個，具體編譯時候如果報錯則安裝相關提示補安裝相關缺漏）

下載keepalived：

http://www.keepalived.org/software/keepalived-1.2.6.tar.gz

安裝命令：

# tar zxvf keepalived-1.2.6.tar.gz

# cd keepalived-1.2.6

# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-431.el6.x86_64 #指定配置文件存放路徑，指定使用內核源碼中的頭文件。

#make

#make install

#ln -s /usr/local/sbin/keepalived /sbin/ #把啓動命令軟連接到sbin，方便以後直接使用

能通過keepalived --help查看相關信息即安裝成功。

配置LVS：

主LVS：

! Configuration File for keepalived

global_defs {

notification_email { #配置告警郵件發送

XXXX@qq.com

}

notification_email_from Keepalived@localhost

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER #主LVS標誌

interface eth1 #設置對外服務口

virtual_router_id 51

priority 100 #設置優先級，優先級高的爲主機

advert_int 1 #設置同步時間間隔

authentication { #設置主備LVS驗證類型和密碼，兩邊必須一致

auth_type PASS

auth_pass 1111

}

virtual_ipaddress { #設置虛擬服務IP

192.168.153.110

}

virtual_server 192.168.153.110 80 {

delay_loop 6 #健康檢查時間間隔

lb_algo rr #負載均衡算法，此爲輪詢算法

lb_kind DR #負載均衡轉發規則，也就是IP轉發規則，DR轉發方式最快，但是所有機器必須在一個網段內，還有NAT和TUNEL兩種轉發方式

protocol TCP

real_server 192.168.153.131 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

real_server 192.168.153.135 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

備用LVS配置：複製上面配置修改兩處即可：1.state MASTER改爲state BACKUP 2.將priority 100改低，這裏改爲80

配置Realserver

對所有Realserver重複做一下步驟

在/etc/init.d下創建腳本lvsrs

1 #!/bin/bash

3 VIP=192.168.153.110

4 ./etc/rc.d/init.d/functions

5 case "$1" in

6 start)

7 echo "Start LVS of Realserver!"

8 /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up

9 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

10 echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

11 echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

12 echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

14 ;;

16 stop)

18 /sbin/ifconfig lo:0 down

19 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

20 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

21 echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

22 echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

23 echo "RealServer Stoped"

25 ;;

27 *)

29 echo "Usage: $0 {start|stop}"

31 exit 1

33 esac

修改其執行權限：chmod 755 /etc/init.d/lvsrs

執行啓動腳本：

#service lvsrs start

查看其ip：

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:8 errors:0 dropped:0 overruns:0 frame:0

TX packets:8 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:480 (480.0 b) TX bytes:480 (480.0 b)

lo:0 Link encap:Local Loopback

inet addr:192.168.153.110 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:16436 Metric:1

在realserver上安裝apache並且啓動！

在 /var/www/hmtl/下添加index.html測試主頁，不同server用不同界面進行區分。

啓動LVS並且測試：

主備LVS啓動：/etc/init.d/keepalived start

查看啓動狀態：

主LVS：

Feb 4 20:11:41 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Transition to MASTER STATE

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Entering MASTER STATE

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) setting protocol VIPs.

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.153.110

Feb 4 20:11:42 rex Keepalived_healthcheckers[9521]: Netlink reflector reports IP 192.168.153.110 added

Feb 4 20:11:47 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.153.110

備LVS：

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Received higher prio advert

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Entering BACKUP STATE

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) removing protocol VIPs.

Feb 4 20:18:16 rex Keepalived_healthcheckers[9521]: Netlink reflector reports IP 192.168.153.110 removed

Feb 4 20:19:06 rex dhclient[1265]: DHCPREQUEST on eth2 to 192.168.153.254 port 67 (xid=0x6f9b7b38)

用ip addr和ipvsadm查看路由情況

主LVS# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:3e:ce:ce brd ff:ff:ff:ff:ff:ff

inet 192.168.153.133/24 brd 192.168.153.255 scope global eth1

inet 192.168.153.110/32 scope global eth1

inet6 fe80::20c:29ff:fe3e:cece/64 scope link

valid_lft forever preferred_lft forever

備LVS# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:0e:6c:b0 brd ff:ff:ff:ff:ff:ff

inet 192.168.153.134/24 brd 192.168.153.255 scope global eth2

inet6 fe80::20c:29ff:fe0e:6cb0/64 scope link

valid_lft forever preferred_lft forever

[root@rex ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP bogon:http rr

-> bogon:http Route 1 0 0

測試：

1.高可用性功能測試--主備lvs切換

把主lvs stop！然後查看備lvs日誌，然後把主恢復，再看備lvs的日誌

Feb 4 20:11:41 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Transition to MASTER STATE

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Entering MASTER STATE

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) setting protocol VIPs.

Feb 4 20:11:42 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.153.110

Feb 4 20:11:42 rex Keepalived_healthcheckers[9521]: Netlink reflector reports IP 192.168.153.110 added

Feb 4 20:11:47 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.153.110

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Received higher prio advert

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) Entering BACKUP STATE

Feb 4 20:18:16 rex Keepalived_vrrp[9522]: VRRP_Instance(VI_1) removing protocol VIPs.

Feb 4 20:18:16 rex Keepalived_healthcheckers[9521]: Netlink reflector reports IP 192.168.153.110 removed

Feb 4 20:19:06 rex dhclient[1265]: DHCPREQUEST on eth2 to 192.168.153.254 port 67 (xid=0x6f9b7b38)

Feb 4 20:19:06 rex dhclient[1265]: DHCPACK from 192.168.153.254 (xid=0x6f9b7b38)

Feb 4 20:19:06 rex dhclient[1265]: bound to 192.168.153.134 -- renewal in 737 seconds.

Feb 4 20:19:06 rex NetworkManager[1241]: <info> (eth2): DHCPv4 state changed renew -> renew

Feb 4 20:19:06 rex NetworkManager[1241]: <info> address 192.168.153.134

Feb 4 20:19:06 rex NetworkManager[1241]: <info> prefix 24 (255.255.255.0)

Feb 4 20:19:06 rex NetworkManager[1241]: <info> gateway 192.168.153.2

Feb 4 20:19:06 rex NetworkManager[1241]: <info> nameserver '192.168.153.2'

Feb 4 20:19:06 rex NetworkManager[1241]: <info> domain name 'localdomain'

2.負載均衡測試

在本地瀏覽器上訪問192.168.153.110，不斷刷新，可以看到代表不同realserver的主界面。

3.故障切換測試

把realserver1的服務stop掉，查看LVS的日誌，然後恢復再看：

Feb 4 20:25:59 rex Keepalived_healthcheckers[9588]: Netlink reflector reports IP 192.168.153.110 added

Feb 4 20:26:04 rex Keepalived_vrrp[9589]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.153.110

Feb 4 20:27:53 rex Keepalived_healthcheckers[9588]: TCP connection to [192.168.153.135]:80 failed !!!

Feb 4 20:27:53 rex Keepalived_healthcheckers[9588]: Removing service [192.168.153.135]:80 from VS [192.168.153.110]:80

Feb 4 20:27:53 rex Keepalived_healthcheckers[9588]: Remote SMTP server [192.168.200.1]:25 connected.

Feb 4 20:28:14 rex Keepalived_healthcheckers[9588]: Error reading data from remote SMTP server [192.168.200.1]:25.

Feb 4 20:28:29 rex Keepalived_healthcheckers[9588]: TCP connection to [192.168.153.135]:80 success.

Feb 4 20:28:29 rex Keepalived_healthcheckers[9588]: Adding service [192.168.153.135]:80 to VS [192.168.153.110]:80

Feb 4 20:28:29 rex Keepalived_healthcheckers[9588]: Remote SMTP server [192.168.200.1]:25 connected.

【運維小實驗】基於VMware的虛擬Linux集羣搭建-lvs+keepalived

【學技術是慢餐】系統運維知識框架

【運維小實驗】數據鏡像備份實驗inotify+rsync實時備份

我的友情鏈接

微信“小程序”有戲嗎？

【運維小分享】基於Web應用的性能分析及優化案例詳解

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結