linux下安裝部署keepalived，利用keepalived多臺nginx虛擬IP浮動

一、準備

軟件包名稱：keepalived  2.0.19

操作系統環境

演示環境爲centos7.7-x86_64-minimal 操作系統

 

IP	部署	說明
192.168.1.91	--	虛IP映射域名
192.168.1.97	keepalived	主要節點
192.168.1.98	keepalived	備份節點

二、keepalived 編譯安裝

上傳keepalived安裝包至相應目錄，這裏我們指定 /kp/keepalived

[root@pve-97 keepalived]# pwd
/kp/keepalived
[root@pve-97 keepalived]# ll
total 1004
-rw-r--r--. 1 root root 1025062 Jan  8 16:59 keepalived-2.0.19.tar.gz

解壓命令：tar -zxvf keepalived-2.0.19.tar.gz 並進入目錄

[root@pve-97 keepalived-2.0.19]# ll
total 1212
-rw-rw-r--. 1 1000 1000  54387 Oct 20 00:16 aclocal.m4
-rwxr-xr-x. 1 1000 1000   5826 Mar 26  2018 ar-lib
-rw-rw-r--. 1 1000 1000     41 Aug 16  2018 AUTHOR
drwxrwxr-x. 2 1000 1000     44 Oct 20 00:16 bin_install
-rwxrwxr-x. 1 1000 1000     64 Aug 16  2018 build_setup
-rw-rw-r--. 1 1000 1000 494050 Oct 20 00:08 ChangeLog
-rwxr-xr-x. 1 1000 1000   7333 Mar 26  2018 compile
-rwxrwxr-x. 1 1000 1000 405505 Oct 20 00:16 configure
-rw-rw-r--. 1 1000 1000  98443 Oct 20 00:09 configure.ac
-rw-rw-r--. 1 1000 1000    823 Aug 16  2018 CONTRIBUTORS
-rw-rw-r--. 1 1000 1000  18092 Aug 16  2018 COPYING
-rwxr-xr-x. 1 1000 1000  23567 Mar 26  2018 depcomp
drwxrwxr-x. 5 1000 1000    210 Oct 20 00:16 doc
drwxrwxr-x. 3 1000 1000    205 Oct 20 00:16 genhash
-rw-rw-r--. 1 1000 1000   8218 Jul 18 04:10 INSTALL
-rwxr-xr-x. 1 1000 1000  15155 Mar 26  2018 install-sh
drwxrwxr-x. 9 1000 1000    173 Oct 20 00:16 keepalived
-rw-rw-r--. 1 1000 1000   9878 Apr  3  2019 keepalived.spec.in
drwxrwxr-x. 2 1000 1000   4096 Oct 20 00:16 lib
-rw-rw-r--. 1 1000 1000   1807 Feb  3  2019 Makefile.am
-rw-rw-r--. 1 1000 1000  28929 Oct 20 00:16 Makefile.in
-rwxr-xr-x. 1 1000 1000   6872 Mar 26  2018 missing
-rw-rw-r--. 1 1000 1000   2083 Oct 17 01:21 README.md
drwxrwxr-x. 3 1000 1000     41 May  9  2019 snap
-rw-rw-r--. 1 1000 1000   5908 Aug 17  2018 TODO

執行 ./configure --prefix=/kp/keepalived 配置安裝路徑

如果提示

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

需要安裝依賴軟件 yum -y install libnl libnl-devel

如果提示

configure: error: libnfnetlink headers missing

需要安裝依賴軟件 yum install -y libnfnetlink-devel

再執行 ./configure --prefix=/kp/keepalived

最後 make && make install

編譯安裝成功後，會自動在路徑/usr/lib/systemd/system/下生成keepalived.service文件

[root@pve-97 keepalived]# ll /usr/lib/systemd/system/|grep keepalive
-rw-r--r--. 1 root root  398 Jan  8 17:25 keepalived.service

三、配置開機啓動服務

keepalived默認執行/etc/keepalived/keepalived.conf,所以先創建該目錄並拷貝配置

mkdir /etc/keepalived

cp /kp/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

cp /kp/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/keepalived

cp /kp/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

配置開啓啓動

systemctl enable keepalived.service

四、修改keepalived配置

修改 /etc/keepalived/keepalived.conf 以下是修改後keepalived.conf全部配置,加#號註釋的爲需要修改的地方。

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id pve-97
   vrrp_skip_check_adv_addr
#   vrrp_strict              #這個要註釋掉，不然會ping不通 vip
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script chk_nginx {          # 檢測nginx腳本 定義
    script "/kp/keepalived/check_nginx_pid.sh"    #最後手動執行下此腳本，以確保此腳本能夠正常執行
    interval 2                          #（檢測腳本執行的間隔，單位是秒）
    weight 2
}

vrrp_instance VI_1 {
    # 指定keepalived的角色，“MASTER”表示此主機是主服務器，“BACKUP”表示此主機是備用服務器
    state MASTER
    # 指定網卡接口，這裏改爲我們當前使用的網卡
    interface ens18
    # 虛擬路由標識，這個標識是一個數字，同一個vrrp實例使用唯一的標識
    # 即同一vrrp_instance下，MASTER和BACKUP必須是一致的
    virtual_router_id 51
    # 定義優先級；數字越大，優先級越高（0-255）
    # 在同一個vrrp_instance下，“MASTER”的優先級必須大於“BACKUP”的優先級
    priority 100
    # 設定MASTER與BACKUP負載均衡器之間同步檢查的時間間隔，單位是秒
    advert_int 1
    # 設置驗證類型和密碼
    authentication {
        # 設置驗證類型，主要有PASS和AH兩種
        auth_type PASS
        # 設置驗證密碼，在同一個vrrp_instance下，MASTER與BACKUP必須使用相同的密碼才能正常通信
        auth_pass 1111
    }
    
    virtual_ipaddress {
        # 虛擬IP爲10.10.0.10/8;綁定接口爲ens18;別名ha:net，主備相同
        192.168.1.91 dev ens18 label ha:net
    }
    
    track_script {
        chk_nginx            #調用檢測nginx腳本
    }
}

使用命令 systemctl start keepalived.service 啓動keepalived。

使用命令 ip addr show | grep inet 前後觀察，可以看到 VIP 192.168.1.91已經綁定。

[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 
[root@pve-97 ~]# systemctl start keepalived.service
[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

可以通過另一臺服務器來ping 192.168.1.91

[root@pve-98 keepalived]# ping 192.168.1.91
PING 192.168.1.91 (192.168.1.91) 56(84) bytes of data.
64 bytes from 192.168.1.91: icmp_seq=1 ttl=64 time=0.545 ms
64 bytes from 192.168.1.91: icmp_seq=2 ttl=64 time=0.240 ms
64 bytes from 192.168.1.91: icmp_seq=3 ttl=64 time=0.218 ms
64 bytes from 192.168.1.91: icmp_seq=4 ttl=64 time=0.254 ms

五、部署備份服務器

備份服務器同樣部署，只是配置需要修改，其中 priority 要低於MASTER 的配置值

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id pve-98
   vrrp_skip_check_adv_addr
#   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script chk_nginx {          # 檢測nginx腳本 定義
    script "/kp/keepalived/check_nginx_pid.sh"    #最後手動執行下此腳本，以確保此腳本能夠正常執行
    interval 2                          #（檢測腳本執行的間隔，單位是秒）
    weight 2
}

vrrp_instance VI_1 {
    # 指定keepalived的角色，“MASTER”表示此主機是主服務器，“BACKUP”表示此主機是備用服務器
    state BACKUP
    # 指定網卡接口，這裏改爲我們當前使用的網卡
    interface ens18
    # 虛擬路由標識，這個標識是一個數字，同一個vrrp實例使用唯一的標識
    # 即同一vrrp_instance下，MASTER和BACKUP必須是一致的
    virtual_router_id 51
    # 定義優先級；數字越大，優先級越高（0-255）
    # 在同一個vrrp_instance下，“MASTER”的優先級必須大於“BACKUP”的優先級
    priority 50
    # 設定MASTER與BACKUP負載均衡器之間同步檢查的時間間隔，單位是秒
    advert_int 1
    # 設置驗證類型和密碼
    authentication {
        # 設置驗證類型，主要有PASS和AH兩種
        auth_type PASS
        # 設置驗證密碼，在同一個vrrp_instance下，MASTER與BACKUP必須使用相同的密碼才能正常通信
        auth_pass 1111
    }
    # 有故障時是否激活郵件通知
    #smtp_alert
    # 禁止搶佔服務
    # 默認情況，當MASTER服務掛掉之後，BACKUP自動升級爲MASTER並接替它的任務
    # 當MASTER服務恢復後，升級爲MASTER的BACKUP服務又自動降爲BACKUP，把工作權交給原MASTER
    # 當配置了nopreempt，MASTER從掛掉到恢復，不再將服務搶佔過來。
    #nopreempt
    # 虛擬IP，兩個節點設置必須一樣。可以設置多個，一行寫一個

    virtual_ipaddress {
        # 虛擬IP爲10.10.0.10/8;綁定接口爲ens18;別名ha:net，主備相同
        192.168.1.91 dev ens18 label ha:net
    }
    
    track_script {
        chk_nginx            #調用檢測nginx腳本
    }
}

六、驗證高可用性

1、模擬宕機

使用 arp -a 命令可以看到 浮動ip 192.168.1.91 與 192.168.1.97 的mac 地址相同，說明綁定在97服務器上

通過瀏覽器訪問，可以輕鬆訪問到97的nginx 

把97服務器keepalived 停止，浮動ip 192.168.1.91 飄在 192.168.1.98 上

[root@pve-97 ~]# systemctl stop keepalived.service
[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 
[root@pve-98 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.98/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:e8d:1b19:6be2:930f/64 scope global noprefixroute dynamic 
    inet6 fec0::f:e831:5c3b:a61f:e311/64 scope site noprefixroute dynamic 
    inet6 fe80::e48:6d46:5d45:6f37/64 scope link noprefixroute 

192.168.1.91 與 192.168.1.98 的mac 地址相同

2、模擬單臺nginx不可用

破壞nginx的配置文件，使其無法正常啓動，比如加個無效字符串

events {
    worker_connections  1024;
}

kp
http {
    include       mime.types;
    default_type  application/octet-stream;

調用檢測腳本/kp/keepalived/check_nginx_pid.sh，發現nginx報錯

[root@pve-97 keepalived]# /kp/keepalived/check_nginx_pid.sh 
nginx: [emerg] unknown directive "kp" in /usr/local/nginx/conf/nginx.conf:17

keepalived 被自己通過 檢測腳本停止，查看檢測日誌 more /kp/keepalived/check_ng.log

[root@pve-97 keepalived]# more check_ng.log 
2020/01/10-09:41:25 nginx down,keepalived will stop
2020/01/10-09:41:27 nginx down,keepalived will stop
2020/01/10-09:41:29 nginx down,keepalived will stop
2020/01/10-09:41:31 nginx down,keepalived will stop

查看keepalived狀態和ip信息，已經切換

[root@pve-97 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2020-01-10 09:43:01 CST; 16min ago
  Process: 6023 ExecStart=/kp/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 6024 (code=exited, status=0/SUCCESS)

Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: Assigned address fe80::338d:1893:770:6678 for interface ens18
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: Registering gratuitous ARP shared channel
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: (VI_1) removing VIPs.
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: (VI_1) Entering BACKUP STATE (init)
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: VRRP sockpool: [ifindex(2), family(IPv4), proto(112), unicast(0), fd(11,12)]
Jan 10 09:43:00 pve-97 systemd[1]: Stopping LVS and VRRP High Availability Monitor...
Jan 10 09:43:00 pve-97 Keepalived[6024]: Stopping
Jan 10 09:43:01 pve-97 Keepalived_vrrp[6025]: Stopped - used 0.003279 user time, 0.000000 system time
Jan 10 09:43:01 pve-97 Keepalived[6024]: Stopped Keepalived v2.0.19 (10/19,2019)
Jan 10 09:43:01 pve-97 systemd[1]: Stopped LVS and VRRP High Availability Monitor.
[root@pve-97 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

恢復nginx的配置文件，啓動keepalived,VIP正常飄回

[root@pve-97 keepalived]# systemctl start keepalived
[root@pve-97 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

七、非CentOS配置腳本

在路徑/kp/keepalived/check_nginx_pid.sh腳本

#!/bin/bash
#時間變量，用於記錄日誌
d=`date --date today +%Y/%m/%d-%H:%M:%S`
#計算nginx進程數量
n=`ps -C nginx --no-heading|wc -l`
#如果進程爲0，則啓動nginx，並且再次檢測nginx進程數量，
if [ $n -eq "0" ]; then
        /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf #嘗試啓動nginx
        n2=`ps -C nginx --no-heading|wc -l`
        #如果還爲0，說明nginx無法啓動，此時需要關閉keepalived
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /etc/keepalived/check_ng.log
                service keepalived stop # 停止keepalived
        fi
fi