linux下安装部署keepalived，利用keepalived多台nginx虚拟IP浮动

一、准备

软件包名称：keepalived  2.0.19

操作系统环境

演示环境为centos7.7-x86_64-minimal 操作系统

 

IP	部署	说明
192.168.1.91	--	虚IP映射域名
192.168.1.97	keepalived	主要节点
192.168.1.98	keepalived	备份节点

二、keepalived 编译安装

上传keepalived安装包至相应目录，这里我们指定 /kp/keepalived

[root@pve-97 keepalived]# pwd
/kp/keepalived
[root@pve-97 keepalived]# ll
total 1004
-rw-r--r--. 1 root root 1025062 Jan  8 16:59 keepalived-2.0.19.tar.gz

解压命令：tar -zxvf keepalived-2.0.19.tar.gz 并进入目录

[root@pve-97 keepalived-2.0.19]# ll
total 1212
-rw-rw-r--. 1 1000 1000  54387 Oct 20 00:16 aclocal.m4
-rwxr-xr-x. 1 1000 1000   5826 Mar 26  2018 ar-lib
-rw-rw-r--. 1 1000 1000     41 Aug 16  2018 AUTHOR
drwxrwxr-x. 2 1000 1000     44 Oct 20 00:16 bin_install
-rwxrwxr-x. 1 1000 1000     64 Aug 16  2018 build_setup
-rw-rw-r--. 1 1000 1000 494050 Oct 20 00:08 ChangeLog
-rwxr-xr-x. 1 1000 1000   7333 Mar 26  2018 compile
-rwxrwxr-x. 1 1000 1000 405505 Oct 20 00:16 configure
-rw-rw-r--. 1 1000 1000  98443 Oct 20 00:09 configure.ac
-rw-rw-r--. 1 1000 1000    823 Aug 16  2018 CONTRIBUTORS
-rw-rw-r--. 1 1000 1000  18092 Aug 16  2018 COPYING
-rwxr-xr-x. 1 1000 1000  23567 Mar 26  2018 depcomp
drwxrwxr-x. 5 1000 1000    210 Oct 20 00:16 doc
drwxrwxr-x. 3 1000 1000    205 Oct 20 00:16 genhash
-rw-rw-r--. 1 1000 1000   8218 Jul 18 04:10 INSTALL
-rwxr-xr-x. 1 1000 1000  15155 Mar 26  2018 install-sh
drwxrwxr-x. 9 1000 1000    173 Oct 20 00:16 keepalived
-rw-rw-r--. 1 1000 1000   9878 Apr  3  2019 keepalived.spec.in
drwxrwxr-x. 2 1000 1000   4096 Oct 20 00:16 lib
-rw-rw-r--. 1 1000 1000   1807 Feb  3  2019 Makefile.am
-rw-rw-r--. 1 1000 1000  28929 Oct 20 00:16 Makefile.in
-rwxr-xr-x. 1 1000 1000   6872 Mar 26  2018 missing
-rw-rw-r--. 1 1000 1000   2083 Oct 17 01:21 README.md
drwxrwxr-x. 3 1000 1000     41 May  9  2019 snap
-rw-rw-r--. 1 1000 1000   5908 Aug 17  2018 TODO

执行 ./configure --prefix=/kp/keepalived 配置安装路径

如果提示

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

需要安装依赖软件 yum -y install libnl libnl-devel

如果提示

configure: error: libnfnetlink headers missing

需要安装依赖软件 yum install -y libnfnetlink-devel

再执行 ./configure --prefix=/kp/keepalived

最后 make && make install

编译安装成功后，会自动在路径/usr/lib/systemd/system/下生成keepalived.service文件

[root@pve-97 keepalived]# ll /usr/lib/systemd/system/|grep keepalive
-rw-r--r--. 1 root root  398 Jan  8 17:25 keepalived.service

三、配置开机启动服务

keepalived默认执行/etc/keepalived/keepalived.conf,所以先创建该目录并拷贝配置

mkdir /etc/keepalived

cp /kp/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

cp /kp/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/keepalived

cp /kp/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

配置开启启动

systemctl enable keepalived.service

四、修改keepalived配置

修改 /etc/keepalived/keepalived.conf 以下是修改后keepalived.conf全部配置,加#号注释的为需要修改的地方。

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id pve-97
   vrrp_skip_check_adv_addr
#   vrrp_strict              #这个要注释掉，不然会ping不通 vip
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script chk_nginx {          # 检测nginx脚本 定义
    script "/kp/keepalived/check_nginx_pid.sh"    #最后手动执行下此脚本，以确保此脚本能够正常执行
    interval 2                          #（检测脚本执行的间隔，单位是秒）
    weight 2
}

vrrp_instance VI_1 {
    # 指定keepalived的角色，“MASTER”表示此主机是主服务器，“BACKUP”表示此主机是备用服务器
    state MASTER
    # 指定网卡接口，这里改为我们当前使用的网卡
    interface ens18
    # 虚拟路由标识，这个标识是一个数字，同一个vrrp实例使用唯一的标识
    # 即同一vrrp_instance下，MASTER和BACKUP必须是一致的
    virtual_router_id 51
    # 定义优先级；数字越大，优先级越高（0-255）
    # 在同一个vrrp_instance下，“MASTER”的优先级必须大于“BACKUP”的优先级
    priority 100
    # 设定MASTER与BACKUP负载均衡器之间同步检查的时间间隔，单位是秒
    advert_int 1
    # 设置验证类型和密码
    authentication {
        # 设置验证类型，主要有PASS和AH两种
        auth_type PASS
        # 设置验证密码，在同一个vrrp_instance下，MASTER与BACKUP必须使用相同的密码才能正常通信
        auth_pass 1111
    }
    
    virtual_ipaddress {
        # 虚拟IP为10.10.0.10/8;绑定接口为ens18;别名ha:net，主备相同
        192.168.1.91 dev ens18 label ha:net
    }
    
    track_script {
        chk_nginx            #调用检测nginx脚本
    }
}

使用命令 systemctl start keepalived.service 启动keepalived。

使用命令 ip addr show | grep inet 前后观察，可以看到 VIP 192.168.1.91已经绑定。

[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 
[root@pve-97 ~]# systemctl start keepalived.service
[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

可以通过另一台服务器来ping 192.168.1.91

[root@pve-98 keepalived]# ping 192.168.1.91
PING 192.168.1.91 (192.168.1.91) 56(84) bytes of data.
64 bytes from 192.168.1.91: icmp_seq=1 ttl=64 time=0.545 ms
64 bytes from 192.168.1.91: icmp_seq=2 ttl=64 time=0.240 ms
64 bytes from 192.168.1.91: icmp_seq=3 ttl=64 time=0.218 ms
64 bytes from 192.168.1.91: icmp_seq=4 ttl=64 time=0.254 ms

五、部署备份服务器

备份服务器同样部署，只是配置需要修改，其中 priority 要低于MASTER 的配置值

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id pve-98
   vrrp_skip_check_adv_addr
#   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script chk_nginx {          # 检测nginx脚本 定义
    script "/kp/keepalived/check_nginx_pid.sh"    #最后手动执行下此脚本，以确保此脚本能够正常执行
    interval 2                          #（检测脚本执行的间隔，单位是秒）
    weight 2
}

vrrp_instance VI_1 {
    # 指定keepalived的角色，“MASTER”表示此主机是主服务器，“BACKUP”表示此主机是备用服务器
    state BACKUP
    # 指定网卡接口，这里改为我们当前使用的网卡
    interface ens18
    # 虚拟路由标识，这个标识是一个数字，同一个vrrp实例使用唯一的标识
    # 即同一vrrp_instance下，MASTER和BACKUP必须是一致的
    virtual_router_id 51
    # 定义优先级；数字越大，优先级越高（0-255）
    # 在同一个vrrp_instance下，“MASTER”的优先级必须大于“BACKUP”的优先级
    priority 50
    # 设定MASTER与BACKUP负载均衡器之间同步检查的时间间隔，单位是秒
    advert_int 1
    # 设置验证类型和密码
    authentication {
        # 设置验证类型，主要有PASS和AH两种
        auth_type PASS
        # 设置验证密码，在同一个vrrp_instance下，MASTER与BACKUP必须使用相同的密码才能正常通信
        auth_pass 1111
    }
    # 有故障时是否激活邮件通知
    #smtp_alert
    # 禁止抢占服务
    # 默认情况，当MASTER服务挂掉之后，BACKUP自动升级为MASTER并接替它的任务
    # 当MASTER服务恢复后，升级为MASTER的BACKUP服务又自动降为BACKUP，把工作权交给原MASTER
    # 当配置了nopreempt，MASTER从挂掉到恢复，不再将服务抢占过来。
    #nopreempt
    # 虚拟IP，两个节点设置必须一样。可以设置多个，一行写一个

    virtual_ipaddress {
        # 虚拟IP为10.10.0.10/8;绑定接口为ens18;别名ha:net，主备相同
        192.168.1.91 dev ens18 label ha:net
    }
    
    track_script {
        chk_nginx            #调用检测nginx脚本
    }
}

六、验证高可用性

1、模拟宕机

使用 arp -a 命令可以看到 浮动ip 192.168.1.91 与 192.168.1.97 的mac 地址相同，说明绑定在97服务器上

通过浏览器访问，可以轻松访问到97的nginx 

把97服务器keepalived 停止，浮动ip 192.168.1.91 飘在 192.168.1.98 上

[root@pve-97 ~]# systemctl stop keepalived.service
[root@pve-97 ~]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 
[root@pve-98 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.98/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:e8d:1b19:6be2:930f/64 scope global noprefixroute dynamic 
    inet6 fec0::f:e831:5c3b:a61f:e311/64 scope site noprefixroute dynamic 
    inet6 fe80::e48:6d46:5d45:6f37/64 scope link noprefixroute 

192.168.1.91 与 192.168.1.98 的mac 地址相同

2、模拟单台nginx不可用

破坏nginx的配置文件，使其无法正常启动，比如加个无效字符串

events {
    worker_connections  1024;
}

kp
http {
    include       mime.types;
    default_type  application/octet-stream;

调用检测脚本/kp/keepalived/check_nginx_pid.sh，发现nginx报错

[root@pve-97 keepalived]# /kp/keepalived/check_nginx_pid.sh 
nginx: [emerg] unknown directive "kp" in /usr/local/nginx/conf/nginx.conf:17

keepalived 被自己通过 检测脚本停止，查看检测日志 more /kp/keepalived/check_ng.log

[root@pve-97 keepalived]# more check_ng.log 
2020/01/10-09:41:25 nginx down,keepalived will stop
2020/01/10-09:41:27 nginx down,keepalived will stop
2020/01/10-09:41:29 nginx down,keepalived will stop
2020/01/10-09:41:31 nginx down,keepalived will stop

查看keepalived状态和ip信息，已经切换

[root@pve-97 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2020-01-10 09:43:01 CST; 16min ago
  Process: 6023 ExecStart=/kp/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 6024 (code=exited, status=0/SUCCESS)

Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: Assigned address fe80::338d:1893:770:6678 for interface ens18
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: Registering gratuitous ARP shared channel
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: (VI_1) removing VIPs.
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: (VI_1) Entering BACKUP STATE (init)
Jan 10 09:43:00 pve-97 Keepalived_vrrp[6025]: VRRP sockpool: [ifindex(2), family(IPv4), proto(112), unicast(0), fd(11,12)]
Jan 10 09:43:00 pve-97 systemd[1]: Stopping LVS and VRRP High Availability Monitor...
Jan 10 09:43:00 pve-97 Keepalived[6024]: Stopping
Jan 10 09:43:01 pve-97 Keepalived_vrrp[6025]: Stopped - used 0.003279 user time, 0.000000 system time
Jan 10 09:43:01 pve-97 Keepalived[6024]: Stopped Keepalived v2.0.19 (10/19,2019)
Jan 10 09:43:01 pve-97 systemd[1]: Stopped LVS and VRRP High Availability Monitor.
[root@pve-97 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

恢复nginx的配置文件，启动keepalived,VIP正常飘回

[root@pve-97 keepalived]# systemctl start keepalived
[root@pve-97 keepalived]# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.1.97/24 brd 192.168.1.255 scope global noprefixroute ens18
    inet 192.168.1.91/32 scope global ha:net
    inet6 2002:c064:6401:f:a35e:b5ec:a220:c79d/64 scope global noprefixroute dynamic 
    inet6 fec0::f:bad3:87a4:760d:3c0b/64 scope site noprefixroute dynamic 
    inet6 fe80::338d:1893:770:6678/64 scope link noprefixroute 

七、非CentOS配置脚本

在路径/kp/keepalived/check_nginx_pid.sh脚本

#!/bin/bash
#时间变量，用于记录日志
d=`date --date today +%Y/%m/%d-%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0，则启动nginx，并且再次检测nginx进程数量，
if [ $n -eq "0" ]; then
        /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf #尝试启动nginx
        n2=`ps -C nginx --no-heading|wc -l`
        #如果还为0，说明nginx无法启动，此时需要关闭keepalived
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /etc/keepalived/check_ng.log
                service keepalived stop # 停止keepalived
        fi
fi