基於keepalived+sentinel+redis主從的redis高可用性架構方案

基於keepalived+sentinel+redis主從的redis高可用性架構方案

環境說明

在一般小項目中的我們常用redis主從模式,redis數據庫是單進程單線程的架構,使用單機或簡單的主從模式,來儘量保證緩存數據庫的可持久化,當單節點出問題時,我們可以從節點來獲取數據。
而當在規模大的項目或生產環境要求高的時,一般redis高可用性架構。 一般來說,redis高可用性架構有在致三類高可用性: redis主從模式、redis哨兵模式、redis cluster模式。

redis主從模式:設置一個redis server master 主機,一個redis server slave(從機)。 從機定期從主機同步數據。 一般使用的配置是master端提供的讀寫服務,而在slave端提供只讀服務。中心化的架構
redis哨兵模式:Redis-Sentinel是Redis官方推薦的高可用性(HA)解決方案,當用Redis做Master-slave的高可用方案時,假如master宕機了,Redis本身(包括它的很多客戶端)都沒有實現自動進行主備切換,而Redis-sentinel本身也是一個獨立運行的進程,它能監控多個master-slave集羣,發現master宕機後能進行自動切換。Sentinel由一個或多個Sentinel 實例 組成的Sentinel 系統可以監視任意多個主服務器,以及這些主服務器屬下的所有從服務器,並在被監視的主服務器進入下線狀態時,自動將下線主服務器屬下的某個從服務器升級爲新的主服務器。 中心化的架構。
redis cluster模式:redis cluster在設計的時候,就考慮到了去中心化,也就是說,集羣中的每個節點都是平等的關係,都是對等的,每個節點都保存各自的數據和整個集羣的狀態。每個節點都和其他所有節點連接,而且這些連接保持活躍,這樣就保證了我們只需要連接集羣中的任意一個節點,就可以獲取到其他節點的數據。去中心化的架構。

這次高可用性實驗,我們實踐基於keepalived+redis-sentinel+redis(master-slave)實現的高可用性架構方案,通過keepalived來實現對外統一的VIP服務,當集羣中redis節點出現問題時,則可自動進行故障切換、VIP飄逸,以此來實現對應用端的高可用性訪問。

環境信息

操作中系統 ip 主機名 軟件
Linux7.5 192.168.174.129 srvb keepalived+sentinel+redis (master)
Linux7.5 192.168.174.130 srvc keepalived+sentinel+redis(slave)
Linux7.5 192.168.174.131 srvd keepalivedd+sentinel+redis

對外提供應用連接的IP地址(VIP)爲:192.168.174.130

架構圖
在這裏插入圖片描述

環境搭建

安裝redis

wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar zxvf redis-5.0.5.tar.gz
cd redis-5.0.5
make PREFIX=/usr/local/redis install # 將redis安裝/usr/local/redis目錄

配置命令行引用
ln -s /usr/local/redis/bin/redis-server /usr/bin/redis-server
ln -s /usr/local/redis/bin/redis-server /usr/bin/redis-sentinel
ln -s /usr/local/redis/bin/redis-cli /usr/bin/redis-cli

以上srvb srvc srvd三個節點都執行相應的操作

安裝keepalived

目錄規劃:
/usr/local/keepalived #安裝目錄

下載:
https://www.keepalived.org/download.html

cd /usr/local/src/keepalived-1.2.6
configure --prefix=/usr/local/keepalived
make
make install
查看安裝後文件:
ls -l /usr/local/keepalived/
總用量 16
drwxr-xr-x 2 root root 4096 8月 29 14:46 bin
drwxr-xr-x 5 root root 4096 8月 29 14:46 etc
drwxr-xr-x 2 root root 4096 8月 29 14:46 sbin
drwxr-xr-x 3 root root 4096 8月 29 14:46 share

安裝後文件說明:
etc配置示例及開機啓動的配置文件
ls -l etc
drwxr-xr-x 3 root root 4096 8月 29 14:46 keepalived
drwxr-xr-x 3 root root 4096 8月 29 14:46 rc.d
drwxr-xr-x 2 root root 4096 8月 29 14:46 sysconfig

安裝之後的目錄結構:
.
├── bin
│ └── genhash
├── etc
│ ├── keepalived
│ │ ├── keepalived.conf
│ │ └── samples
│ │ ├── client.pem
│ │ ├── dh1024.pem
│ │ ├── keepalived.conf.fwmark
│ │ ├── keepalived.conf.HTTP_GET.port
│ │ ├── keepalived.conf.inhibit
│ │ ├── keepalived.conf.IPv6
│ │ ├── keepalived.conf.misc_check
│ │ ├── keepalived.conf.misc_check_arg
│ │ ├── keepalived.conf.quorum
│ │ ├── keepalived.conf.sample
│ │ ├── keepalived.conf.SMTP_CHECK
│ │ ├── keepalived.conf.SSL_GET
│ │ ├── keepalived.conf.status_code
│ │ ├── keepalived.conf.track_interface
│ │ ├── keepalived.conf.virtualhost
│ │ ├── keepalived.conf.virtual_server_group
│ │ ├── keepalived.conf.vrrp
│ │ ├── keepalived.conf.vrrp.localcheck
│ │ ├── keepalived.conf.vrrp.lvs_syncd
│ │ ├── keepalived.conf.vrrp.routes
│ │ ├── keepalived.conf.vrrp.scripts
│ │ ├── keepalived.conf.vrrp.static_ipaddress
│ │ ├── keepalived.conf.vrrp.sync
│ │ ├── root.pem
│ │ └── sample.misccheck.smbcheck.sh
│ ├── mkdir
│ ├── rc.d
│ │ └── init.d
│ │ └── keepalived
│ └── sysconfig
│ └── keepalived
├── sbin
│ └── keepalived
└── share
└── man
├── man1
│ └── genhash.1
├── man5
│ └── keepalived.conf.5
└── man8
└── keepalived.8

sbin/keepalived keepalived執行文件
bin/gensh 執行文件

將keepalived註冊爲系統服務:

cp /usr/local/keepalived/sbin/keepalived /usr/local/sbin
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig

etc/sysconfig/keepalived 這個配置文件,在使用service管理啓動服務的時候,默認的從這個文件中讀取啓動參數。也就是啓動腳本的參數文件。
讀取KEEPALIVED_OPTIONS中配置的參數來啓動

通過查看/etc/rc.d/init.d/keepalived執行腳本內容
我們可以分析keepalived-1.2.6版本的啓動配置文件爲 /etc/keepalived/keepalived.conf配置
腳本部分內容:

cat /etc/rc.d/init.d/keepalived
#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid  #PID文件存放的位置
# config: /etc/keepalived/keepalived.conf  # 這個配置文件存放的目錄,所以我新建這個目錄
# chkconfig: - 21 79
# description: Start and stop Keepalived

# Source function library
. /etc/rc.d/init.d/functions

# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived

RETVAL=0

prog="keepalived"

start() {
    echo -n $"Starting $prog: "
    daemon keepalived ${KEEPALIVED_OPTIONS}
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}

stop() {
    echo -n $"Stopping $prog: "
    killproc keepalived
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}

reload() {
    echo -n $"Reloading $prog: "
    killproc keepalived -1
    RETVAL=$?
    echo
}

mkdir -p /etc/keepalived/
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

/usr/local/keepalived/etc/keepalived/keepalived.conf 這個文件是安裝之後的示例配置文件。 我們需要修改的內容,可以根據這個進行相應的修改

啓動:
service keepalived start
默認會有三個進程,一個father進程,兩個子進程。
father負責監控兩個子進程,分別是VRRP和checkers

存在的問題:
問題1:
[root@JHGYlmsHSTEST init.d]# service keepalived start
正在啓動 keepalived:/bin/bash: keepalived: command not found
[失敗]

原因和處理:
cp /usr/local/keepalived/sbin/keepalived /usr/sbin

使用serivce管理服務時,則執行命令必有是在/usr/sbin目錄下纔可以的。

問題2:
2、MASTER主機message日誌中不斷有Received lower prio advert, forcing new election記錄:
原因:防火牆導致
解決方法:1、關閉防火牆;
2、或者在防火牆中開放目標主機中的數據包

配置redis主從

配置主從的架構

目錄規劃:
配置文件存放:/usr/local/redis/conf
數據文件存放:/usr/local/redis/data

redis master配置文件:

#master : redis.conf文件
port 6379
bind 0.0.0.0
pidfile "/var/run/redis_6379.pid"
#cluster-enabled yes
#cluster-config-file nodes.conf
#cluster-node-timeout 5000
appendonly yes
daemonize yes
#requirepass redis

dbfilename "dump.rdb"
dir "/usr/local/redis/data"
save 900 1
save 300 10
save 60 1000
# Generated by CONFIG REWRITE
#replicaof 192.168.174.131 6379

redis slave配置文件

[root@srvc ~]# cat /usr/local/redis/conf/redis.conf 
port 6379
bind 0.0.0.0
pidfile "/var/run/redis_6379.pid"
#cluster-enabled yes
#cluster-config-file nodes.conf
#cluster-node-timeout 5000
appendonly yes
daemonize yes
#requirepass redis

dbfilename "dump.rdb"
dir "/usr/local/redis/data"
save 900 1
save 300 10
save 60 1000
slaveof 192.168.174.129 6379

啓動主從模式
start_redis.sh

[root@srvb ~]# cat redis/start_redis.sh 

REDIS_DIR=/usr/local/redis

redis-server ${REDIS_DIR}/conf/redis.conf

ssh srvc <<EOF
redis-server ${REDIS_DIR}/conf/redis.conf
EOF

查看redis主從狀態:

[root@srvb redis]# cat chk_redis.sh 
redis-cli -p 6379 info|grep role
[root@srvb redis]# ./chk_redis.sh 
role:master

配置redis-sentinel哨兵模式

配置文件:
/usr/local/redis/conf/sentinel
端口規劃:
我們在此環境中設置redis-sentinel使用的是27000端口

配置文件內容

[root@srvb redis]# cat /usr/local/redis/conf/sentinel.conf 
port 27000
bind 0.0.0.0
daemonize yes
dir "/usr/local/redis/data"
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 192.168.174.129 6379 2
sentinel config-epoch mymaster 4
sentinel leader-epoch mymaster 4
sentinel known-replica mymaster 192.168.174.130 6379

在srvb srvc srvd三個主機都需要做相同的配置。 注意:這裏我們使用的是/usr/local/redis/data做爲sentinel的數據存放目錄,因爲沒有配置緩存數據的持久化文件rdb或aof文件。所以可以直接使用。如果需要將緩存數據持久化,則需要指定不同的rdb和aof文件名來區分是redis還是的sentinel的的持久化文件

啓動三個節點的sentinel:

[root@srvb redis]# cat start_sentinel.sh 

REDIS_DIR=/usr/local/redis

redis-sentinel ${REDIS_DIR}/conf/sentinel.conf

ssh srvc <<EOF
redis-sentinel ${REDIS_DIR}/conf/sentinel.conf
EOF

ssh srvd <<EOF
redis-sentinel ${REDIS_DIR}/conf/sentinel.conf
EOF

查看redis-sentinel狀態:
redis-cli -p 27000
info sentinel

# 在主節點執行
127.0.0.1:27000> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.174.129:6379,slaves=2,sentinels=3

配置keepalived

配置文件目錄:
/etc/keepalived
配置文件
keepalived.conf

查看配置文件內容

srvb(redis-master)

[root@srvb redis]# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id HA
}

vrrp_script check_redis {
        script "redis-cli -h 192.168.174.129 -p 6379 info|grep role:master >/dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 100
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

srvc(redis-slave):

[root@srvc ~]# cat redis/start_redis.sh 

REDIS_DIR=/usr/local/redis

redis-server ${REDIS_DIR}/conf/redis.conf

ssh srvc <<EOF
redis-server ${REDIS_DIR}/conf/redis.conf
EOF

ssh srvd <<EOF
redis-server ${REDIS_DIR}/conf/redis.conf
EOF
[root@srvc ~]# 
[root@srvc ~]#  cat /etc/keepalived/keepalived.conf
global_defs {
   router_id HA
}

vrrp_script check_redis {
        script "redis-cli -h 192.168.174.130 -p 6379 info|grep role:master >/dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 90
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

srvd: 第三個節點的keepalived
這個節點上只部署有keepalived+sentinel

配置文件內容:

[root@srvd ~]# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id HA
}

vrrp_script check_redis {
        script "redis-cli -h 192.168.174.130 -p 6379 info|grep role:master >/dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 80
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

三個節點的keepalived的配置通過優先級的值的不同來設置,srvb是redis-master所在節點,在初始階段我們將設置優先級最高。

啓動keepalived:

[root@srvb redis]# cat start_keepalived.sh 
service keepalived start

ssh srvb << EOF
service keepalived start
EOF

ssh srvc << EOF
service keepalived start
EOF
[root@srvb redis]# 

啓動之後查看
srvb節點:

# ip a
[root@srvb redis]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:c3:d5:ce brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.129/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.174.137/24 scope global secondary eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fec3:d5ce/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:e2:5a:ca brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:e2:5a:ca brd ff:ff:ff:ff:ff:ff
[root@srvb redis]# 

srvc節點

[root@srvc ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:de:98:68 brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.130/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fede:9868/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff

srvd節點

[root@srvd ~]#  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d5:e2:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.131/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed5:e28b/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:1f:76:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:1f:76:64 brd ff:ff:ff:ff:ff:ff

至此,keepalived+sentinel+redis(master-slave)高可用性架構搭建完成。

切換測試

關閉主redis-master:
在srvb上執行

[root@srvb redis]# ps -ef|grep redis
root      22580      1  0 20:03 ?        00:00:53 redis-server 0.0.0.0:6379
root      22628      1  0 20:05 ?        00:01:11 redis-sentinel 0.0.0.0:27000 [sentinel]
root      58417  44097  0 22:28 pts/3    00:00:00 grep --color=auto redis
[root@srvb redis]# redis-cli -p 6379 shutdown 
[root@srvb redis]# ps -ef|grep redis
root      22628      1  0 20:05 ?        00:01:12 redis-sentinel 0.0.0.0:27000 [sentinel]
root      58524  44097  0 22:29 pts/3    00:00:00 grep --color=au

至srvc上查看slave提升爲主:

[root@srvc redis]# cat chk_redis.sh 
redis-cli -p 6379 info|grep role
[root@srvc redis]# ./chk_redis.sh 
role:master

可以看到此時備的slave上已提升爲主了。

查看VIP是否切換到srvc節點上:

[root@srvc redis]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:de:98:68 brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.130/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.174.137/24 scope global secondary eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fede:9868/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff

可以看到:
inet 192.168.174.137/24 scope global secondary eno16777736
valid_lft forever preferred_lft forever

VIP地址已從srvb切換到srvc節點上。 對前端應用來說這種是平滑的切換。
業務可能感知到短時間的中斷,但是很快會恢復。

補充說明

1、當redis-sentinel模式檢測並發現redis服務主從切換時,在這個過程中會修改原主從redis的配置文件,配置修改成切換後的應用狀態。以便服務恢復時,能夠按照新的配置來啓動主從架構
2、當啓動redis-sentinel時,本身的sentinel.conf配置文件會加入一此些集羣的配置。

發佈了63 篇原創文章 · 獲贊 6 · 訪問量 5萬+
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章