Redis集羣方案redis-twemproxy-keepalived

轉載原文自：http://ju.outofmemory.cn/entry/60425

redis現在的版本還不支持分佈式，這會在3.0版本加上。Redis目前只支持“主從同步”和“從從同步”，所以需要根據需求應用自己來實現。

tp ------ 192.168.10.15

tp1------ 192.168.10.18

tp tp1 虛擬vip: 192.168.10.243

 

v1 ------ 192.168.10.224

v2 ------ 192.168.10.14

v1 v2 虛擬vip : 192.168.10.241

 

v3 ------ 192.168.10.16

v4 ------ 192.168.10.17

v3 v4 虛擬vip: 192.168.10.242

 

 

1. 1.       先做redis 熱備節點 2組

 

R1 與 R2 爲第一組節點

V1 (192.168.10.224)  V2 (192.168.10.14) 通過keepalived

 

主(master)：  V1

備份(backup)：V2

虛擬ip  192.168.10.241

 

 

搭建步驟：主備相同

      1. 安裝redis

　　　　

　　　　wget http://redis.googlecode.com/files/redis-2.0.0-rc4.tar.gz

tar zxvf redis-2.0.0-rc4.tar.gz

cd redis-2.0.0-rc4

make

cp redis.conf /etc/ 這個文件是redis啓動的配置文件

cp redis-benchmark redis-cli redis-server /usr/bin/

 

redis-server /etc/redis.conf

redis-cli

注意修改配置：daemonize yes 

 

      2. 安裝高可用軟件 keepalived

　

wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz

tar zxvf keepalived-1.1.20.tar.gz

cd keepalived-1.1.20

./configure --with-kernel-dir=/usr/src/kernels/2.6.18-164.el5-x86_64/

make && make install

 

cp /usr/local/etc/rc.d/init.d/keepalived  /etc/rc.d/init.d/

cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp /usr/local/etc/keepalived/keepalived.conf  /etc/keepalived/

cp /usr/local/sbin/keepalived  /usr/sbin

/etc/init.d/keepalived start

 

 

報錯：

!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!

解決

yum install openssl-devel

 

報錯

configure: error: Popt libraries is required

解決

yum install popt-devel

 

 

 

配置文件修改注意所監控網卡硬件名稱

 

V1配置文件

Keepalived:

 

state MASTER     

 

virtual_ipaddress {

             192.168.10.241  ###VIP

        }

 

 

redis_backup.sh :

 

SLAVEOF 192.168.10.14 6379

 

redis_master.sh:

 

SLAVEOF 192.168.10.14 6379

 

 

V2 配置文件

 

Keepalived:

 

 

state BACKUP     

 

virtual_ipaddress {

             192.168.10.241  ###VIP

        }

 

redis_backup.sh :

 

SLAVEOF 192.168.10.224  6379

 

redis_master.sh:

 

SLAVEOF 192.168.10.224 6379

 

 

測試此互備節點：

 

1.啓動Master上的Redis

redis-server /etc/redis.conf

2.啓動Slave上的Redis

redis-server /etc/redis.conf

3.啓動Master上的Keepalived

/etc/init.d/keepalived start

4.啓動Slave上的Keepalived

  /etc/init.d/keepalived start

5.嘗試通過VIP連接Redis:

redis-cli -h 192.168.10.241 INFO

連接成功，Slave也連接上來了。

role:master

6.嘗試插入一些數據：

redis-cli -h 192.168.10.241  SET Hellor1 Redis

OK

從VIP讀取數據

redis-cli -h 192.168.10.241  GET Hellor1

"Redis"

從Master讀取數據

redis-cli -h 192.168.10.224 GET Hellor1

"Redis"

從Slave讀取數據

  redis-cli -h 192.168.10.14 GET Hellor1

"Redis"

 

 

下面，模擬故障產生：

 

將Master上的Redis進程殺死：

killall -9 redis-server

查看Master上的Keepalived日誌

tailf /var/log/keepalived-redis-state.log

[fault]

Mon Dec 30 10:06:13 CST 2013

同時Slave上的日誌顯示：

tailf /var/log/keepalived-redis-state.log

[master]

Mon Dec 30 10:06:17 CST 2013

Being master....

Run SLAVEOF cmd ...

OK

Run SLAVEOF NO ONE cmd ...

OK

然後我們可以發現，Slave已經接管服務，並且擔任Master的角色了。

redis-cli -h 192.168.10.241 INFO

redis-cli -h 192.168.10.14 INFO

role:master

然後我們恢復Master的Redis進程

redis-server /etc/redis.conf

查看Master上的Keepalived日誌

tailf /var/log/keepalived-redis-state.log

[master]

Mon Dec 30 10:10:13 CST 2013

Being master....

Run SLAVEOF cmd ...

OK

Run SLAVEOF NO ONE cmd ...

OK

同時Slave上的日誌顯示：

tailf /var/log/keepalived-redis-state.log

[backup]

Mon Dec 30 10:10:12 CST 2013

Being slave....

Run SLAVEOF cmd ...

OK

可以發現目前的Master已經再次恢復了Master的角色，故障切換以及自動恢復都成功了。

 

 

R3 與 R4 爲第二組節點

參考第一組即可

 

V3(192.168.10.16)  V4 (192.168.10.17) 通過keepalived

 

主(master)：  V3

備份(backup)：V4

虛擬ip  192.168.10.242

virtual_router_id 不同

 

 

 

 

1. 2.       做twemproxy 代理互備

 

 

Tp (192.168.10.15)

Tp1(192.168.10.18)

 

搭建步驟：

 

1.安裝 twemproxy

 

 

yum install automake

yum install libtool

git clone git://github.com/twitter/twemproxy.git

cd twemproxy

 

CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy  --enable-debug=log

 

編譯的時候報錯

 

autoreconf: Entering directory `.' 

autoreconf: configure.ac: not using Gettext 

autoreconf: running: aclocal --force -I m4 

configure.ac:8: error: Autoconf version 2.64 or higher is required 

configure.ac:8: the top level 

autom4te: /usr/bin/m4 failed with exit status: 63 

aclocal: autom4te failed with exit status: 63 

autoreconf: aclocal failed with exit status: 63 

 

原因是autoconf版本過低

 

升級

查看當前版本

#rpm -qf /usr/bin/autoconf  

autoconf-2.63-5.1.el6.noarch

 

卸載當前版本

rpm -e --nodeps autoconf-2.63   

 

下載新版本

wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.64.tar.gz

 

 

解壓安裝

tar zxvf autoconf-2.64.tar.gz 

cd autoconf-2.64 

./configure --prefix=/usr 

make && make install 

 

查看是否安裝成功

/usr/bin/autoconf -V

 

 

CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy  --enable-debug=log

make && make install 

 

cp conf/nutcracker.yml /usr/local/twemproxy/conf/ 

 

測試

/usr/local/twemproxy/sbin/nutcracker -t

nutcracker: configuration file 'conf/nutcracker.yml' syntax is ok

 

--------------------------------------------------------------------------

啓動命令

調試啓動

 

/usr/local/twemproxy/sbin/nutcracker -c

 

以守護進程啓動

/usr/local/twemproxy/sbin/nutcracker -d -c

 

 

2.安裝 keepalived

參考上面安裝步驟即可

 

 

修改配置文件

 

Tp 配置文件

 

Keepalived：

 

vrrp_script chk_t {

                script "/etc/keepalived/scripts/t_check.sh"   ###監控腳本

                interval 2                                        ###監控時間

}

vrrp_instance VI_1 {

        state MASTER                            ###設置爲MASTER

        interface eth1                          ###監控網卡

        virtual_router_id 53

        priority 101                            ###權重值

        authentication {

                     auth_type PASS             ###加密

                     auth_pass redis            ###密碼

        }

        track_interface {

            eth1

        }

        virtual_ipaddress {

             192.168.10.243                         ###VIP

        }

        notify_master /etc/keepalived/scripts/t_master.sh

        notify_backup /etc/keepalived/scripts/t_backup.sh

        notify_fault  /etc/keepalived/scripts/t_fault.sh

        notify_stop   /etc/keepalived/scripts/t_stop.sh

}

 

/usr/local/twemproxy/conf/nutcracker.yml

redis:

  listen: 192.168.10.243:6379

  hash: fnv1a_64

  distribution: ketama

  auto_eject_hosts: true

  redis: true

  server_retry_timeout: 2000

  server_failure_limit: 1

  servers:

   - 192.168.10.241:6379:1 node1

   - 192.168.10.242:6379:1 node2

 

 

 

Tp1 配置文件

 

Keepalived：

 

vrrp_script chk_t {

                script "/etc/keepalived/scripts/t_check.sh"   ###監控腳本

                interval 2                                        ###監控時間

}

vrrp_instance VI_1 {

        state BACKUP                            ###設置爲BACKUP

        interface eth1                          ###監控網卡

        virtual_router_id 53

        priority 99                            ###權重值

        authentication {

                     auth_type PASS             ###加密

                     auth_pass redis            ###密碼

        }

         track_interface {

            eth1

        }

        virtual_ipaddress {

             192.168.10.243                         ###VIP

        }

        notify_master /etc/keepalived/scripts/t_master.sh

        notify_backup /etc/keepalived/scripts/t_backup.sh

        notify_fault  /etc/keepalived/scripts/t_fault.sh

        notify_stop   /etc/keepalived/scripts/t_stop.sh

}

 

/usr/local/twemproxy/conf/nutcracker.yml

redis:

  listen: 192.168.10.243:6379

  hash: fnv1a_64

  distribution: ketama

  auto_eject_hosts: true

  redis: true

  server_retry_timeout: 2000

  server_failure_limit: 1

  servers:

   - 192.168.10.241:6379:1 node1

   - 192.168.10.242:6379:1 node2

 

測試

redis-cli -h 192.168.10.243

redis> set pp pp

OK

redis> get pp

"pp"

 

關掉tp 查看tp1 日誌

tailf /var/log/keepalived-t-state.log

[backup]

Tue Dec 31 14:37:20 CST 2013

Being slave....

[master]

Tue Dec 31 14:47:28 CST 2013

Being master....

Run twemproxy ...

 

Tp2 接管 vip 繼續提供服務

redis-cli -h 192.168.10.243

redis> get pp

"pp"

 

重啓tp 查看tp1 日誌

tailf /var/log/keepalived-t-state.log

[backup]

Tue Dec 31 14:54:37 CST 2013

Being slave....

 

redis-cli -h 192.168.10.243

redis> get pp

"pp"

 Tp接管主服務 tp2切換備份服務

3.可能遇到相關問題和處理辦法：
1. 如果兩機器不能連通，請注意關掉防火牆：　
service iptables stop #臨時關閉
chkconfig iptables off #永久關閉
2. windows上編寫的shell腳本，要注意通過dos2unix 命令轉換，否則shell腳本可能有不可見符號不能正常執行。