redis cluster初探之部署

一、簡介

在3.0版本之前，redis通過哨兵實現主從的高可用，在3.0版本之後，redis官方推出了高可用的redis集羣解決方案。

重點知識：數據分區
分佈式數據庫是將數據根據分區規則劃分到多個節點上，每個節點負責存儲一部分數據；常用的分區規則有哈希分區與順序分區兩種，兩種方式各有優勢，redis集羣使用的是哈希分區的虛擬槽分區方式
redis集羣採用的虛擬槽分區，所有的鍵通過計算（slot=CRC16（key）&16383）映射到0～16383整數槽位內，每一個節點負責一部分的槽位。

二、redis安裝

前提：節點間公鑰推送 互免密

2.1、配置ansible hosts

ansible所在節點：192.168.143.130

vim /etc/ansible/hosts 
[redis-cluster]
192.168.143.131
192.168.143.132

2.2、在ansible所在節點安裝ruby環境

yum install rubygem-psych ruby-irb ruby-devel rubygem-rdoc rubygem-io-console rubygems rubygem-abrt rebygem-json ruby-libs rubygem-bigdecimal -y

2.3、在ansible所在節點下載redis

Redis：4.0.14 https://redis.io/download
Redis-gem：3.3.5 https://rubygems.org/gems/redis/versions/

2.4、在ansible所在節點安裝redis-gem（redis的ruby客戶端）

[root@node1 local]# gem install redis
Successfully installed redis-3.3.5
Parsing documentation for redis-3.3.5
Installing ri documentation for redis-3.3.5
1 gem installed

2.5、在ansible所在節點安裝redis節點（集羣模式）

解壓

cd /usr/local/
tar -xzvf redis-4.0.14.tar.gz -C .

編譯安裝

cd redis-4.0.14/
make && make install

準備工作

mkdir redis-cluster/
cp redis-4.0.14/src/redis-server redis-cluster/
cp redis-4.0.14/src/redis-cli redis-cluster/

準備公共配置文件

vim redis-cluster/redis_comm.conf 
#集羣相關的配置
cluster-enabled yes
cluster-node-timeout 15000
cluster-require-full-coverage no

在所有節點創建redis集羣工作目錄

ansible redis-cluster -m file -a 'path=/home/redis/redis_cluster/redis_node state=directory'

將redis-server和redis-cli放在集羣各節點的/usr/bin目錄下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis-server dest=/usr/bin/redis-server mode=755'
ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis-cli dest=/usr/bin/redis-cli mode=755'

將公共配置文件放在集羣各節點的/home/redis/redis_cluster/redis_node下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis_comm.conf dest=/home/redis/redis_cluster/redis_node/redis_comm.conf mode=644'`

創建redis節點腳本（redis_node_build.sh ）

#!/bin/bash
#
#Redis can open up to 16
redis_node=16
redis_workdir="/home/redis/redis_cluster/redis_node"
start_port=9001
redis_conf="redis.conf"
#Initialize the redis process
rm -rf ${redis_workdir}/90*
ps -ef|grep redis-server|grep -v grep
if [ $? -eq 0 ];then
    kilall redis-server
    echo "The node has been reset successfully"
else
    echo "Node initialization succeeded"
Fi
#Modify kernel parameters: maximum listening queue length per port
conn_num=`sysctl -a|grep net.core.somaxconn|awk -F "=" '{print $2}'`
if [ $conn_num -ne 16384 ];then
    sed -i 's/net.core.somaxconn//g' /etc/sysctl.conf
    echo "net.core.somaxconn=16384" >> /etc/sysctl.conf
    sysctl -p
    sysctl -w net.core.somaxconn=16384
else
    echo "net.core.somaxconn = ${conn_num}"
fi
#Modify the maximum number of file descriptors that a process can open
echo 'ulimit -n 16384' >> /etc/profile
source /etc/profile
#Create a redis node
for ((i=0;i<$redis_node;i++));
do
    next_port=$(expr $start_port + $i)
    #Create an node directory
    mkdir -p $redis_workdir/$next_port
    cd $redis_workdir/$next_port
    #Prepare each node configuration file
    echo "include $redis_workdir/redis_comm.conf" >> $redis_conf
    echo "port ${next_port}" >> $redis_conf
    echo "pidfile ${redis_workdir}/${next_port}/redis.pid" >> $redis_conf
    echo "dir ${redis_workdir}/${next_port}" >> $redis_conf
    echo "logfile ${redis_workdir}/${next_port}/redis.log" >> $redis_conf
    #Run the redis node
    /usr/bin/redis-server redis.conf
done

將創建redis節點的腳本放在集羣各節點的/home/redis/redis_cluster/redis_node下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis_node_build.sh dest=/home/redis/redis_cluster/redis_node/redis_node_build.sh  mode=755'

在各節點執行創建redis節點腳本

ansible redis-cluster -m shell -a 'cd /home/redis/redis_cluster/redis_node/;./redis_node_build.sh &> /dev/null'

查看進程

三、redis-cluster安裝

3.1、準備

 cp /usr/local/redis-4.0.14/src/redis-trib.rb /usr/local/redis-cluster/

3.2、生成集羣ip列表（全主集羣）

cd /usr/local/redis-cluster/
vim redis_cluster_build.py
#!/usr/bin/python
#
ret = "/usr/local/redis-cluster/redis-trib.rb create --replicas 0 "
ip=open('./iplist-redis')
for i in ip:
    for j in range(16):
        ret += i.replace('\n','')+":"+str(9001+j)+" "
print ret

3.3、創建集羣

/usr/local/redis-cluster/redis-trib.rb create --replicas 0 192.168.143.131:9001 192.168.143.131:9002 192.168.143.131:9003 192.168.143.131:9004 192.168.143.131:9005 192.168.143.131:9006 192.168.143.131:9007 192.168.143.131:9008 192.168.143.131:9009 192.168.143.131:9010 192.168.143.131:9011 192.168.143.131:9012 192.168.143.131:9013 192.168.143.131:9014 192.168.143.131:9015 192.168.143.131:9016 192.168.143.132:9001 192.168.143.132:9002 192.168.143.132:9003 192.168.143.132:9004 192.168.143.132:9005 192.168.143.132:9006 192.168.143.132:9007 192.168.143.132:9008 192.168.143.132:9009 192.168.143.132:9010 192.168.143.132:9011 192.168.143.132:9012 192.168.143.132:9013 192.168.143.132:9014 192.168.143.132:9015 192.168.143.132:9016 

Ps：create參數是redis-trib.rb命令創建集羣的參數 --replicas 1參數意思是主從各佔節點總數的一半，節點列表的第一個節點是master，第二個節點是第一個節點的slave，依次往後；--replicas 0參數爲創建全master集羣，意思是節點列表的所有節點都是master

3.4、查看集羣狀態

[root@node2 ~]# redis-cli -h 192.168.143.131 -p 9001
192.168.143.131:9001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:32
cluster_size:32
cluster_current_epoch:32
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1198
cluster_stats_messages_pong_sent:1174
cluster_stats_messages_sent:2372
cluster_stats_messages_ping_received:1143
cluster_stats_messages_pong_received:1198
cluster_stats_messages_meet_received:31
cluster_stats_messages_received:2372

參考文章：
Redis-cluster docs： https://redis.io/topics/cluster-tutorial
replicas參數： https://redis.io/commands/cluster-replicas
Redis-cluster command： https://redis.io/commands#cluster
Redis-cluster-create scripts： https://github.com/antirez/redis/blob/unstable/utils/create-cluster/create-cluster