Codis介紹
Codis是一個分佈式Redis解決方案,對於上層的應用來說,連接到 Codis Proxy 和連接原生的 Redis Server 沒有明顯的區別(有一些命令不支持),上層應用可以像使用單機的Redis一樣使用,Codis底層會處理請求轉發,不停機的數據遷移等工作,對於客戶端是透明的。
Codis架構
Codis引入Group的概念,每個Group包括1個Redis Master及至少一個Redis Slave。如當前Master有問題,運維人員可以通過Dashboard切換到Slave。
支持熱數據遷移(Auto Rebalance),修改了Redis Server 源碼,稱之爲Codis Server。
Codis 採用預先分片(Pre-Sharding)機制,事先規定好了,分成1024個slots(最多支持後端1024個Codis Server),這些路由保存在zookeeper中。
Zookeeper還維護Codis Server Group信息,並提供分佈式鎖等服務。
Codis搭建
主機資源
192.168.10.2
192.168.10.3
192.168.10.4
Zookeeper
192.168.10.2
192.168.10.3
192.168.10.4
Zookeeper安裝
/etc/hosts
###zookeeper for codis
192.168.10.2 zookeeper-node1
192.168.10.3 zookeeper-node2
192.168.10.4 zookeeper-node3
Zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper/data/
clientPort=2181
maxClientCnxns=50
server.1=zookeeper-node1:2888:3888 #2888爲程序監控端口,3888選舉通信端口
server.2=zookeeper-node2:2888:3888
server.3=zookeeper-node3:2888:3888
Mkdir zookeeper/data/
Echo “1” > zookeeper/data/myid #生成ID,myid對用zoo.cfg中的server.ID
#啓動
#/usr/local/zookeeper/bin/zkServer.sh start
Go語言安裝
Codis使用最新版本,支持新功能,查看
https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md
需要go版本1.7.3
下載地址:https://studygolang.com/dl/golang/go1.7.3.linux-amd64.tar.gz
#tar -zxf go1.7.3.linux-amd64.tar.gz -C /usr/local/
#vi /etc/profile
#####go path for codis
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
#source /etc/profile
# go version
go version go1.7.3 linux/amd64
Codis安裝
Codis使用最新3.2版本
下載目錄爲 $GOPATH/src/github.com/CodisLabs/codis
# mkdir -p /usr/local/go/src/github.com/CodisLabs
#cd $_ && git clone https://github.com/CodisLabs/codis.git -b release3.2
#cd codis
#make ##直接通過make編譯
# ls bin/
assets codis-admin codis-dashboard codis-fe codis-ha codis-proxy codis-server redis-benchmark redis-cli redis-sentinel version
#####bin內生成可執行文件,bin/assets文件夾是codis-dashboard http服務需要的前端資源,需要和codis-dashboard放置在同一文件夾下
###創建codis目錄
#mkdir /usr/local/codis
#mkdir -p /usr/local/codis/{logs,conf,scripts,db,run}
#mkdir -p /usr/local/codis/db/{redis_data_6380,redis_data_6381}
###copy codis bin目錄
#cp -rf /usr/local/go/src/github.com/CodisLabs/codis/bin/ /usr/local/codis/
###codis本身只有codis-server,沒有redis-cli,需要把redis-3.2.11安裝包中的redis-cli copy 到/usr/local/codis/bin/下面(此步驟如果有redis-cli可以省略)
#cd /usr/local/go/src/github.com/CodisLabs/codis/extern/redis-3.2.11/src/
#cp redis-cli /usr/local/codis/bin/
Codis redis配置
#cd /usr/local/go/src/github.com/CodisLabs/codis/extern/redis-3.2.11
#cp redis.conf /usr/local/codis/conf/redis6380.conf
####Redis.conf 關閉了rdb aof
daemonize yes
pidfile /usr/local/codis/run/redis-6380.pid
port 6380
tcp-backlog 511
timeout 86400
tcp-keepalive 60
loglevel notice
logfile "/usr/local/codis/logs/redis-6380.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump-6380.rdb
dir /usr/local/codis/db/redis_data_6380
masterauth "123456"
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
requirepass "123456"
maxmemory 10gb
maxmemory-policy volatile-lru
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 0 0 0
client-output-buffer-limit pubsub 0 0 0
hz 10
aof-rewrite-incremental-fsync yes
#啓動redis
/usr/local/codis/bin/codis-server /usr/local/codis/conf/redis-6380.conf &
/usr/local/codis/bin/codis-server /usr/local/codis/conf/redis-6381.conf &
Codis dashboard
Codis dashboard僅在需要啓動管理截面的codis 的服務器上操作
Codis 3.0的dashboard,作爲集羣管理工具,支持Codis-proxy codis-server的添加、刪除以及數據遷移等操作。在集羣狀態發生改變時,codis-dashboard維護集羣下所有codis-proxy的狀態一致性。
對於同一業務集羣,同一時刻codis-dashboard只能有0或者1個。
所有對集羣的修改都必須通過codis-dashboard完成。
####################################
生成配置文件
/usr/local/codis/bin/codis-dashboard --default-config | tee /usr/local/codis/conf/dashboard.conf
##Dashboard.conf
coordinator_name = "zookeeper"
coordinator_addr = "192.168.10.2:2181,192.168.10.3:2181,192.168.10.4:2181"
product_name = "codis-demo"
product_auth = "123456"
admin_addr = "0.0.0.0:18080"
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = "
coordinator_name 外部存儲類型,接受zookeeper/etcd
coordinator_addr 外部存儲地址
product_name 集羣名稱,滿足正則\w[\w\.\-]*
product_auth 集羣密碼,默認爲空
admin_addr RESTful API端口
###########
啓動 codis-dashboard
#nohup /usr/local/codis/bin/codis-dashboard --ncpu=4 --config=/usr/local/codis/conf/dashboard.conf --log=/usr/local/codis/logs/dashboard.log --log-level=WARN &
--說明:
--ncpu=N 最大使用cpu個數
-c CONF,--conf=CONF 指定啓動配置文件
-l FILE , --log=FILE 設置log輸出文件
--log-level=LEVEL 設置log輸出等級:INFO,WARN,DEBUG,ERROR;默認INFO,推薦WARM
正常關閉dashboard bin/codis-admin --dashboard=*.*.*.*:18080 --shutdown
Codis proxy
生成proxy配置文件
# /usr/local/codis/bin/codis-proxy --default-config | tee /usr/local/codis/conf/proxy.conf
# grep -vE '^$|^#' conf/proxy.conf
product_name = "codis-demo"
product_auth = "123456"
session_auth = ""
admin_addr = "0.0.0.0:11080"
proto_type = "tcp4"
proxy_addr = "0.0.0.0:19000"
jodis_name = "zookeeper"
jodis_addr = "zookeeper-ip"
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false
proxy_datacenter = ""
proxy_max_clients = 1000
proxy_max_offheap_size = "1024mb"
proxy_heap_placeholder = "256mb"
backend_ping_period = "5s"
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"
backend_max_pipeline = 20480
backend_primary_only = false
backend_primary_parallel = 1
backend_replica_parallel = 1
backend_keepalive_period = "75s"
backend_number_databases = 16
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"
session_send_bufsize = "64kb"
session_send_timeout = "30s"
session_max_pipeline = 10000
session_keepalive_period = "75s"
session_break_on_failure = false
metrics_report_server = ""
metrics_report_period = "1s"
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""
--說明:
product_name 集羣名稱,參考dashboard參數說明
product_auth 集羣密碼,默認爲空,此處密碼爲訪問dashboard codis proxy codis server
admin_addr RESTful API 端口
proto_type redis端口類型,接受tcp/tcp4/tcp6/unix/unixpacket
proxy_addr redis端口地址或者路徑
jodis_addr 註冊zookeeper地址
jodis_timeout 註冊session timeouot時間,單位second
backend_ping_period codis-server探活週期,單位second,0表示禁止
Session_max_timeout client 連接最大讀超時,單位second,0表示禁止
Session_max_bufsize client聯繫讀寫緩衝區大小,單位Byte
session_max_pipeline client連接最大popeline大小
session_keepalive_period client的tcp keepalive 週期,僅tcp有效,0表示禁止
啓動
#nohup /usr/local/codis/bin/codis-proxy --ncpu=4 --config=/usr/local/codis/conf/proxy.conf --log=/usr/local/codis/logs/proxy.log --log-level=WARN &
Codis-proxy啓動後,處於waiting狀態,監控proxy_add地址,但是不會accept連續,添加到集羣並完成集羣狀態的同步,才能改變狀態爲online.
添加proxy的方法
1、通過codis-fe添加:通過add proxy按鈕,將admin_addr加入到集羣
2、通過codis-admin命令行添加
#/usr/local/codis/bin/codis-admin --dashboard=*.*.*.*:18080 --create-proxy -x *.*.*.*:11080
18080爲dashboard的admin_addr地址
11080爲proxy的admin_addr地址
添加過程,dashboard會完成如下一系列動作:
@獲取proxy信息,對集羣name以及auth進行驗證,並將其信息寫入到外部存儲中;
@同步slots狀態
@標記proxy狀態爲online,此後proxy開始accept連接並開始提供服務
關閉proxy
#/usr/local/codis/bin/codis-admin --proxy=*.*.*.*:11080 --auth=”” --shutdown
Codis-fe
配置啓動codis FE集羣管理界面(僅需要啓動dashiboard的codis服務器上操作)
生成配置文件
#/usr/local/codis/bin/codis-admin --dashboard-list --zookeeper=192.168.10.3 | tee /usr/local/codis/conf/codis.json
注意:此處生成的dashboard配置爲獲取的主機名,寫成IP方式。
啓動codis-fe
nohup /usr/local/codis/bin/codis-fe --ncpu=2 --log=/usr/local/codis/logs/fe.log --log-level=WARN --dashboard-list=/usr/local/codis/conf/codis.json --listen=192.168.10.2:18090 &
添加proxy
添加group
添加server
每個group包含一主、多從server
Slots
按照group進行slots
------------------參考
http://navyaijm.blog.51cto.com/4647068/1637688
http://blog.csdn.net/shmiluwei/article/details/51958359
https://www.cnblogs.com/xmzncc/p/6218694.html
https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md
http://www.cnblogs.com/reblue520/p/6874925.html
Sentinels
Sentinel.conf
protected-mode no
port 26379
logfile "/usr/local/codis/logs/sentinel.log"
dir “/tmp”
/usr/local/codis/bin/redis-sentinel /usr/local/codis/conf/sentinel.conf &
---參考
http://www.bubuko.com/infodetail-2316020.html