Memcached HA架構探索

magent是一款開源的Memcached代理服務器軟件，可以用它做一些高可用架構嘗試。目前magent已更新到0.6，我在centos 6.0 64bit機器上面未編譯通過，所以我在這裏用0.5的源碼來測試;

google項目地址：http://code.google.com/p/memagent/

 

一、安裝步驟：

1、編譯安裝libevent：

wget https://github.com/downloads/libevent/libevent/libevent-2.0.16-stable.tar.gz
tar zxvf libevent-2.0.16-stable.tar.gz
cd libevent-2.0.16-stable/
./configure --prefix=/usr
make && make install
cd ../

2、編譯安裝Memcached：

wget http://memcached.googlecode.com/files/memcached-1.4.14.tar.gz
tar zxvf memcached-1.4.14.tar.gz
cd memcached-1.4.14/
./configure --prefix=/usr --with-libevent=/usr
make && make install
cd ../

3、編譯安裝magent：

mkdir magent
cd magent/
wget http://memagent.googlecode.com/files/magent-0.5.tar.gz
tar zxvf magent-0.5.tar.gz
/sbin/ldconfig
vim Makefile
CFLAGS = -Wall -O2 -g
改爲：
CFLAGS = -lrt -Wall -O2 -g
sed -i "s#LIBS = -levent#LIBS = -levent -lm#g" Makefile
vim ketama.h
加入
#ifndef SSIZE_MAX
#define SSIZE_MAX     32767
#endif
make
cp magent /usr/bin/magent
cd ../

安裝中常遇到的問題

gcc -lrt -Wall -g -O2 -I/usr/local/include -m64 -c -o magent.o magent.c

magent.c: In function ‘writev_list’:

magent.c:729: error: ‘SSIZE_MAX’ undeclared (first use in this function)

magent.c:729: error: (Each undeclared identifier is reported only once

magent.c:729: error: for each function it appears in.)

make: *** [magent.o] Error 1

解決辦法：

vim ketama.h

加入

#ifndef SSIZE_MAX
#define SSIZE_MAX     32767
#endif

二、高可用網絡架構

此架構生產環境需要負載均衡器做負載，具體使用什麼軟件或硬件就要看你的具體情況來定了。                             

生產環境每臺服務器分別啓動一個memcached進程和magen進程做冗餘

memcached -d -m 256 -l 192.168.10.11 -p 11211 -u www -c 10240 -P /tmp/memcached.pid  -vv >/home/logs/memcached.txt 2>&1
memcached -d -m 128 -l 192.168.10.12 -p 11211 -u www -c 10240 -P /tmp/memcached.pid  -vv >/home/logs/memcached.txt 2>&1
memcached -d -m 128 -l 192.168.10.13 -p 11211 -u www -c 10240 -P /tmp/memcached.pid  -vv >/home/logs/memcached.txt 2>&1

magent -u www -n 51200 -l 192.168.10.11 -p 12000 -s 192.168.10.12:11211 -s 192.168.10.13:11211 -b 192.168.10.11:11211
magent -u www -n 51200 -l 192.168.10.12 -p 12000 -s 192.168.10.12:11211 -s 192.168.10.13:11211 -b 192.168.10.11:11211
magent -u www -n 51200 -l 192.168.10.13 -p 12000 -s 192.168.10.12:11211 -s 192.168.10.13:11211 -b 192.168.10.11:11211

參數說明：

-s 爲要寫入的memcached， 

-b 爲備份用的memcached。

說明：測試環境用magent和memached的不同端口來實現，在生產環境中可以將magent和memached作爲一組放到多臺服務器上。

看到了吧這樣的架構最好做負載了，用一個VIP分別映射三臺magent的12000端口即可。

#實驗環境測試過程：

[root@odb ~]# telnet 127.0.0.1 12000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
set key 0 0 8                       <—在10000端口設置key的值
88888888
STORED
quit
Connection closed by foreign hos
[root@odb ~]# telnet 127.0.0.1 11211
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get key                     <—在11211端口獲取key的值成功
VALUE key 0 8
88888888
END
quit
Connection closed by foreign host.
[root@odb ~]# telnet 127.0.0.1 11211
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get key                     <—在11211端口獲取key的值成功
VALUE key 0 8
88888888
END
quit
Connection closed by foreign host.

高可用性測試：
[root@odb ~]# ps aux |grep -v grep |grep memcached
root     23455  0.0  0.0  5012 1796 ?        Ss   09:22   0:00 memcached -m 1 -u root -d -l 127.0.0.1 -p 11212
root     24950  0.0  0.0  4120 1800 ?        Ss   10:58   0:00 memcached -m 1 -u root -d -l 127.0.0.1 -p 11211
[root@odb ~]# ps aux |grep -v grep |grep ‘magent -u’
root     25919  0.0  0.0  2176  484 ?        Ss   12:00   0:00 magent -u root -n 51200 -l 127.0.0.1 -p 10000 -s 127.0.0.1:11211 -b 127.0.0.1:11212
root     25925  0.0  0.0  3004  484 ?        Ss   12:00   0:00 magent -u root -n 51200 -l 127.0.0.1 -p 11000 -s 127.0.0.1:11212 -b 127.0.0.1:11211
[root@odb ~]# telnet 127.0.0.1 10000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
set stone 0 0 6                      <—在10000端口設置stone的值
123456
STORED
quit
Connection closed by foreign host.
[root@odb ~]# telnet 127.0.0.1 11000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
set shidl 0 0 6                      <—在11000端口設置shidl的值
666666
STORED
get stone                      <—在11000端口獲取stone的值成功
VALUE stone 0 6
123456
END
incr stone 2                      <—在11000端口修改stone的值成功
123458
get stone
VALUE stone 0 6                     <—在11000端口驗證stone的值，證明上面的修改成功
123458
END
get shidl                      <—在11000端口獲取shidl的值成功
VALUE shidl 0 6
666666
END
quit                     <—退出11000端口
Connection closed by foreign host.
[root@odb ~]# telnet 127.0.0.1 10000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get stone                      <—在10000端口獲取stone的值，已被修改
VALUE stone 0 6
123458
END
get shidl                      <—在10000端口獲取shidl的值成功
VALUE shidl 0 6
666666
END
delete shidl                      <—在10000端口刪除shidl
DELETED
get shidl                      <—在10000端口刪除shidl生效
END
quit
Connection closed by foreign host.
[root@odb ~]# telnet 127.0.0.1 11000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get shidl                      <—在11000端口驗證刪除shidl生效
END
get stone                      <—在11000端口獲取stone的值成功
VALUE stone 0 6
123458
END
quit
Connection closed by foreign host.

Down機模擬測試：

Down掉11211端口的memcached：

[root@odb ~]# kill -9 24950
[root@odb ~]# telnet 127.0.0.1 10000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get stone                      <—在10000依然可以獲取stone的值
VALUE stone 0 6
123458
END
quit
Connection closed by foreign host.
[root@odb ~]# telnet 127.0.0.1 11000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get stone                      <—在11000依然可以獲取stone的值
VALUE stone 0 6
123458
END
quit
Connection closed by foreign host.

Down掉11000端口的magent：

[root@odb ~]# kill -9 25925
[root@odb ~]# telnet 127.0.0.1 10000
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is ‘^]’.
get stone                      <—在10000依然可以獲取stone的值
VALUE stone 0 6
123458
END
quit
Connection closed by foreign host.