Memcache分佈式實現原理---Java_Memcache 源碼分析



    Memcache本身只是一個內存緩存服務器，用來緩存數據以緩解數據庫壓力，但是我們經常會聽到分佈工Memcache，那麼它是如何實現的呢？在使用Java操作Memcache時，我們通常會藉助Java_Memcache來幫助我們完成各項操作， get/set/delete等。下面我們閱讀一下Java_Memcache的源碼來窺探一二。（注：網上很難找到Java_Memcache最新的源代碼，但是可以利用Java 反編譯工具來查看其源碼，具體操作步驟可以參見我的另一篇博客--Eclipse設置Java反編譯）

Java_Memcache操作步驟

    首先，在應用程序初始化或第一次始用MemcachedClient時需要SockIOPool完成對Memcache客戶端連接Memcache Server的配置參數初始化，類似於數據庫連接池管理，主要包括IP&端口，連接數控制，連接空閒時間等。

SockIOPool sockpool= SockIOPool.getInstance();
//設置緩存服務器地址，可以設置多個實現分佈式緩存
sockpool.setServers(new String[]{"127.0.0.1:11211","127.0.0.1:11212"});
//設置初始連接,最小最大連接
sockpool.setInitConn(5);
sockpool.setMinConn(5);
sockpool.setMaxConn(250);
//設置每個連接最大空閒時間1個小時
sockpool.setMaxIdle(1000 * 3600);
sockpool.setMaintSleep(30);
sockpool.setNagle(false);
sockpool.setSocketTO(3000);
sockpool.setSocketConnectTO(0);
sockpool.initialize();
然後就可以獲取MemCachedClient實例，調用對應的get/set方法。
MemCachedClient memCache = new MemCachedClient();
Object obj = memCache.get(key);
//memCache.set(key, value);

SockIOPool初始化

public class SockIOPool
{
    //注意，受保護的構造函數
    protected SockIOPool()
    {
        poolMultiplier = 3;
        minConn = 5;
        maxConn = 100;
        maxIdle = 300000L;
        maxBusyTime = 30000L;
        socketTO = 3000;
        socketConnectTO = 3000;
        failover = true;
        failback = true;
        nagle = false;
    }

    public static SockIOPool getInstance()
    {
        SockIOPool sockiopool = new SockIOPool();
        sockiopool.schoonerSockIOPool = SchoonerSockIOPool.getInstance("default");
        return sockiopool;
    }
    public void initialize()
    {
        schoonerSockIOPool.initialize();
    }
    public void setServers(String as[])
    {
        schoonerSockIOPool.setServers(as);
    }
    public void setInitConn(int i)
    {
        schoonerSockIOPool.setInitConn(i);
    }
    ......
    private int poolMultiplier;
    private int minConn;
    private int maxConn;
    private long maxIdle;
    private long maxBusyTime;
    private int socketTO;
    private int socketConnectTO;
    private boolean failover;
    private boolean failback;
    private boolean nagle;
    //連接信息都委託給schoonnerSockIOPool管理
    private SchoonerSockIOPool schoonerSockIOPool;
    ......
}

SchoonerSockIOPool 初始化

public class SchoonerSockIOPool
{
    protected SchoonerSockIOPool(boolean flag)
    {
        initialized = false;
        minConn = 8;
        maxConn = 32;
        maxBusyTime = 30000L;
        maintSleep = 30000L;
        socketTO = 30000;
        socketConnectTO = 3000;
        maxIdle = 1000L;
        aliveCheck = false;
        failover = true;
        failback = true;
        nagle = false;
        hashingAlg = 0;
        totalWeight = Integer.valueOf(0);
        bufferSize = 1049600;
        isTcp = flag;//true,這裏是採用TCP和memcache進行通信
    }
    public static SchoonerSockIOPool getInstance(String s)
    {
        synchronized(pools)
        {
            if(!pools.containsKey(s))
            {
                SchoonerSockIOPool schoonersockiopool = new SchoonerSockIOPool(true);
                pools.putIfAbsent(s, schoonersockiopool);
            }
        }
        return (SchoonerSockIOPool)pools.get(s);
    }
    public final void setServers(String as[])
    {
        servers = as;
    }
    public final void setInitConn(int i)
    {
        if(i < minConn)
            minConn = i;
    }
    ......
    public void initialize()
    {
        initDeadLock.lock();
        if(servers == null || servers.length <= 0)
        {
            if(log.isErrorEnabled())
                log.error("++++ trying to initialize with no servers");
            throw new IllegalStateException("++++ trying to initialize with no servers");
        }
        socketPool = new HashMap(servers.length);
        hostDead = new ConcurrentHashMap();
        hostDeadDur = new ConcurrentHashMap();
    <strong>//注意，這裏提供了一致性Hash和一般Hash算法，我們先分析一般Hash</strong>
        if(hashingAlg == 3)
            populateConsistentBuckets();
        else
            populateBuckets();
        initialized = true;
        initDeadLock.unlock();
        ......
    }
    ......
    private static ConcurrentMap pools = new ConcurrentHashMap();
    boolean initialized;
    private int minConn;
    private int maxConn;
    private long maxBusyTime;
    private long maintSleep;
    private int socketTO;
    private int socketConnectTO;
    private static int recBufferSize = 128;
    private long maxIdle;
    private boolean aliveCheck;
    private boolean failover;
    private boolean failback;
    private boolean nagle;
    private int hashingAlg;
    private final ReentrantLock initDeadLock = new ReentrantLock();
}

Memcache  一般Hash算法

我們先分析一下一般Hash算法，關於一致性Hash算法和虛擬節點(利用權重)的問題，會另起一篇單獨探討，在這裏不考慮Server權重的問題

SchoonerSockIOPool 類

private void populateBuckets()
{
    buckets = new ArrayList();
    for(int i = 0; i < servers.length; i++)
    {
	if(weights != null && weights.length > i){
            for(int j = 0; j < weights[i].intValue(); j++)
		buckets.add(servers[i]);
	} else{
	    buckets.add(servers[i]);
	}
	Object obj;
	if(authInfo != null)
	    obj = new AuthSchoonerSockIOFactory(servers[i], isTcp, bufferSize, socketTO, socketConnectTO, nagle, authInfo);
	else
	    obj = new SchoonerSockIOFactory(servers[i], isTcp, bufferSize, socketTO, socketConnectTO, nagle);
	GenericObjectPool genericobjectpool = new GenericObjectPool(((org.apache.commons.pool.PoolableObjectFactory) (obj)), maxConn, (byte)1, maxIdle, maxConn);
	((SchoonerSockIOFactory) (obj)).setSockets(genericobjectpool);
	socketPool.put(servers[i], genericobjectpool);
    }
}

1.將server IP/port放入List

2.爲每個server創建Socket工廠

3.爲每個server創建對象池，以便管理socket連接，類似於數據庫連接池、線程池

4.將server和對應的對象池放入map中

SchoonerSockIOFactory

public class SchoonerSockIOFactory extends BasePoolableObjectFactory
{
    public SchoonerSockIOFactory(String s, boolean flag, int i, int j, int k, boolean flag1) {
        host = s;
        isTcp = flag;
        bufferSize = i;
        socketTO = j;
        socketConnectTO = k;
        nagle = flag1;
    }

    public Object makeObject()throws Exception {
        SchoonerSockIO schoonersockio = createSocket(host);
        return schoonersockio;
    }
    ......     

    protected final SchoonerSockIO createSocket(String s) throws Exception {
        Object obj = null;
        if(isTcp)
            obj = new SchoonerSockIOPool.TCPSockIO(sockets, s, bufferSize, socketTO, socketConnectTO, nagle);
        else
            obj = new SchoonerSockIOPool.UDPSockIO(sockets, s, bufferSize, socketTO);
        return ((SchoonerSockIO) (obj));
    }

    <strong>//工廠持有對象池的引用</strong>
    public void setSockets(GenericObjectPool genericobjectpool){
        sockets = genericobjectpool;
    }

    protected GenericObjectPool sockets;
    protected String host;
    protected int bufferSize;
    protected int socketTO;
    protected int socketConnectTO;
    protected boolean isTcp;
    protected boolean nagle;
}

genericobjectpool

package org.apache.commons.pool.impl;
public class GenericObjectPool extends BaseObjectPool implements ObjectPool

{
    ......
    public GenericObjectPool(PoolableObjectFactory factory, int maxActive, 	byte whenExhaustedAction, long maxWait, int maxIdle){
        this(factory, maxActive, whenExhaustedAction, maxWait, maxIdle, 0, false, false, -1L, 3, 1800000L, false);
    }
    ......
    public GenericObjectPool(PoolableObjectFactory factory, int maxActive, byte whenExhaustedAction, long maxWait, int maxIdle, int minIdle, boolean testOnBorrow, boolean testOnReturn, long timeBetweenEvictionRunsMillis, int numTestsPerEvictionRun, long minEvictableIdleTimeMillis, boolean testWhileIdle, long softMinEvictableIdleTimeMillis, boolean lifo){
        ......
        _allocationQueue = new LinkedList();
        _factory = factory;
        _maxActive = maxActive;
        _lifo = lifo;
        switch(whenExhaustedAction)
        {
        case 0: // '\0'
        case 1: // '\001'
        case 2: // '\002'
            _whenExhaustedAction = whenExhaustedAction;
            break;

        default:
            throw new IllegalArgumentException("whenExhaustedAction " + whenExhaustedAction + " not recognized.");
        }
        ......
        _pool = new CursorableLinkedList();
        startEvictor(_timeBetweenEvictionRunsMillis);
    }
    protected synchronized void startEvictor(long delay)
    {
        if(null != _evictor){
            EvictionTimer.cancel(_evictor);
            _evictor = null;
        }
        if(delay > 0L){
            _evictor = new Evictor();
            EvictionTimer.schedule(_evictor, delay, delay);
        }
    }

對象池自身有單獨的線程來維護對象池。關於池的知識，後續單獨分析。

MemCachedClient操作數據

MemCachedClient實例化

public class MemCachedClient
{
    public MemCachedClient()
    {
        this(((String) (null)), true, false);
    }
    public MemCachedClient(String s, boolean flag, boolean flag1)
    {
        BLAND_DATA_SIZE = "       ".getBytes();
        if(flag1)
            client = new BinaryClient(s);
        else
            client = ((MemCachedClient) (flag ? ((MemCachedClient) (new AscIIClient(s))) : ((MemCachedClient) (new AscIIUDPClient(s)))));
    }
    ......
    MemCachedClient client;
}

根據前面MemCachedClient操作步驟，我們知道這裏創建的是AscIIClinet實例

public class AscIIClient extends MemCachedClient
{
    public AscIIClient(String s)
    {
        super((MemCachedClient)null);
        transCoder = new ObjectTransCoder();
        poolName = s;
        init();
    }
    private void init()
    {
        sanitizeKeys = true;
        primitiveAsString = false;
        compressEnable = false;
        compressThreshold = 30720L;
        defaultEncoding = "UTF-8";
        poolName = poolName != null ? poolName : "default";
        pool = SchoonerSockIOPool.getInstance(poolName);
    }
    ......
}

    MemCachedClient操作數據的方法很多，有get/set/add/replace/gets/delete/addOrIncr等，還有一系列的重載方法，我們在此不一一分析，僅分析一下get/set方法，就足以深入學習和探討MemCachedClient的實現機制，其他方法基本類似。

public boolean set(String s, Object obj)
{
    return client.set(s, obj);
}
......
public Object get(String s)
{
    return client.get(s);
}

MemCachedClient.set

public boolean set(String s, Object obj)
{
    return client.set(s, obj);
}

實際操作的是AScIIClient的set方法

public class AscIIClient extends MemCachedClient
{
    public boolean set(String s, Object obj)
    {
        return set("set", s, obj, null, null, Long.valueOf(0L), primitiveAsString);
    }
    private boolean set(String s, String s1, Object obj, Date date, Integer integer, Long long1, boolean flag)
    {
        SchoonerSockIO schoonersockio;
        int i;
        String s2;
        ......
        try
        {
            s1 = sanitizeKey(s1);//UTF-8編碼
        }
        catch(UnsupportedEncodingException unsupportedencodingexception)
        {
            ......
        }
        ......
        schoonersockio = pool.getSock(s1, integer);//利用Key進行Hash，獲取對應的Server
        ......
        if(date == null)
            date = new Date(0L);
        i = NativeHandler.getMarkerFlag(obj);
        s2 = s + " " + s1 + " " + i + " " + date.getTime() / 1000L + " ";
        boolean flag1;
        schoonersockio.writeBuf.clear();
        schoonersockio.writeBuf.put(s2.getBytes());
        int j = schoonersockio.writeBuf.position();
        schoonersockio.writeBuf.put(BLAND_DATA_SIZE);
        if(long1.longValue() != 0L)
            schoonersockio.writeBuf.put((new StringBuilder()).append(" ").append(long1.toString()).toString().getBytes());
        schoonersockio.writeBuf.put(B_RETURN);
        SockOutputStream sockoutputstream = new SockOutputStream(schoonersockio);
        int k = 0;
        if(i != 0)
        {
            byte abyte0[];
            if(flag)
                abyte0 = obj.toString().getBytes(defaultEncoding);
            else
                abyte0 = NativeHandler.encode(obj);
            sockoutputstream.write(abyte0);
            k = abyte0.length;
        } else
        {
            k = transCoder.encode(sockoutputstream, obj);
        }
        schoonersockio.writeBuf.put(B_RETURN);
        byte abyte1[] = (new Integer(k)).toString().getBytes();
        int l = schoonersockio.writeBuf.position();
        schoonersockio.writeBuf.position(j);
        schoonersockio.writeBuf.put(abyte1);
        schoonersockio.writeBuf.position(l);
        schoonersockio.flush();
        String s3 = (new SockInputStream(schoonersockio, 2147483647)).getLine();
        if(!"STORED\r\n".equals(s3))
            break MISSING_BLOCK_LABEL_538;
        flag1 = true;
        if(schoonersockio != null)
        {
            schoonersockio.close();
            schoonersockio = null;
        }
        ......
    }

Pool.getSocket

public final SchoonerSockIO getSock(String s, Integer integer)
{
    ......
    if(i == 1)
    {
        //如果只有一個server，直接建立與該Server的connection
        SchoonerSockIO schoonersockio = hashingAlg != 3 ? getConnection((String)buckets.get(0)) : getConnection((String)consistentBuckets.get(consistentBuckets.firstKey()));
        return schoonersockio;
    }
    HashSet hashset = new HashSet(Arrays.asList(servers));
    long l = getBucket(s, integer);//利用Hahs算法找到對應的Server
    String s1 = hashingAlg != 3 ? (String)buckets.get((int)l) : (String)consistentBuckets.get(Long.valueOf(l));
    do{
        if(hashset.isEmpty())
            break;
        SchoonerSockIO schoonersockio1 = getConnection(s1);//建立該Server的Connection
        if(schoonersockio1 != null)
            return schoonersockio1;
        ......
    }
}

public final SchoonerSockIO getConnection(String s)
{
    ......
    GenericObjectPool genericobjectpool = (GenericObjectPool)socketPool.get(s);//獲取ObjectPool
    SchoonerSockIO schoonersockio;
    try{
        schoonersockio = (SchoonerSockIO)genericobjectpool.borrowObject();//通過池來獲取連接
    }catch(Exception exception){
        schoonersockio = null;
    }
    ......
    return schoonersockio;
}

sockoutputstream.write

public final class SockOutputStream extends OutputStream
{
    public final void write(byte abyte0[])throws IOException {
        write(abyte0, 0, abyte0.length);
    }

    public final void write(byte abyte0[], int i, int j) throws IOException
    {
        <strong>//將字符數組寫入緩衝區</strong>
        if(j == 0)
            return;
        if(sock.writeBuf.remaining() >= j)
        {
            sock.writeBuf.put(abyte0, i, j);
        } else
        {
            int k = 0;
            boolean flag = false;
            for(int i1 = 0; (i1 = j - k) > 0;)
            {
                int l = sock.writeBuf.remaining();
                l = l >= i1 ? i1 : l;
                if(l == 0)
                    writeToChannel();
                else
                    sock.writeBuf.put(abyte0, i, l);
                k += l;
            }

        }
        count += j;
    }

MemCachedClient.get

實際操作的是AscIIClient的get方法

public class AscIIClient extends MemCachedClient
{
    public Object get(String s)
    {
        return get(s, null);
    }
    ......
    private Object get(String s, String s1, Integer integer, boolean flag)
    {
        SchoonerSockIO schoonersockio;
        String s2;
         
        try{
            s1 = sanitizeKey(s1<strong>);//UTF-8編碼</strong>
        }
        ......
        schoonersockio = pool.getSock(s1, integer<strong>);//與set方法類似，利用Hash算法建立與server的socket連接</strong>
        ......
        schoonersockio.writeBuf.clear();
        schoonersockio.writeBuf.put(s2.getBytes());
        schoonersockio.writeBuf.put(B_RETURN);
        schoonersockio.flush();
        i = 0;
        j = 0;
        sockinputstream = new SockInputStream(schoonersockio, 2147483647);
        ......
            byte abyte0[] = sockinputstream.getBuffer();
            if((j & 2) == 2){
                GZIPInputStream gzipinputstream = new GZIPInputStream(new ByteArrayInputStream(abyte0));
                ByteArrayOutputStream bytearrayoutputstream = new ByteArrayOutputStream(abyte0.length);
                byte abyte1[] = new byte[2048];
                int i1;
                while((i1 = gzipinputstream.read(abyte1)) != -1) 
                    bytearrayoutputstream.write(abyte1, 0, i1);
                abyte0 = bytearrayoutputstream.toByteArray();
                gzipinputstream.close();
            }
            if(primitiveAsString || flag)
                obj1 = new String(abyte0, defaultEncoding);
            else
                obj1 = NativeHandler.decode(abyte0, j);
         .... 
        obj3 = obj1;
        return obj3;
    }

總結：本文概要地描述了Memcache的初始化以及對數據的get/set操作流程，中間涉及到利用Key值進行Hash算法獲取對應的Server，根據Server對應的對象池獲取Socket連接以及SocketInputStream SocketOutputSteam操作數據。

在接下來的系列文章裏，會深入研究各個部分的細節。（2015-05-12）

ps:如果您覺得本文對您的學習有所幫助，請多多支持。