Memcache分布式实现原理---Java_Memcache 源码分析



    Memcache本身只是一个内存缓存服务器，用来缓存数据以缓解数据库压力，但是我们经常会听到分布工Memcache，那么它是如何实现的呢？在使用Java操作Memcache时，我们通常会借助Java_Memcache来帮助我们完成各项操作， get/set/delete等。下面我们阅读一下Java_Memcache的源码来窥探一二。（注：网上很难找到Java_Memcache最新的源代码，但是可以利用Java 反编译工具来查看其源码，具体操作步骤可以参见我的另一篇博客--Eclipse设置Java反编译）

Java_Memcache操作步骤

    首先，在应用程序初始化或第一次始用MemcachedClient时需要SockIOPool完成对Memcache客户端连接Memcache Server的配置参数初始化，类似于数据库连接池管理，主要包括IP&端口，连接数控制，连接空闲时间等。

SockIOPool sockpool= SockIOPool.getInstance();
//设置缓存服务器地址，可以设置多个实现分布式缓存
sockpool.setServers(new String[]{"127.0.0.1:11211","127.0.0.1:11212"});
//设置初始连接,最小最大连接
sockpool.setInitConn(5);
sockpool.setMinConn(5);
sockpool.setMaxConn(250);
//设置每个连接最大空闲时间1个小时
sockpool.setMaxIdle(1000 * 3600);
sockpool.setMaintSleep(30);
sockpool.setNagle(false);
sockpool.setSocketTO(3000);
sockpool.setSocketConnectTO(0);
sockpool.initialize();
然后就可以获取MemCachedClient实例，调用对应的get/set方法。
MemCachedClient memCache = new MemCachedClient();
Object obj = memCache.get(key);
//memCache.set(key, value);

SockIOPool初始化

public class SockIOPool
{
    //注意，受保护的构造函数
    protected SockIOPool()
    {
        poolMultiplier = 3;
        minConn = 5;
        maxConn = 100;
        maxIdle = 300000L;
        maxBusyTime = 30000L;
        socketTO = 3000;
        socketConnectTO = 3000;
        failover = true;
        failback = true;
        nagle = false;
    }

    public static SockIOPool getInstance()
    {
        SockIOPool sockiopool = new SockIOPool();
        sockiopool.schoonerSockIOPool = SchoonerSockIOPool.getInstance("default");
        return sockiopool;
    }
    public void initialize()
    {
        schoonerSockIOPool.initialize();
    }
    public void setServers(String as[])
    {
        schoonerSockIOPool.setServers(as);
    }
    public void setInitConn(int i)
    {
        schoonerSockIOPool.setInitConn(i);
    }
    ......
    private int poolMultiplier;
    private int minConn;
    private int maxConn;
    private long maxIdle;
    private long maxBusyTime;
    private int socketTO;
    private int socketConnectTO;
    private boolean failover;
    private boolean failback;
    private boolean nagle;
    //连接信息都委托给schoonnerSockIOPool管理
    private SchoonerSockIOPool schoonerSockIOPool;
    ......
}

SchoonerSockIOPool 初始化

public class SchoonerSockIOPool
{
    protected SchoonerSockIOPool(boolean flag)
    {
        initialized = false;
        minConn = 8;
        maxConn = 32;
        maxBusyTime = 30000L;
        maintSleep = 30000L;
        socketTO = 30000;
        socketConnectTO = 3000;
        maxIdle = 1000L;
        aliveCheck = false;
        failover = true;
        failback = true;
        nagle = false;
        hashingAlg = 0;
        totalWeight = Integer.valueOf(0);
        bufferSize = 1049600;
        isTcp = flag;//true,这里是采用TCP和memcache进行通信
    }
    public static SchoonerSockIOPool getInstance(String s)
    {
        synchronized(pools)
        {
            if(!pools.containsKey(s))
            {
                SchoonerSockIOPool schoonersockiopool = new SchoonerSockIOPool(true);
                pools.putIfAbsent(s, schoonersockiopool);
            }
        }
        return (SchoonerSockIOPool)pools.get(s);
    }
    public final void setServers(String as[])
    {
        servers = as;
    }
    public final void setInitConn(int i)
    {
        if(i < minConn)
            minConn = i;
    }
    ......
    public void initialize()
    {
        initDeadLock.lock();
        if(servers == null || servers.length <= 0)
        {
            if(log.isErrorEnabled())
                log.error("++++ trying to initialize with no servers");
            throw new IllegalStateException("++++ trying to initialize with no servers");
        }
        socketPool = new HashMap(servers.length);
        hostDead = new ConcurrentHashMap();
        hostDeadDur = new ConcurrentHashMap();
    <strong>//注意，这里提供了一致性Hash和一般Hash算法，我们先分析一般Hash</strong>
        if(hashingAlg == 3)
            populateConsistentBuckets();
        else
            populateBuckets();
        initialized = true;
        initDeadLock.unlock();
        ......
    }
    ......
    private static ConcurrentMap pools = new ConcurrentHashMap();
    boolean initialized;
    private int minConn;
    private int maxConn;
    private long maxBusyTime;
    private long maintSleep;
    private int socketTO;
    private int socketConnectTO;
    private static int recBufferSize = 128;
    private long maxIdle;
    private boolean aliveCheck;
    private boolean failover;
    private boolean failback;
    private boolean nagle;
    private int hashingAlg;
    private final ReentrantLock initDeadLock = new ReentrantLock();
}

Memcache  一般Hash算法

我们先分析一下一般Hash算法，关于一致性Hash算法和虚拟节点(利用权重)的问题，会另起一篇单独探讨，在这里不考虑Server权重的问题

SchoonerSockIOPool 类

private void populateBuckets()
{
    buckets = new ArrayList();
    for(int i = 0; i < servers.length; i++)
    {
	if(weights != null && weights.length > i){
            for(int j = 0; j < weights[i].intValue(); j++)
		buckets.add(servers[i]);
	} else{
	    buckets.add(servers[i]);
	}
	Object obj;
	if(authInfo != null)
	    obj = new AuthSchoonerSockIOFactory(servers[i], isTcp, bufferSize, socketTO, socketConnectTO, nagle, authInfo);
	else
	    obj = new SchoonerSockIOFactory(servers[i], isTcp, bufferSize, socketTO, socketConnectTO, nagle);
	GenericObjectPool genericobjectpool = new GenericObjectPool(((org.apache.commons.pool.PoolableObjectFactory) (obj)), maxConn, (byte)1, maxIdle, maxConn);
	((SchoonerSockIOFactory) (obj)).setSockets(genericobjectpool);
	socketPool.put(servers[i], genericobjectpool);
    }
}

1.将server IP/port放入List

2.为每个server创建Socket工厂

3.为每个server创建对象池，以便管理socket连接，类似于数据库连接池、线程池

4.将server和对应的对象池放入map中

SchoonerSockIOFactory

public class SchoonerSockIOFactory extends BasePoolableObjectFactory
{
    public SchoonerSockIOFactory(String s, boolean flag, int i, int j, int k, boolean flag1) {
        host = s;
        isTcp = flag;
        bufferSize = i;
        socketTO = j;
        socketConnectTO = k;
        nagle = flag1;
    }

    public Object makeObject()throws Exception {
        SchoonerSockIO schoonersockio = createSocket(host);
        return schoonersockio;
    }
    ......     

    protected final SchoonerSockIO createSocket(String s) throws Exception {
        Object obj = null;
        if(isTcp)
            obj = new SchoonerSockIOPool.TCPSockIO(sockets, s, bufferSize, socketTO, socketConnectTO, nagle);
        else
            obj = new SchoonerSockIOPool.UDPSockIO(sockets, s, bufferSize, socketTO);
        return ((SchoonerSockIO) (obj));
    }

    <strong>//工厂持有对象池的引用</strong>
    public void setSockets(GenericObjectPool genericobjectpool){
        sockets = genericobjectpool;
    }

    protected GenericObjectPool sockets;
    protected String host;
    protected int bufferSize;
    protected int socketTO;
    protected int socketConnectTO;
    protected boolean isTcp;
    protected boolean nagle;
}

genericobjectpool

package org.apache.commons.pool.impl;
public class GenericObjectPool extends BaseObjectPool implements ObjectPool

{
    ......
    public GenericObjectPool(PoolableObjectFactory factory, int maxActive, 	byte whenExhaustedAction, long maxWait, int maxIdle){
        this(factory, maxActive, whenExhaustedAction, maxWait, maxIdle, 0, false, false, -1L, 3, 1800000L, false);
    }
    ......
    public GenericObjectPool(PoolableObjectFactory factory, int maxActive, byte whenExhaustedAction, long maxWait, int maxIdle, int minIdle, boolean testOnBorrow, boolean testOnReturn, long timeBetweenEvictionRunsMillis, int numTestsPerEvictionRun, long minEvictableIdleTimeMillis, boolean testWhileIdle, long softMinEvictableIdleTimeMillis, boolean lifo){
        ......
        _allocationQueue = new LinkedList();
        _factory = factory;
        _maxActive = maxActive;
        _lifo = lifo;
        switch(whenExhaustedAction)
        {
        case 0: // '\0'
        case 1: // '\001'
        case 2: // '\002'
            _whenExhaustedAction = whenExhaustedAction;
            break;

        default:
            throw new IllegalArgumentException("whenExhaustedAction " + whenExhaustedAction + " not recognized.");
        }
        ......
        _pool = new CursorableLinkedList();
        startEvictor(_timeBetweenEvictionRunsMillis);
    }
    protected synchronized void startEvictor(long delay)
    {
        if(null != _evictor){
            EvictionTimer.cancel(_evictor);
            _evictor = null;
        }
        if(delay > 0L){
            _evictor = new Evictor();
            EvictionTimer.schedule(_evictor, delay, delay);
        }
    }

对象池自身有单独的线程来维护对象池。关于池的知识，后续单独分析。

MemCachedClient操作数据

MemCachedClient实例化

public class MemCachedClient
{
    public MemCachedClient()
    {
        this(((String) (null)), true, false);
    }
    public MemCachedClient(String s, boolean flag, boolean flag1)
    {
        BLAND_DATA_SIZE = "       ".getBytes();
        if(flag1)
            client = new BinaryClient(s);
        else
            client = ((MemCachedClient) (flag ? ((MemCachedClient) (new AscIIClient(s))) : ((MemCachedClient) (new AscIIUDPClient(s)))));
    }
    ......
    MemCachedClient client;
}

根据前面MemCachedClient操作步骤，我们知道这里创建的是AscIIClinet实例

public class AscIIClient extends MemCachedClient
{
    public AscIIClient(String s)
    {
        super((MemCachedClient)null);
        transCoder = new ObjectTransCoder();
        poolName = s;
        init();
    }
    private void init()
    {
        sanitizeKeys = true;
        primitiveAsString = false;
        compressEnable = false;
        compressThreshold = 30720L;
        defaultEncoding = "UTF-8";
        poolName = poolName != null ? poolName : "default";
        pool = SchoonerSockIOPool.getInstance(poolName);
    }
    ......
}

    MemCachedClient操作数据的方法很多，有get/set/add/replace/gets/delete/addOrIncr等，还有一系列的重载方法，我们在此不一一分析，仅分析一下get/set方法，就足以深入学习和探讨MemCachedClient的实现机制，其他方法基本类似。

public boolean set(String s, Object obj)
{
    return client.set(s, obj);
}
......
public Object get(String s)
{
    return client.get(s);
}

MemCachedClient.set

public boolean set(String s, Object obj)
{
    return client.set(s, obj);
}

实际操作的是AScIIClient的set方法

public class AscIIClient extends MemCachedClient
{
    public boolean set(String s, Object obj)
    {
        return set("set", s, obj, null, null, Long.valueOf(0L), primitiveAsString);
    }
    private boolean set(String s, String s1, Object obj, Date date, Integer integer, Long long1, boolean flag)
    {
        SchoonerSockIO schoonersockio;
        int i;
        String s2;
        ......
        try
        {
            s1 = sanitizeKey(s1);//UTF-8编码
        }
        catch(UnsupportedEncodingException unsupportedencodingexception)
        {
            ......
        }
        ......
        schoonersockio = pool.getSock(s1, integer);//利用Key进行Hash，获取对应的Server
        ......
        if(date == null)
            date = new Date(0L);
        i = NativeHandler.getMarkerFlag(obj);
        s2 = s + " " + s1 + " " + i + " " + date.getTime() / 1000L + " ";
        boolean flag1;
        schoonersockio.writeBuf.clear();
        schoonersockio.writeBuf.put(s2.getBytes());
        int j = schoonersockio.writeBuf.position();
        schoonersockio.writeBuf.put(BLAND_DATA_SIZE);
        if(long1.longValue() != 0L)
            schoonersockio.writeBuf.put((new StringBuilder()).append(" ").append(long1.toString()).toString().getBytes());
        schoonersockio.writeBuf.put(B_RETURN);
        SockOutputStream sockoutputstream = new SockOutputStream(schoonersockio);
        int k = 0;
        if(i != 0)
        {
            byte abyte0[];
            if(flag)
                abyte0 = obj.toString().getBytes(defaultEncoding);
            else
                abyte0 = NativeHandler.encode(obj);
            sockoutputstream.write(abyte0);
            k = abyte0.length;
        } else
        {
            k = transCoder.encode(sockoutputstream, obj);
        }
        schoonersockio.writeBuf.put(B_RETURN);
        byte abyte1[] = (new Integer(k)).toString().getBytes();
        int l = schoonersockio.writeBuf.position();
        schoonersockio.writeBuf.position(j);
        schoonersockio.writeBuf.put(abyte1);
        schoonersockio.writeBuf.position(l);
        schoonersockio.flush();
        String s3 = (new SockInputStream(schoonersockio, 2147483647)).getLine();
        if(!"STORED\r\n".equals(s3))
            break MISSING_BLOCK_LABEL_538;
        flag1 = true;
        if(schoonersockio != null)
        {
            schoonersockio.close();
            schoonersockio = null;
        }
        ......
    }

Pool.getSocket

public final SchoonerSockIO getSock(String s, Integer integer)
{
    ......
    if(i == 1)
    {
        //如果只有一个server，直接建立与该Server的connection
        SchoonerSockIO schoonersockio = hashingAlg != 3 ? getConnection((String)buckets.get(0)) : getConnection((String)consistentBuckets.get(consistentBuckets.firstKey()));
        return schoonersockio;
    }
    HashSet hashset = new HashSet(Arrays.asList(servers));
    long l = getBucket(s, integer);//利用Hahs算法找到对应的Server
    String s1 = hashingAlg != 3 ? (String)buckets.get((int)l) : (String)consistentBuckets.get(Long.valueOf(l));
    do{
        if(hashset.isEmpty())
            break;
        SchoonerSockIO schoonersockio1 = getConnection(s1);//建立该Server的Connection
        if(schoonersockio1 != null)
            return schoonersockio1;
        ......
    }
}

public final SchoonerSockIO getConnection(String s)
{
    ......
    GenericObjectPool genericobjectpool = (GenericObjectPool)socketPool.get(s);//获取ObjectPool
    SchoonerSockIO schoonersockio;
    try{
        schoonersockio = (SchoonerSockIO)genericobjectpool.borrowObject();//通过池来获取连接
    }catch(Exception exception){
        schoonersockio = null;
    }
    ......
    return schoonersockio;
}

sockoutputstream.write

public final class SockOutputStream extends OutputStream
{
    public final void write(byte abyte0[])throws IOException {
        write(abyte0, 0, abyte0.length);
    }

    public final void write(byte abyte0[], int i, int j) throws IOException
    {
        <strong>//将字符数组写入缓冲区</strong>
        if(j == 0)
            return;
        if(sock.writeBuf.remaining() >= j)
        {
            sock.writeBuf.put(abyte0, i, j);
        } else
        {
            int k = 0;
            boolean flag = false;
            for(int i1 = 0; (i1 = j - k) > 0;)
            {
                int l = sock.writeBuf.remaining();
                l = l >= i1 ? i1 : l;
                if(l == 0)
                    writeToChannel();
                else
                    sock.writeBuf.put(abyte0, i, l);
                k += l;
            }

        }
        count += j;
    }

MemCachedClient.get

实际操作的是AscIIClient的get方法

public class AscIIClient extends MemCachedClient
{
    public Object get(String s)
    {
        return get(s, null);
    }
    ......
    private Object get(String s, String s1, Integer integer, boolean flag)
    {
        SchoonerSockIO schoonersockio;
        String s2;
         
        try{
            s1 = sanitizeKey(s1<strong>);//UTF-8编码</strong>
        }
        ......
        schoonersockio = pool.getSock(s1, integer<strong>);//与set方法类似，利用Hash算法建立与server的socket连接</strong>
        ......
        schoonersockio.writeBuf.clear();
        schoonersockio.writeBuf.put(s2.getBytes());
        schoonersockio.writeBuf.put(B_RETURN);
        schoonersockio.flush();
        i = 0;
        j = 0;
        sockinputstream = new SockInputStream(schoonersockio, 2147483647);
        ......
            byte abyte0[] = sockinputstream.getBuffer();
            if((j & 2) == 2){
                GZIPInputStream gzipinputstream = new GZIPInputStream(new ByteArrayInputStream(abyte0));
                ByteArrayOutputStream bytearrayoutputstream = new ByteArrayOutputStream(abyte0.length);
                byte abyte1[] = new byte[2048];
                int i1;
                while((i1 = gzipinputstream.read(abyte1)) != -1) 
                    bytearrayoutputstream.write(abyte1, 0, i1);
                abyte0 = bytearrayoutputstream.toByteArray();
                gzipinputstream.close();
            }
            if(primitiveAsString || flag)
                obj1 = new String(abyte0, defaultEncoding);
            else
                obj1 = NativeHandler.decode(abyte0, j);
         .... 
        obj3 = obj1;
        return obj3;
    }

总结：本文概要地描述了Memcache的初始化以及对数据的get/set操作流程，中间涉及到利用Key值进行Hash算法获取对应的Server，根据Server对应的对象池获取Socket连接以及SocketInputStream SocketOutputSteam操作数据。

在接下来的系列文章里，会深入研究各个部分的细节。（2015-05-12）

ps:如果您觉得本文对您的学习有所帮助，请多多支持。