基於netty、zookeeper手寫RPC框架之四——實現異步請求和連接池管理

上一篇文章說到,現在這種每發起請求一次就新建一個client鏈接,但是如果在併發比較高的情況下,就會造成資源浪費,如果通過client和server進行長期鏈接,把需要處理的請求存到client裏面,並且通過異步的形式返回,便會減少資源浪費。

這裏有兩個主要的問題,1、如何實現異步返回?2、如何把client和server對應起來?

先看第一個問題,所謂的異步返回,可以以點奶茶爲例,當顧客(消費方)向奶茶店(服務端)點了一杯奶茶(發起請求),奶茶店給他一個號碼(憑證),然後客戶拿着號就去逛街幹其他事情,過了一段時間回來問奶茶店有沒有好了,此時奶茶店根據他的號碼來給他對應的奶茶。

這裏就是爲什麼上篇文章說的請求要帶上requestID,這就相當於憑證的標識,而客戶端鏈接可以保存這些請求並且返回一個憑證,當有返回時,把對應的結果塞到憑證裏面,客戶端可以做自己的事情或者一直等待憑證有結果,這時候結合線程池去做這個處理

第二個問題,需要一個鏈接池來管理鏈接和對應的遠端關係

看看編碼實現

第一個問題最重要的就是如何設計一個憑證,這時候需要用到Future接口來實現自己的憑證,通過一個內部同步器的管理,這個同步器主要是起到自旋改變狀態的作用,同步器因爲只有一個線程節點,會不斷自旋嘗試獲取鎖,獲取鎖的條件是遠端有返回並且改變了同步器的值。



/**
 * @author: lele
 * @date: 2019/11/21 上午10:54
 * 實現異步返回
 */
public class RpcFuture implements Future<Object> {

    private RpcResponse rpcResponse;

    private RpcRequest rpcRequest;

    /**
     * 自定義同步器,這裏只是用來通過自選改變狀態
     */
     private Sync sync;

    public RpcFuture(RpcRequest rpcRequest) {
        this.rpcRequest = rpcRequest;
        this.sync = new Sync();
    }


    @Override
    public boolean cancel(boolean mayInterruptIfRunning) {
        throw new UnsupportedOperationException();
    }

    @Override
    public boolean isCancelled() {
        throw new UnsupportedOperationException();
    }

    /**
     * 返回狀態是否改變了
     * @return
     */
    @Override
    public boolean isDone() {
        return sync.isDone();
    }

    /**
     * 賦值並設置同步器鎖狀態爲1
     * @param response
     */
    public void done(RpcResponse response) {
        this.rpcResponse = response;
        sync.tryRelease(1);

    }


    /**
     * 自選等待結果,這裏一直執行acquire
     * 直到tryacquire方法return true即state爲1
     * @return
     * @throws InterruptedException
     * @throws ExecutionException
     */
    @Override
    public Object get() throws InterruptedException, ExecutionException {
        sync.acquire(-1);
        return this.rpcResponse;

    }

    /**
     * 超時拋異常
     * @param timeout
     * @param unit
     * @return
     * @throws InterruptedException
     * @throws ExecutionException
     * @throws TimeoutException
     */
    @Override
    public Object get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException {
      //超時獲取
        boolean success = sync.tryAcquireNanos(-1, unit.toNanos(timeout));
        return get(success);
    }

    private Object get(boolean success) {
        if (success) {
            return this.rpcResponse;
        } else {
            throw new RuntimeException("超時:requestID" + rpcRequest.getRequestId() +
                    " method:" + rpcRequest.getMethodName() + " interface:" + rpcRequest.getInterfaceName()
            );
        }
    }



    /**
     * 繼承同步器,這裏只是用來自旋改變狀態,根據state來實現,state初始爲0
     */
    static class Sync extends AbstractQueuedSynchronizer {
        /**
         * 嘗試獲取鎖,如果獲取不了,加入同步隊列,阻塞自己,只由同步隊列的頭自旋獲取鎖
         * 當狀態爲1,即有結果返回時可以獲取鎖進行後續操作,設置result
         *這裏只有一個節點,會不斷自選嘗試獲取鎖
         * @param arg
         * @return
         */
        @Override
        protected boolean tryAcquire(int arg) {
            return getState() == 1;
        }

        /**
         * 用於遠端有返回時,設置狀態變更
         * 從頭喚醒同步隊列的隊頭下一個等待的節點,如果下一個節點爲空,則從隊尾喚醒
         * @param arg
         * @return
         */
        @Override
        protected boolean tryRelease(int arg) {
            //把狀態設置爲1,給tryAcquire獲取鎖進行操作
            return getState() == 0 ? compareAndSetState(0, 1) : true;
        }

        public boolean isDone() {
            return getState() == 1;
        }
    }

}

憑證寫好了,接下來就寫處理憑證的handler類,這個類有一個發送請求的方法,主要是把憑證和對應的標識存儲起來,並在有結果返回時設置返回狀態

package com.gdut.rpcstudy.demo.framework.protocol.netty.asyn;

import com.gdut.rpcstudy.demo.framework.URL;
import com.gdut.rpcstudy.demo.framework.protocol.netty.asyn.RpcFuture;
import com.gdut.rpcstudy.demo.framework.serialize.tranobject.RpcRequest;
import com.gdut.rpcstudy.demo.framework.serialize.tranobject.RpcResponse;
import io.netty.buffer.Unpooled;
import io.netty.channel.*;
import lombok.Data;
import lombok.EqualsAndHashCode;

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;

/**
 * @author: lele
 * @date: 2019/11/21 下午4:07
 * 異步模式下的處理
 */
@Data
@EqualsAndHashCode(callSuper = false)
public class NettyAsynHandler extends SimpleChannelInboundHandler<RpcResponse> {
    //key:requestId,value自定義future
    private ConcurrentHashMap<String, RpcFuture> resultMap = new ConcurrentHashMap<>();

    private volatile Channel channel;

    //對應的遠端URL
    private final URL url;


    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        this.channel = ctx.channel();
    }

    public void close() {
        this.channel.close();
    }

    @Override
    protected void channelRead0(ChannelHandlerContext channelHandlerContext, RpcResponse s) throws Exception {
        System.out.println("收到結果:" + s);
        String requestId = s.getRequestId();
        //設置完成並移除future
        RpcFuture future = resultMap.get(requestId);
        if (s != null) {
            future.done(s);
            resultMap.remove(requestId);
        }
    }


    public RpcFuture sendRequest(RpcRequest rpcRequest) {
        final CountDownLatch latch = new CountDownLatch(1);
        RpcFuture future = new RpcFuture(rpcRequest);
        //放到請求列表裏面
        resultMap.put(rpcRequest.getRequestId(), future);
        //發送請求
        channel.writeAndFlush(rpcRequest).addListener(new ChannelFutureListener() {
            @Override
            public void operationComplete(ChannelFuture channelFuture) throws Exception {
                System.out.println("發送了消息" + rpcRequest.toString());
                latch.countDown();
            }
        });
        try {
            //等待結果
            latch.await();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return future;
    }
}

第一個問題的兩個比較核心的類就編寫到這裏。

然後到第二個問題,如何實現一個鏈接管理池?要爲每個服務下的每個server建立一個客戶端,則需要先通過註冊中心獲取地址,再進行後續操作,這裏獲取每個服務下的server地址

 public static Map<String,List<URL>> getAllURL(){
        Map<String,List<URL>> mapList=null;
        try {
            List<String> servcieList=client.getChildren().forPath("/");
            mapList=new HashMap<>(servcieList.size());
            for (String s : servcieList) {
                mapList.put(s,getService(s));
            }
        
        } catch (Exception e) {
            e.printStackTrace();
        }
        return mapList;
    }

獲取地址後,我們可以建立關係了,採用{serviceName:{serverAddress:clienthandler}}的形式進行存儲,從client handler存放的就是上面說的客戶端處理器,通過這裏我們可以通過不同的選擇方式挑一個出來發送請求,這裏採用的選擇方式是輪詢。

大概是這樣的形式去獲取憑證,管理池根據服務名輪詢地選取一個客戶端,並且把憑證讓他處理,然後返回憑證,可這裏有個隱患,客戶端連接到服務端是需要時間的,再沒有可用的客戶端情況下,會報空指針異常

 public RpcFuture sendFuture(String serviceName, RpcRequest request) {
        NettyAsynHandler handler=ConnectManager.getInstance().getConnectionWithPolling(serviceName);
        RpcFuture future = handler.sendRequest(request);
        return future;
    }

這時候我們需要爲加入鎖和條件隊列(Condition),當滿足條件時,纔可以鏈接,不然就自旋等待直到條件滿足,當客戶端鏈接上u對應的server,我們纔可以獲取對應的客戶端處理器,但是服務有這麼多,用一個鎖是可以,但是調用不相干的服務也要被鎖嗎?這時候可以細粒度化我們的鎖,爲每個服務建立一個鎖及其相應的條件隊列。這是連接池需要注意的點,下面看看代碼


/**
 * @author: lele
 * @date: 2019/11/21 上午11:58
 * 管理連接池
 */
public class ConnectManager {


    private Boolean isShutDown = false;

    /**
     * 客戶端鏈接服務端超時時間
     */
    private long connectTimeoutMillis = 5000;

    /**
     * 自定義6個線程組用於客戶端服務
     */
    private EventLoopGroup eventLoopGroup = new NioEventLoopGroup(6);

    /**
     * 存放服務對應的訪問數,用於輪詢
     */
    private Map<String, AtomicInteger> pollingMap = new ConcurrentHashMap<>();

    /**
     * 對於每個服務都有一個鎖,每個鎖都有個條件隊列,用於控制鏈接獲取
     */
    private Map<String, Object[]> serviceCondition = new ConcurrentHashMap<>();

    /**
     * 存放服務端地址和handler的關係
     */
    private Map<String, Map<URL, NettyAsynHandler>> serverClientMap = new ConcurrentHashMap<>();

    /**
     * 用來初始化客戶端
     */
    private ThreadPoolExecutor clientBooter = new ThreadPoolExecutor(
            16, 16, 600, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1024)
            , new BooterThreadFactory(), new ThreadPoolExecutor.AbortPolicy());

    private static class Holder {
        private static final ConnectManager j = new ConnectManager();
    }

    private ConnectManager() {
        //初始化時把所有的url加進去,這裏可能沒有可用鏈接,所以需要添加對節點的監聽
        Map<String, List<URL>> allURL = ZkRegister.getAllURL();
        for (String s : allURL.keySet()) {
            //爲每個服務添加鎖和條件隊列,通過條件隊列控制客戶端鏈接獲取
            ReentrantLock lock =new ReentrantLock();
            Condition condition = lock.newCondition();
            serviceCondition.put(s,new Object[]{lock,condition});
        }
        addServer(allURL);
    }

    public static ConnectManager getInstance() {

        return Holder.j;
    }


    /**
     * 添加該服務對應的鏈接和handler
     * @param serviceName
     * @param url
     * @param handler
     */
    public void addConnection(String serviceName, URL url, NettyAsynHandler handler) {
        Map<URL, NettyAsynHandler> handlerMap;
        if (!serverClientMap.containsKey(serviceName)) {
            handlerMap = new HashMap<>();
        } else {
            handlerMap = serverClientMap.get(serviceName);
        }

        handlerMap.put(url, handler);
        //添加服務名和對應的url:客戶端鏈接
        serverClientMap.put(serviceName, handlerMap);
        //喚醒等待客戶端鏈接的線程
        signalAvailableHandler(serviceName);
    }

    /**
     * 獲取對應服務下的handler,通過輪詢獲取
     * @param servicName
     * @return
     */
    public NettyAsynHandler getConnectionWithPolling(String servicName) {
        Map<URL, NettyAsynHandler> urlNettyAsynHandlerMap = serverClientMap.get(servicName);
        int size = 0;
        //先嚐試獲取
        if (urlNettyAsynHandlerMap != null) {
            size = urlNettyAsynHandlerMap.size();
        }
        //不行就自選等待
        while (!isShutDown && size <= 0) {
            try {
                //自旋等待可用服務出現,因爲客戶端與服務鏈接需要一定的時間,如果直接返回會出現空指針異常
                boolean available = waitingForHandler(servicName);
                if (available) {
                    urlNettyAsynHandlerMap = serverClientMap.get(servicName);
                    size = urlNettyAsynHandlerMap.size();
                }
            } catch (InterruptedException e) {
                throw new RuntimeException("出錯", e);
            }
        }
        //獲取對應的訪問次數
        AtomicInteger count = pollingMap.get(servicName);
        int index = (count.getAndAdd(1) + size) % size;

        Iterator<Map.Entry<URL, NettyAsynHandler>> iterator = urlNettyAsynHandlerMap.entrySet().iterator();
        //取出相應的handler
        NettyAsynHandler nettyAsynHandler = null;
        for (int i = 0; i <= index; i++) {
            nettyAsynHandler = iterator.next().getValue();
        }
        return nettyAsynHandler;
    }

    /**
     * 等待一定時間,等handler和相應的server建立建立鏈接,用條件隊列控制
     * @param serviceName
     * @return
     * @throws InterruptedException
     */
    private boolean waitingForHandler(String serviceName) throws InterruptedException {
        Object[] objects = serviceCondition.get(serviceName);
        ReentrantLock lock = (ReentrantLock) objects[0];
        lock.lock();
        Condition condition= (Condition) objects[1];
        try {
            return condition.await(this.connectTimeoutMillis, TimeUnit.MILLISECONDS);
        } finally {
            lock.unlock();
        }
    }

    public void removeURL(URL url) {
        List<String> list = new ArrayList<>();
        for (Map.Entry<String, Map<URL, NettyAsynHandler>> map : serverClientMap.entrySet()) {
            for (Map.Entry<URL, NettyAsynHandler> urlNettyAsynHandlerEntry : map.getValue().entrySet()) {
                if (urlNettyAsynHandlerEntry.getKey().equals(url)) {
                    urlNettyAsynHandlerEntry.getValue().close();
                    list.add(map.getKey() + "@" + urlNettyAsynHandlerEntry.getKey());
                }
            }
        }
        for (String s : list) {
            String[] split = s.split("@");
            serverClientMap.get(split[0]).remove(split[1]);
        }

    }

    /**
     * 釋放對應服務的條件隊列,代表有客戶端鏈接可用了
     * @param serviceName
     */
    private void signalAvailableHandler(String serviceName) {
        Object[] objects = serviceCondition.get(serviceName);
        ReentrantLock lock = (ReentrantLock) objects[0];
        lock.lock();
        Condition condition= (Condition) objects[1];
        try {
            condition.signalAll();
        } finally {
            lock.unlock();
        }
    }

    /**
     * 添加server,並啓動對應的服務器
     * @param allURL
     */
    public void addServer(Map<String, List<URL>> allURL) {

        for (String s : allURL.keySet()) {
            pollingMap.put(s, new AtomicInteger(0));
            List<URL> urls = allURL.get(s);
            for (URL url : urls) {
                //提交創建任務
                clientBooter.submit(new Runnable() {
                    @Override
                    public void run() {
                        createClient(s, eventLoopGroup, url);
                    }
                });
            }
        }
        System.out.println("初始化客戶端ing");
    }

    /**
     * 創建客戶端,持久化鏈接
     * @param serviceName
     * @param eventLoopGroup
     * @param url
     */
    public void createClient(String serviceName, EventLoopGroup eventLoopGroup, URL url) {
        Bootstrap b = new Bootstrap();
        b.group(eventLoopGroup)
                .channel(NioSocketChannel.class)
                .handler((new ChannelInitializer<SocketChannel>() {
                    @Override
                    protected void initChannel(SocketChannel ch) throws Exception {
                        ch.pipeline()
                                //把request實體變爲字節
                                .addLast(new RpcEncoder(RpcRequest.class))
                                //把返回的response字節變爲對象
                                .addLast(new RpcDecoder(RpcResponse.class))
                                .addLast(new NettyAsynHandler(url));
                    }
                }));

        ChannelFuture channelFuture = b.connect(url.getHostname(), url.getPort());

        channelFuture.addListener(new ChannelFutureListener() {
            @Override
            public void operationComplete(final ChannelFuture channelFuture) throws Exception {
                //鏈接成功後的操作,把相應的url地址和客戶端鏈接存入
                if (channelFuture.isSuccess()) {
                    NettyAsynHandler handler = channelFuture.channel().pipeline().get(NettyAsynHandler.class);
                    addConnection(serviceName, url, handler);
                }
            }
        });
    }


    /**
     * 關閉方法,關閉每個客戶端鏈接,釋放所有鎖,關掉創建鏈接的線程池,和客戶端的處理器
     */
    public void stop() {
        isShutDown = true;
        for (Map<URL, NettyAsynHandler> urlNettyAsynHandlerMap : serverClientMap.values()) {
            urlNettyAsynHandlerMap.values().forEach(e -> e.close());
        }
        for(String s:serviceCondition.keySet()){
            signalAvailableHandler(s);
        }
        clientBooter.shutdown();
        eventLoopGroup.shutdownGracefully();
    }


    /**
     * 啓動客戶端鏈接的自定義線程工廠
     */
    static class BooterThreadFactory implements ThreadFactory {

        private static final AtomicInteger poolNumber = new AtomicInteger(1);
        private final ThreadGroup group;
        private final AtomicInteger threadNumber = new AtomicInteger(1);
        private final String namePrefix;

        BooterThreadFactory() {
            group = new ThreadGroup("connectManger");
            group.setDaemon(false);
            group.setMaxPriority(5);
            namePrefix = "clientBooter-" +
                    poolNumber.getAndIncrement() +
                    "-thread-";
        }

        @Override
        public Thread newThread(Runnable r) {
            Thread t = new Thread(group, r,
                    namePrefix + threadNumber.getAndIncrement(),
                    0);
            return t;
        }
    }

}

這兩個問題解決好後,接下來對整體框架做修改,首先是接口註解,增加了異步模式,對象工廠根據註解來選擇不同的代理方式,

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)

//用於接口上,name爲服務名,zk則在註冊服務改爲 服務名/ip,服務端通過傳來的接口名通過反射獲取類,或者通過給spring託管獲取其class
public @interface RpcStudyClient {
    String name();
    //結果返回是異步還是同步模式
    int mode() default sync;
    int sync=0;
    int asyn=1;

}

@Data
@EqualsAndHashCode(callSuper = false)
public class RpcStudyClientFactoryBean implements FactoryBean<Object> {
    private Class<?> type;

    @Override
    public Object getObject() throws Exception {
        //根據RpcStudeClient的mode字段選擇以哪種方式代理
        RpcStudyClient annotation = type.getAnnotation(RpcStudyClient.class);
        int mode = annotation.mode();
        return mode==RpcStudyClient.asyn?ProxyFactory.getAsyncProxy(this.type):ProxyFactory.getProxy(this.type);
    }

    @Override
    public Class<?> getObjectType() {
        return this.type;
    }
}

代理工廠的新增異步代理,這裏需要自旋獲取結果,不然會導致結果獲取不到。

 public static <T> T getAsyncProxy(Class interfaceClass) {
        return (T) Proxy.newProxyInstance(interfaceClass.getClassLoader(), new Class[]{interfaceClass}, new InvocationHandler() {
            @Override
            public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {

                //指定所用協議
                Protocol protocol = ProtocolFactory.netty();
                RpcStudyClient annotation = (RpcStudyClient) interfaceClass.getAnnotation(RpcStudyClient.class);
                String requestId = UUID.randomUUID().toString().replace("-", "");
                //封裝方法參數
                RpcRequest rpcRequest = new RpcRequest(requestId, interfaceClass.getName(), method.getName(), args, method.getParameterTypes(), annotation.mode());
                Future<RpcFuture> res = futureTask.submit(new Callable<RpcFuture>() {

                    @Override
                    public RpcFuture call() throws Exception {
                        RpcFuture res = protocol.sendFuture(annotation.fetch(), annotation.name(), rpcRequest);
                        //先嚐試一次
                        if (res.isDone()) {
                            return res;
                        }
                        //不行就自旋等待
                        while (!res.isDone()) {

                        }
                        return res;
                    }
                    ;
                });

                //發送請求
                //這裏的管理連接池通過服務名去訪問zk,獲取可用的url
                return returnResult(res.get());
            }
        });
    }

    /**
     * 具體的處理異步返回的方法
     * @param res
     * @return
     * @throws ExecutionException
     * @throws InterruptedException
     */
    public static Object returnResult(RpcFuture res) throws ExecutionException, InterruptedException {
        RpcResponse response = (RpcResponse) res.get();
        if (response.getError() != null) {
            throw new RuntimeException(response.getError());
        } else {
            return response.getResult();
        }
    }

請求實體類也增加多一個屬性mode,標識是同步還是異步模式,給serverhandler做處理,如果是異步,則不關閉鏈接,不然會導致客戶端發的請求無法接受,同步則關閉。

@Data
@AllArgsConstructor
public class RpcRequest  {

    private String requestId;

    private String interfaceName;

    private String methodName;

    private Object[] params;
    //防止重載
    private Class[] paramsTypes;
    //是否異步
    private int mode;
}


 
        ctx.writeAndFlush(response).addListener(new ChannelFutureListener() {
            @Override
            public void operationComplete(ChannelFuture channelFuture) throws Exception {
                System.out.println("發送了結果"+response);
//如果是mode爲同步,就直接關閉連接
                if(rpcRequest.getMode()==RpcStudyClient.sync){
                    ctx.channel().close();
                }
//   當異步模式時,不關閉鏈接
            }
        });

這裏還可以爲異步請求添加回調處理,慢請求日誌記錄等等,都可以在自定義的憑證裏面下文章。異步請求和連接池大概就到這裏,具體代碼https://github.com/97lele/rpcstudy/tree/withconcurrent

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章