手把手教你學習netty源碼及原理

本文通過netty的簡單例子，從源碼視角分析netty工作原理。netty是基於reactor的高性能網絡nio框架，對nio的阻塞、異步、reactor模式不熟悉的同學可以參考上一篇的博文 https://blog.csdn.net/Houson_c/article/details/86114771。

netty的核心組件

channel：對應jdkchannel的抽象，還有其他實現類如epollniochannel，代表一個socket連接的

channelpipeline：是事件處理管道，channel的register、連接、讀寫事件的在pipeline中流通，被channelhandler攔截處理

channelhandler：處理channel事件的邏輯實現，可以用來做解碼、編碼、業務邏輯處理

NioEventLoop/NioEventLoopGroup：線程池抽象，包含worker線程，用來執行io事件handler的代碼和用戶task、定時任務等，一個channel對應一個eventloop避免多線程競爭。

netty使用實踐

先來看一個簡單的netty的客戶端和服務端的實例

服務端代碼：

這個是一個服務端將當前時間返回給客戶端的簡單實例，當客戶端連接到來時將當前時間直接返回。

public class NettyServer {
    private int port;

    public NettyServer(int port) {
        this.port = port;
    }

    public void run() throws Exception {
        // (1)創建主從reactor,線程池
        EventLoopGroup bossGroup = new NioEventLoopGroup(1);
        EventLoopGroup workerGroup = new NioEventLoopGroup(1);
        try {
            // (2)啓動器類
            ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup)
                // (3)指定io類型，epoll，udp，oio,kqueue
                .channel(NioServerSocketChannel.class)
                // (4)創建Inithandler
                .childHandler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    public void initChannel(SocketChannel ch) throws Exception {
                        //（5）定義pipeline處理過程,運行在的worker線程池
                        ch.pipeline().addLast("ahandler",new TimeServerHandler());
                    }
                })
                // (6)配置main reactor屬性，影響連接建立例如socket連接數等
                .option(ChannelOption.SO_BACKLOG, 128)
                //(7)配置subreactor的屬性，影響io處理
                .childOption(ChannelOption.SO_KEEPALIVE, true);

            // (8)Bind and start to accept incoming connections.
            ChannelFuture f = b.bind(port).sync();

            f.channel().closeFuture().sync();
        } finally {
            workerGroup.shutdownGracefully();
            bossGroup.shutdownGracefully();
        }
    }

  
    /**
     *  基於事件驅動的handler
     */
    public static class TimeServerHandler extends ChannelInboundHandlerAdapter {
        //(1)新連接到來
        @Override
        public void channelActive(final ChannelHandlerContext ctx) {
            final ByteBuf time = ctx.alloc().buffer(4);
            time.writeInt((int) (System.currentTimeMillis() / 1000L + 2208988800L));
            // (2)回寫時間數據

            final ChannelFuture f = ctx.writeAndFlush(time);
            f.addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture future) {
                    assert f == future;
                    ctx.close();
                }
            });
        }
        //(3)接受客戶端數據
        @Override
        public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
            super.channelRead(ctx, msg);
        }
        //讀取完成事件，一般不用
        @Override
        public void channelReadComplete(ChannelHandlerContext ctx) throws Exception {
            super.channelReadComplete(ctx);
        }
        //發生異常事件觸發
        @Override
        public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
            cause.printStackTrace();
            ctx.close();
        }
    }
}

    public static void main(String[] args) throws Exception {
        int port = 6666;
        if (args.length > 0) {
            port = Integer.parseInt(args[0]);
        }

        new NettyServer(port).run();
    }

簡單分析上面的服務端代碼，

(1)創建主從reactor,線程池，這裏爲了方便調試，設置線程池線程數量都爲1

(2)創建啓動器類對象，netty的輔助類

(3)指定io類型，epoll，udp，oio,kqueue

(4)創建Inithandler

(5)定義pipeline處理過程,運行在的worker線程池

(6)配置main reactor屬性，影響連接建立例如socket連接數等

(7)配置subreactor的屬性，影響io處理

(8)Bind and start to accept incoming connections.開始接受客戶端連接，進行初始化。

客戶端代碼:

客戶端連接到服務端之後，通過handler直接打印服務端返回的結果。

public class NettyClient {

    public static void main(String[] args) throws InterruptedException {
        String host = "localhost";
        int port = 6666;
        EventLoopGroup workerGroup = new NioEventLoopGroup();

        try {
            Bootstrap b = new Bootstrap(); // (1)
            b.group(workerGroup); // (2)
            b.channel(NioSocketChannel.class); // (3)
            b.option(ChannelOption.SO_KEEPALIVE, true); // (4)
            b.handler(new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addLast(new TimeClientHandler());
                }
            });

            // Start the client.
            ChannelFuture f = b.connect(host, port).sync(); // (5)

            // Wait until the connection is closed.
            f.channel().closeFuture().sync();
        } finally {
            workerGroup.shutdownGracefully();
        }

    }

    public static class TimeClientHandler extends ChannelInboundHandlerAdapter {

        @Override
        public void channelRead(ChannelHandlerContext ctx, Object msg) {
            ByteBuf m = (ByteBuf) msg; // (1)
            try {
            	//打印結果
                long currentTimeMillis = (m.readUnsignedInt() - 2208988800L) * 1000L;
                System.out.println(new Date(currentTimeMillis));
                ctx.close();
            } finally {
                m.release();
            }
        }

        @Override
        public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
            cause.printStackTrace();
            ctx.close();
        }
    }
}

上述代碼可以直接運行可以看到運行結果爲，客戶端輸出：

Sat Feb 09 22:25:01 CST 2019

netty源碼分析

基於上面的例子，來分析一下netty重初始化到客戶端連接建立在到完成讀寫io操作的流程。

源碼環境搭建

下載netty的源碼，如果使用的是maven依賴可以直接通過maven的導入源碼和文檔

添加maven依賴

   		<dependency>
               <groupId>io.netty</groupId>
               <artifactId>netty-all</artifactId>
               <version>4.1.30.Final</version>
           </dependency>

下載源碼和文檔

以調試模式起啓動NettyServer服務端程序，注意idea中使用斷點是設置Thread，方便選擇線程進行調試。因爲boss線程和worker線程都是在不停的運行不選擇線程調試容易迷失，代碼中只用了一個boss和一個worker線程。

NettyServer啓動初始化流程

從NettyServer的例子的代碼來看

// (1)創建主從reactor,線程池
        EventLoopGroup bossGroup = new NioEventLoopGroup(1);
        EventLoopGroup workerGroup = new NioEventLoopGroup(1);

首先，創建NioEventLoopGroup，EventLoopGroup是線程池的封裝。

從繼承關係看出NioEventLopGroup是基於javanio的實現，還有epoll、Kqueue的實現。

NioEventLoopGroup的構造方法直接調用類父類MultithreadEventLoopGroup的構造方法，代碼如下

 protected MultithreadEventLoopGroup(int nThreads, Executor executor, Object... args) {
 		//默認是DEFAULT_EVENT_LOOP_THREADS = NettyRuntime.availableProcessors() * 2
        super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, executor, args);
    }


protected MultithreadEventExecutorGroup(int nThreads, Executor executor,
                                            EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) {
            throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        }

        if (executor == null) {
            //每個executor都是線程池
            executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        }
		//創建EventLoop數組
        children = new EventExecutor[nThreads];

        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
            	//創建NioEventLoop，每個EventLoop創建一個線程
                children[i] = newChild(executor, args);
                success = true;
            } catch (Exception e) {
                // TODO: Think about if this is a good exception type
                throw new IllegalStateException("failed to create a child event loop", e);
            } finally {
                //容錯處理，省略。。。
            }
        }
		//創建Eventloop選擇器，用於選擇一個NioEventloop進行io事件處理
        chooser = chooserFactory.newChooser(children);

        //省略。。。
    }

主要做了這幾步：

DEFAULT_EVENT_LOOP_THREADS = NettyRuntime.availableProcessors() * 2
創建EventLoop數組 children = new EventExecutor[nThreads];
創建NioEventLoop，每個EventLoop創建一個線程， children[i] = newChild(executor, args);
創建Eventloop選擇器，用於選擇一個NioEventloop進行io事件處理

bossgroup和workergroup都是這樣的創建流程。上面幾步中，重點看一下NioEventLoop的創建和初始化過程。

NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
                 SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {
        super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);
        if (selectorProvider == null) {
            throw new NullPointerException("selectorProvider");
        }
        if (strategy == null) {
            throw new NullPointerException("selectStrategy");
        }
        provider = selectorProvider;
        //每個NioEventLoop都創建selector
        final SelectorTuple selectorTuple = openSelector();
        selector = selectorTuple.selector;
        unwrappedSelector = selectorTuple.unwrappedSelector;
        selectStrategy = strategy;
    }

既然每個eventloop是一個線程那麼他的run方法肯定是eventloop的核心邏輯，看一下他的實現

@Override
    protected void run() {
        for (;;) {
            try {
            	//判斷任務隊列中是否有任務待執行的任務
                switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
                    case SelectStrategy.CONTINUE:
                        continue;
                    case SelectStrategy.SELECT:
                        select(wakenUp.getAndSet(false));
                     
                        if (wakenUp.get()) {
                            selector.wakeup();
                        }
                        // fall through
                    default:
                }

                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                if (ioRatio == 100) {
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        runAllTasks();
                    }
                } else {
                    final long ioStartTime = System.nanoTime();
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        final long ioTime = System.nanoTime() - ioStartTime;
                        runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
            // Always handle shutdown even if the loop processing threw an exception.
            try {
                if (isShuttingDown()) {
                    closeAll();
                    if (confirmShutdown()) {
                        return;
                    }
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
        }
    }

run方法中無限循環selector監聽是否有io事件，最終會進入下面的代碼執行io事件處理

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
               
                return;
            }
            
            if (eventLoop != this || eventLoop == null) {
                return;
            }
            // close the channel if the key is not valid anymore
            unsafe.close(unsafe.voidPromise());
            return;
        }

        try {
            int readyOps = k.readyOps();
            
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }

這個方法根據selectionKey不同的的事件最終調用Unsafe的對應方法進行處理，Unsafe是netty內部的輔助類，開發者不應該直接調用該類的方法所以叫unsafe。

這就是EventLoopGroup的初始化過程，先不看具體的事件處理邏輯，繼續看NettyServer啓動的初始化過程的代碼，從上面的例子看出，server啓動是在bind()方法開始的，看下具體的實現。

private ChannelFuture doBind(final SocketAddress localAddress) {
		//初始化和註冊serverSocket到Selector
        final ChannelFuture regFuture = initAndRegister();
        final Channel channel = regFuture.channel();
        if (regFuture.cause() != null) {
            return regFuture;
        }

        if (regFuture.isDone()) {
            // At this point we know that the registration was complete and successful.
            ChannelPromise promise = channel.newPromise();
            //執行bind
            doBind0(regFuture, channel, localAddress, promise);
            return promise;
        } else {
            // Registration future is almost always fulfilled already, but just in case it's not.
            final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);
            regFuture.addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture future) throws Exception {
                    Throwable cause = future.cause();
                    if (cause != null) {
                        // Registration on the EventLoop failed so fail the ChannelPromise directly to not cause an
                        // IllegalStateException once we try to access the EventLoop of the Channel.
                        promise.setFailure(cause);
                    } else {
                        // Registration was successful, so set the correct executor to use.
                        // See https://github.com/netty/netty/issues/2586
                        promise.registered();

                        doBind0(regFuture, channel, localAddress, promise);
                    }
                }
            });
            return promise;
        }
    }

AbstractBootstrap的dobind方法中進行了initAndRegister將ServerScoektChannel初始化和註冊到Selector，然後執行了dobind0執行bind操作，先看initAndRegister

final ChannelFuture initAndRegister() {
        Channel channel = null;
        try {
        	//默認通過反射創建Serverchannel，NioServerScoketChanel
            channel = channelFactory.newChannel();
            //初始化ServerChannel，添加Acceptor到pipeline
            init(channel);
        } catch (Throwable t) {
            //......
        }
		//註冊到selector
        ChannelFuture regFuture = config().group().register(channel);
        if (regFuture.cause() != null) {
            if (channel.isRegistered()) {
                channel.close();
            } else {
                channel.unsafe().closeForcibly();
            }
        }

       //......

        return regFuture;
    }

serverchannel的init

@Override
    void init(Channel channel) throws Exception {
        //配置屬性。。。。。

        p.addLast(new ChannelInitializer<Channel>() {
            @Override
            public void initChannel(final Channel ch) throws Exception {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                	//添加配置的handler
                    pipeline.addLast(handler);
                }

                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                    	//添加Acceptor到ServerSocketChannel的pipline
                        pipeline.addLast(new ServerBootstrapAcceptor(
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });
            }
        });
    }

ServerChannel的register最終會調用到NioEventLoop的Register方法

@Override
    public ChannelFuture register(final ChannelPromise promise) {
        ObjectUtil.checkNotNull(promise, "promise");
        promise.channel().unsafe().register(this, promise);
        return promise;
    }

Channel的大部分操作都是調用的Unsafe的方法實現的，最終會調用AbstractUnsafe的register0進行註冊。

private void register0(ChannelPromise promise) {
            try {
                // check if the channel is still open as it could be closed in the mean time when the register
                // call was outside of the eventLoop
                if (!promise.setUncancellable() || !ensureOpen(promise)) {
                    return;
                }
                boolean firstRegistration = neverRegistered;
                //將channel註冊到selector，最終是通過jdk的nio api實現的（nio實現下）
                doRegister();
                neverRegistered = false;
                registered = true;

                // Ensure we call handlerAdded(...) before we actually notify the promise. This is needed as the
                // user may already fire events through the pipeline in the ChannelFutureListener.
                //添加用戶定義的事件Handler對象
                pipeline.invokeHandlerAddedIfNeeded();

                safeSetSuccess(promise);
                //觸發channelRegister事件，通過pipeline進行觸發，並通過handler鏈進行傳播
                pipeline.fireChannelRegistered();
                // Only fire a channelActive if the channel has never been registered. This prevents firing
                // multiple channel actives if the channel is deregistered and re-registered.
                if (isActive()) {
                    if (firstRegistration) {
                        //觸發channelActive事件，通過pipeline進行觸發，並通過handler鏈進行傳播
                        pipeline.fireChannelActive();
                    } else if (config().isAutoRead()) {
                        // This channel was registered before and autoRead() is set. This means we need to begin read
                        // again so that we process inbound data.
                        //
                        // See https://github.com/netty/netty/issues/4805
                        //註冊讀事件監聽到selector
                        beginRead();
                    }
                }
            } catch (Throwable t) {
                // Close the channel directly to avoid FD leak.
                closeForcibly();
                closeFuture.setClosed();
                safeSetFailure(promise, t);
            }
        }

register0調用doregister最終調用jdk的register方法進行channel的註冊到selector，之後通過 pipeline.invokeHandlerAddedIfNeeded();將用戶定義的編碼器、解碼器、inboundhandler等添加到pipeline中，最後，在pipeline中觸發channelRegister和channelActive事件。

回到dobind方法，在initRegister完成之後，會調用dobind0方法，進行ServerSocket的bind，如下

private static void doBind0(
            final ChannelFuture regFuture, final Channel channel,
            final SocketAddress localAddress, final ChannelPromise promise) {

        // This method is invoked before channelRegistered() is triggered.  Give user handlers a chance to set up
        // the pipeline in its channelRegistered() implementation.
        channel.eventLoop().execute(new Runnable() {
            @Override
            public void run() {
                if (regFuture.isSuccess()) {
                    channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);
                } else {
                    promise.setFailure(regFuture.cause());
                }
            }
        });
    }

調用channel的bind方法進行bind操作，並且是在channel的eventloop中執行，最終是調用了pipeline的bind方法進行bind操作，在pipeline 鏈中的調用流程爲：

pipeline.bind->tail.bind->chennelHandlerContext.bind->head.bind->unsafe.bind->javaChannel.bind

@Override
        public final void bind(final SocketAddress localAddress, final ChannelPromise promise) {
            assertEventLoop();

            if (!promise.setUncancellable() || !ensureOpen(promise)) {
                return;
            }

            // See: https://github.com/netty/netty/issues/576
            if (Boolean.TRUE.equals(config().getOption(ChannelOption.SO_BROADCAST)) &&
                localAddress instanceof InetSocketAddress &&
                !((InetSocketAddress) localAddress).getAddress().isAnyLocalAddress() &&
                !PlatformDependent.isWindows() && !PlatformDependent.maybeSuperUser()) {
                // Warn a user about the fact that a non-root user can't receive a
                // broadcast packet on *nix if the socket is bound on non-wildcard address.
                logger.warn(
                        "A non-root user can't receive a broadcast packet if the socket " +
                        "is not bound to a wildcard address; binding to a non-wildcard " +
                        "address (" + localAddress + ") anyway as requested.");
            }

            boolean wasActive = isActive();
            try {
                doBind(localAddress);
            } catch (Throwable t) {
                safeSetFailure(promise, t);
                closeIfClosed();
                return;
            }

            if (!wasActive && isActive()) {
                invokeLater(new Runnable() {
                    @Override
                    public void run() {
                    //觸發active事件
                        pipeline.fireChannelActive();
                    }
                });
            }

            safeSetSuccess(promise);
        }

其中tail爲TailContext，head爲HeadContext，是netty在pipline的首尾自動添加的Handler，可以看到最終調用的式jdk的channel.bind方法。

bind完成後通過 pipeline.fireChannelActive();在pipline中發出channelActive事件，pipeline的鏈式處理流程爲

DefaultChannelPipeline.fireChannelActive->AbstractChannelHandlerContext.invokeChannelActive->head.invokeChannelActive->自定義的TimeServerHandler.channelActive->tail.channelActive

小結：

Server的啓動流程關鍵節點總結如下：

創建EventLoppGroup，創建EventLopp，每個eventLoop創建selector，啓動select循環
創建ServerChannel，創建對應的pipeline，創建Acceptor並添加到pipeline，
將ServerChannel註冊到Selector，監聽事件操作爲0
添加初始化的事件處理Handler
pipeline觸發channelRegister事件
執行bind
pipeline觸發channelActive事件

初始化流程剔除掉一些細節其實和上一篇文章的server啓動主流程基本是一致的，多了一些pipeline的步驟，這也是netty的優勢所在，這裏面涉及的各個組件如果不熟悉的童鞋可以參考,後續也會繼續推出相關blog
https://waylau.gitbooks.io/essential-netty-in-action/

客戶端建立連接

在Server創建和初始化之後NioEventLoop的select循環已經啓動了，當客戶端socket連接到來時操作系統層面會發出連接事件，Severchannel會監聽該事件並創建客戶端對應的NioSocketChannel連接，再看一下select循環的實現。

@Override
    protected void run() {
        for (;;) {
            try {
                switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
                    case SelectStrategy.CONTINUE:
                        continue;
                    case SelectStrategy.SELECT:
                        select(wakenUp.getAndSet(false));

                        if (wakenUp.get()) {
                            selector.wakeup();
                        }
                        // fall through
                    default:
                }

                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                if (ioRatio == 100) {
                    try {
                        //處理selectionKey的事件
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        runAllTasks();
                    }
                } else {
                    final long ioStartTime = System.nanoTime();
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        //根據iorate確定執行非io任務的事件
                        final long ioTime = System.nanoTime() - ioStartTime;
                        runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
           
        }
    }

很明顯，processSelectedKeys處理了io事件，netty對jdk的SelectedKeys進行了優化會調用，processSelectedKeysOptimized方法，最終到下面的代碼

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
              
                return;
            }
            
            if (eventLoop != this || eventLoop == null) {
                return;
            }
            // close the channel if the key is not valid anymore
            unsafe.close(unsafe.voidPromise());
            return;
        }

        try {
            int readyOps = k.readyOps();
            // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
            // the NIO JDK channel implementation may throw a NotYetConnectedException.
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }

可以看到這裏對SelectionKey.OP_WRITE,SelectionKey.OP_READ,SelectionKey.OP_ACCEPT進行了判斷，我們來看SelectionKey.OP_ACCEPT的處理，調用了unsafe.read()方法進行處理

最終會調用到NioServerSocketChannel的readMessages方法

@Override
    protected int doReadMessages(List<Object> buf) throws Exception {
        //accept客戶端連接
        SocketChannel ch = SocketUtils.accept(javaChannel());

        try {
            if (ch != null) {
                //創建客戶端的channel，把channel返回給ServerBootstrapAcceptor處理
                buf.add(new NioSocketChannel(this, ch));
                return 1;
            }
        } catch (Throwable t) {
            logger.warn("Failed to create a new channel from an accepted socket.", t);

            try {
                ch.close();
            } catch (Throwable t2) {
                logger.warn("Failed to close a socket.", t2);
            }
        }

        return 0;
    }

上述方法在把NioSocketChannel放在buf中返回，然後在pipeline中fireread，而上面server啓動初始化時創建了ServerBootstrapAcceptor並將其添加到了pipeline，最終會調用ServerBootstrapAcceptor的channelRead（）

@Override
        @SuppressWarnings("unchecked")
        public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;
			//給客戶端channel添加用戶定義的handler
            child.pipeline().addLast(childHandler);

            setChannelOptions(child, childOptions, logger);

            for (Entry<AttributeKey<?>, Object> e: childAttrs) {
                child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());
            }

            try {
                //註冊Read監聽，和ServerChannel的register類似的流程
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }

創建完客戶端channel之後會從到worker線程池的選擇一個eventloop，將channel註冊該eventloop的selector上進行read事件監聽，這裏的register步驟和serverchannel的register類似，不過register0是在worker線程池的eventloop執行了。這裏開始連接就建立完成了，調試代碼時，這裏應該要切換成worker線程進行調試了。

連接流程小結：

select循環監聽accept事件
SocketUtils.accept建立連接創建客戶端SocketChannel
Acceptor從childGroup中選擇一個eventloop把客戶端的NioSocketChannel register到eventloop的selector中，eventloop的select循環監聽客戶端channel的io事件

處理讀事件

由上可以知道，客戶端channel建立連接之後所有的操作都在WorkerEvenloopGroup中的NioEventLoop中進行了，再次看NioEventLoop的的run方法（第三次）。

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
              
                return;
            }
            
            if (eventLoop != this || eventLoop == null) {
                return;
            }
            // close the channel if the key is not valid anymore
            unsafe.close(unsafe.voidPromise());
            return;
        }

        try {
            int readyOps = k.readyOps();
            // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
            // the NIO JDK channel implementation may throw a NotYetConnectedException.
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }

workerEventloop的unsafe是AbstractNioByteChannel的子類，而BossGroup的AbstractNioMessageChannel子類，實現如下

@Override
        public final void read() {
            final ChannelConfig config = config();
            if (shouldBreakReadReady(config)) {
                clearReadPending();
                return;
            }
            final ChannelPipeline pipeline = pipeline();
            //使用配置的ByteBufAllocator進行內存肥培
            final ByteBufAllocator allocator = config.getAllocator();
            //內存分配輔助類
            final RecvByteBufAllocator.Handle allocHandle = recvBufAllocHandle();
            allocHandle.reset(config);

            ByteBuf byteBuf = null;
            boolean close = false;
            try {
                do {
                	//可以根據歷史內存分配大小進行guess，推測分配的大小
                    byteBuf = allocHandle.allocate(allocator);
                    //從channel中讀取butes，存在ByteBuf中
                    allocHandle.lastBytesRead(doReadBytes(byteBuf));
                    
                    if (allocHandle.lastBytesRead() <= 0) {
                        // nothing was read. release the buffer.
                        byteBuf.release();
                        byteBuf = null;
                        //判斷是否讀完整個message數據
                        close = allocHandle.lastBytesRead() < 0;
                        if (close) {
                            // There is nothing left to read as we received an EOF.
                            readPending = false;
                        }
                        break;
                    }
					
                    allocHandle.incMessagesRead(1);
                    readPending = false;
                    //pipeline觸發channelRead事件
                    pipeline.fireChannelRead(byteBuf);
                    byteBuf = null;
                } while (allocHandle.continueReading());//循環讀取直到沒有數據爲止

                allocHandle.readComplete();
                 //pipeline觸發fireChannelReadComplete事件
                pipeline.fireChannelReadComplete();

                if (close) {
                    closeOnRead(pipeline);
                }
            } catch (Throwable t) {
                handleReadException(pipeline, byteBuf, t, close, allocHandle);
            } finally {
                // Check if there is a readPending which was not processed yet.
                // This could be for two reasons:
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method
                //
                // See https://github.com/netty/netty/issues/2254
                if (!readPending && !config.isAutoRead()) {
                    removeReadOp();
                }
            }
        }

上面的流程比較簡單，分配內存然後循環讀取channel的數據，直到整個message讀完，ByteBuf的內存分配和回收比較複雜，如果有興趣可以參考netty權威指南，後續繼續寫這方面的blog。

每次讀取都會觸發channelRead，所以用戶自定的handler會對一個message多次收到ChannelRead方法的回調，需要進行粘包處理，後續會詳細介紹。讀取完數據之後會觸發channelReadCompelete方法

處理寫事件

channel寫數據，直接調用writeAndFlush，流程如下

pipeline.writeAndFlush(msg);->tail.writeAndFlush(msg);->abstractChannelContext.invokeWriteAndFlush->HeadContext.write->unsafe.write

最終是調用unsafe的write方法，

@Override
        public final void write(Object msg, ChannelPromise promise) {
            assertEventLoop();

            ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
            if (outboundBuffer == null) {
                // If the outboundBuffer is null we know the channel was closed and so
                // need to fail the future right away. If it is not null the handling of the rest
                // will be done in flush0()
                // See https://github.com/netty/netty/issues/2362
                safeSetFailure(promise, WRITE_CLOSED_CHANNEL_EXCEPTION);
                // release message now to prevent resource-leak
                ReferenceCountUtil.release(msg);
                return;
            }

            int size;
            try {
                msg = filterOutboundMessage(msg);
                size = pipeline.estimatorHandle().size(msg);
                if (size < 0) {
                    size = 0;
                }
            } catch (Throwable t) {
                safeSetFailure(promise, t);
                ReferenceCountUtil.release(msg);
                return;
            }
			//將write的message放在outbuffer中，通過flush，寫到channel
            outboundBuffer.addMessage(msg, size, promise);
        }

unsafe.write將message放在outbuffer中，通過flush，寫到channel，看一下unsafe的flush

調用NioSocketChannel進行寫入，然後調用jdk 的channel寫入到socket

@Override
    protected void doWrite(ChannelOutboundBuffer in) throws Exception {
        SocketChannel ch = javaChannel();
        int writeSpinCount = config().getWriteSpinCount();
        do {
            if (in.isEmpty()) {
                // All written so clear OP_WRITE
                clearOpWrite();
                // Directly return here so incompleteWrite(...) is not called.
                return;
            }

            // Ensure the pending writes are made of ByteBufs only.
            int maxBytesPerGatheringWrite = ((NioSocketChannelConfig) config).getMaxBytesPerGatheringWrite();
            ByteBuffer[] nioBuffers = in.nioBuffers(1024, maxBytesPerGatheringWrite);
            int nioBufferCnt = in.nioBufferCount();

            // Always us nioBuffers() to workaround data-corruption.
            // See https://github.com/netty/netty/issues/2761
            switch (nioBufferCnt) {
                case 0:
                    // We have something else beside ByteBuffers to write so fallback to normal writes.
                    writeSpinCount -= doWrite0(in);
                    break;
                case 1: {
                    // Only one ByteBuf so use non-gathering write
                    // Zero length buffers are not added to nioBuffers by ChannelOutboundBuffer, so there is no need
                    // to check if the total size of all the buffers is non-zero.
                    ByteBuffer buffer = nioBuffers[0];
                    int attemptedBytes = buffer.remaining();
                    //調用jdkchannel write
                    final int localWrittenBytes = ch.write(buffer);
                    if (localWrittenBytes <= 0) {
                        incompleteWrite(true);
                        return;
                    }
                    adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
                    in.removeBytes(localWrittenBytes);
                    --writeSpinCount;
                    break;
                }
                default: {
                    // Zero length buffers are not added to nioBuffers by ChannelOutboundBuffer, so there is no need
                    // to check if the total size of all the buffers is non-zero.
                    // We limit the max amount to int above so cast is safe
                    long attemptedBytes = in.nioBufferSize();
                    final long localWrittenBytes = ch.write(nioBuffers, 0, nioBufferCnt);
                    if (localWrittenBytes <= 0) {
                        incompleteWrite(true);
                        return;
                    }
                    // Casting to int is safe because we limit the total amount of data in the nioBuffers to int above.
                    adjustMaxBytesPerGatheringWrite((int) attemptedBytes, (int) localWrittenBytes,
                            maxBytesPerGatheringWrite);
                    in.removeBytes(localWrittenBytes);
                    --writeSpinCount;
                    break;
                }
            }
        } while (writeSpinCount > 0);

        incompleteWrite(writeSpinCount < 0);
    }

如果沒有將buffer中的數據全部寫完則，註冊write事件到selector進行監聽

protected final void incompleteWrite(boolean setOpWrite) {
        // Did not write completely.
        if (setOpWrite) {
            setOpWrite();
        } else {
            // It is possible that we have set the write OP, woken up by NIO because the socket is writable, and then
            // use our write quantum. In this case we no longer want to set the write OP because the socket is still
            // writable (as far as we know). We will find out next time we attempt to write if the socket is writable
            // and set the write OP if necessary.
            clearOpWrite();

            // Schedule flush again later so other tasks can be picked up in the meantime
            eventLoop().execute(flushTask);
        }
    }

再次看NioEventLoop的是如何處理write事件的，

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
              
                return;
            }
            
            if (eventLoop != this || eventLoop == null) {
                return;
            }
            // close the channel if the key is not valid anymore
            unsafe.close(unsafe.voidPromise());
            return;
        }

        try {
            int readyOps = k.readyOps();
            // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
            // the NIO JDK channel implementation may throw a NotYetConnectedException.
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }

可以看到直接調用了unsafe的flush0，flush0的操作見上面的代碼，當然每次write都是調用outboundhandler的回調方法。

if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

到此，讀寫事件處理也基本介紹完了。

總結

本篇文章主要梳理了netty基於Jdk NIo實現的工作流程，對其他的Nio實現其實是大同小異的，這裏不進行分析了，有興趣的童鞋可以自行參考上面的流程進行閱讀源碼。當然，對netty組件的細節實現還是有待更深入的考究，後續會對以下內容進行單獨分析。

ByteBuf的內部管理，內存分配和回收
channalfuture、channelpromise
channelpipeline和channelhandler的管理
channelhandler、encoder、decoder處理粘包拆包
更多netty實戰和相關實用的技術分析

以上是個人對netty這個優秀的網絡nio框架工作原理的理解，歡迎一起交流！

參考

java nio reactor模式https://blog.csdn.net/Houson_c/article/details/86114771
netty實戰 https://waylau.gitbooks.io/essential-netty-in-action/
netty 權威指南

手把手教你學習netty源碼及原理

手把手教你學習netty源碼及原理

netty的核心組件

netty使用實踐

netty源碼分析

源碼環境搭建

NettyServer啓動初始化流程

客戶端建立連接

處理讀事件

處理寫事件

總結

985 碩士程序員，空窗 4 個月沒有 Offer！

一文搞懂 Spring 循環依賴

賽博鬥地主——使用大語言模型扮演Agent智能體玩牌類遊戲。

VScode右鍵打開(添加到右鍵)

Android的Window和WindowManager

Android高效加載圖片和緩存策略LRU，DiskLRU

Android消息機制

Android IPC機制之IPC概念、Android 多進程和相關基礎知識

字符串匹配--KMP算法

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結