Netty源码解析 -- 事件循环机制实现原理

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要分享Netty中事件循环机制的实现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"源码分析基于Netty 4.1"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"EventLoop"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面分享服务端和客户端启动过程的文章中说过，Netty通过事件循环机制(EventLoop)处理IO事件和异步任务，简单来说，就是通过一个死循环，不断处理当前已发生的IO事件和待处理的异步任务。示例如下"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"while(true) {\n\tprocess(selector.select());\n\n\tprocess(getTask());\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这种事件循环机制也是一种常用的IO事件处理机制，包括Redis，Mysql都使用了类似的机制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"关于异步任务，前面文章说过，EventLoop实现了(jvm)Executor的接口，execute方法可以提供异步任务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"register，bind，connect等操作，都会提交一个任务给EventLoop处理。如"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"if (eventLoop.inEventLoop()) {\n\tregister0(promise);\t\n} else {\n\teventLoop.execute(new Runnable() {\t\n\t\tpublic void run() {\n\t\t\tregister0(promise);\n\t\t}\n\t});\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面看一下Netty中事件循环机制相关的类。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutor，事件执行器，负责处理事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutorGroup维护了一个EventExecutor链表，它继承了ScheduledExecutorService，execute方法通过next方法选择一个EventExecutor，并调用EventLoop#execute处理事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（EventExecutor继承了EventExecutorGroup，可以看做一个特殊的EventExecutorGroup，其execute方法可以提交一个任务任务）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoop，事件循环器，继承了EventExecutor，通过循环不断处理注册于其上的Channel的IO事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoopGroup接口则继承了EventExecutorGroup，负责调度EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor实现了EventExecutor，它会创建一个新线程，并在该线程上处理事件，可以理解为单线程处理器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventExecutorGroup实现EventExecutorGroup，可以理解为多线程处理器（实际上是维护了多个EventExecutor，一个EventExecutor可以理解为一个线程），newChild方法构造具体的EventExecutor。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventExecutorGroup可以配置EventExecutor数量，即线程数量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutorChooserFactory.EventExecutorChooser负责选择一个EventExecutor执行实际操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop继承了SingleThreadEventExecutor，负责处理NIO事件。所以，一个NioEventLoop对象可以看做是一个线程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop也实现了EventLoop接口，它实现了事件循环机制，是Netty核心类。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventLoopGroup继承了MultithreadEventExecutorGroup，并实现了EventLoopGroup，其newChild方法构造具体的EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoopGroup#newChild会构建NioEventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoop各实现类关系如下"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0c/0c798a130a5a981c40b262af818ce0b1.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"启动"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor关键字段"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"private final Queue taskQueue;\t// 待处理异步任务\nprivate volatile Thread thread;\t\t\t\t// EventLoop执行线程，即SingleThreadEventExecutor创建的新线程\nprivate final Executor executor;\t\t\t// java.util.concurrent.Executor，负责创建线程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当我们通过execute方法提交任务时，如果还没有创建执行新线程，会通过SingleThreadEventExecutor#executor一个新线程，并在新线程中调用run方法(run方法由子类实现，负责实现事件循环机制，新线程是EventLoop真正执行线程)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor#execute"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public void execute(Runnable task) {\n\t...\n\n\tboolean inEventLoop = inEventLoop();\n\t// #1\n\taddTask(task);\n\t// #2\n\tif (!inEventLoop) {\n\t\tstartThread();\n\t\t// #3\n\t\tif (isShutdown()) {\n\t\t\t...\n\t\t}\n\t}\n\t// #4\n\tif (!addTaskWakesUp && wakesUpForTask(task)) {\n\t\twakeup(inEventLoop);\n\t}\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1"}]},{"type":"text","text":" 添加任务到待处理列表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2"}]},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"inEventLoop方法，判断当前线程是否为EventLoop执行线程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若当前线程非EventLoop执行线程，调用startThread方法启动一个新的线程，执行run方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里可以理解为启动EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3"}]},{"type":"text","text":" 如果当前EventLoop已关闭，拒绝任务"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4"}]},{"type":"text","text":" 若当前EventLoop线程阻塞正等待IO事件(Selector#select方法)，调用wakeup方法唤醒线程执行该新增任务"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"循环机制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#run方法负责实现NIO事件处理机制。"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"protected void run() {\n\tint selectCnt = 0;\n\t// #1\n\tfor (;;) {\n\n\t\t\tint strategy;\n\t\t\t\n\t\t\t\t// #2\n\t\t\t\tstrategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());\n\t\t\t\tswitch (strategy) {\n\t\t\t\tcase SelectStrategy.CONTINUE:\n\t\t\t\t\tcontinue;\n\n\t\t\t\tcase SelectStrategy.BUSY_WAIT:\n\t\t\t\t\t// fall-through to SELECT since the busy-wait is not supported with NIO\n\n\t\t\t\tcase SelectStrategy.SELECT:\n\t\t\t\t\t// #3\n\t\t\t\t\tlong curDeadlineNanos = nextScheduledTaskDeadlineNanos();\n\t\t\t\t\tif (curDeadlineNanos == -1L) {\n\t\t\t\t\t\tcurDeadlineNanos = NONE; // nothing on the calendar\n\t\t\t\t\t}\n\t\t\t\t\tnextWakeupNanos.set(curDeadlineNanos);\n\t\t\t\t\ttry {\n\t\t\t\t\t\t// #4\n\t\t\t\t\t\tif (!hasTasks()) {\n\t\t\t\t\t\t\tstrategy = select(curDeadlineNanos);\n\t\t\t\t\t\t}\n\t\t\t\t\t} finally {\n\t\t\t\t\t\t// #5\n\t\t\t\t\t\tnextWakeupNanos.lazySet(AWAKE);\n\t\t\t\t\t}\n\t\t\t\t\t// fall through\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\t...\n\t\t\t\n\t\t\t// #6\n\t\t\tselectCnt++;\n\t\t\tcancelledKeys = 0;\n\t\t\tneedsToSelectAgain = false;\n\t\t\tfinal int ioRatio = this.ioRatio;\n\t\t\tboolean ranTasks;\n\t\t\t// #7\n\t\t\tif (ioRatio == 100) {\n\t\t\t\ttry {\n\t\t\t\t\tif (strategy > 0) {\n\t\t\t\t\t\tprocessSelectedKeys();\n\t\t\t\t\t}\n\t\t\t\t} finally {\n\t\t\t\t\t// Ensure we always run tasks.\n\t\t\t\t\tranTasks = runAllTasks();\n\t\t\t\t}\n\t\t\t} else if (strategy > 0) {\n\t\t\t\tfinal long ioStartTime = System.nanoTime();\n\t\t\t\ttry {\n\t\t\t\t\tprocessSelectedKeys();\n\t\t\t\t} finally {\n\t\t\t\t\t// Ensure we always run tasks.\n\t\t\t\t\tfinal long ioTime = System.nanoTime() - ioStartTime;\n\t\t\t\t\tranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tranTasks = runAllTasks(0); // This will run the minimum number of tasks\n\t\t\t}\n\t\t\t// #8\n\t\t\tif (ranTasks || strategy > 0) {\n\t\t\t\tif (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {\n\t\t\t\t\tlogger.debug(\"Selector.select() returned prematurely {} times in a row for Selector {}.\",\n\t\t\t\t\t\t\tselectCnt - 1, selector);\n\t\t\t\t}\n\t\t\t\tselectCnt = 0;\n\t\t\t} else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)\n\t\t\t\tselectCnt = 0;\n\t\t\t}\n\t\t\n\t\t\n\t\t\t// #9\n\t\t\tif (isShuttingDown()) {\n\t\t\t\tcloseAll();\n\t\t\t\tif (confirmShutdown()) {\n\t\t\t\t\treturn;\n\t\t\t\t}\n\t\t\t}\n\t\t\n\t}\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了版面整洁，这里删除了异常处理代码。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1"}]},{"type":"text","text":" 可以看到，这里通过一个死循环不断处理IO事件和异步任务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2"}]},{"type":"text","text":" 如果当前存在待处理的任务，调用selector.selectNow()，这时会跳出switch语句，往下处理事件和任务，否则返回SelectStrategy.SELECT。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3"}]},{"type":"text","text":" curDeadlineNanos，计算延迟任务队列中第一个任务的到期执行时间（即最晚还能延迟多长时间执行），没有任务则返回-1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新nextWakeupNanos为阻塞时间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于频繁调用(jvm)Selector.wakeup会造成性能消耗，NioEventLoop维护了一个唤醒标识nextWakeupNanos。nextWakeupNanos有三种值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NONE -- 执行线程被阻塞;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AWAKE -- 执行线程未阻塞;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其他值 -- 执行线程被超时阻塞，在指定的时间后唤醒;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#wakeup方法中，只有"},{"type":"codeinline","content":[{"type":"text","text":"nextWakeupNanos.getAndSet(AWAKE) != AWAKE"}]},{"type":"text","text":"成功才调用selector.wakeup()方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这时如果还没有任务加入，则执行select，阻塞线程。select方法返回结果作为新的strategy。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#5"}]},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"lazySet方法，设置值之后其他线程在短期内还是可能读到旧值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里将nextWakeupNanos设置为AWAKE，主要是减少wakeup方法中不必要的wakeup操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以使用lazySet方法也没有问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#6"}]},{"type":"text","text":" selectCnt增加"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"旧版本的Java NIO在Linux Epoll实现上存在bug，(jvm)Selector.select方法可能在没有任何就绪事件的情况下返回，导致CPU空转，利用率飙升到100%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"于是，Netty计算select方法重复调用次数selectCnt，并在selectCnt大于SELECTOR"},{"type":"text","marks":[{"type":"italic"}],"text":"AUTO"},{"type":"text","text":"REBUILD_THRESHOLD配置(默认512)时，重建selector，从而规避该问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"幸好在JDK6"},{"type":"text","marks":[{"type":"italic"}],"text":"6u4，JDK7"},{"type":"text","text":"b12已修复该Bug。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#7"}]},{"type":"text","text":" processSelectedKeys方法处理IO事件，runAllTasks方法处理任务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ioRatio表示执行IO事件所占CPU时间百分比，默认50，"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ioTime * (100 - ioRatio) / ioRatio"}]},{"type":"text","text":"，通过ioTime，ioRatio计算处理任务的CPU时间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#8"}]},{"type":"text","text":" 如果执行了任务或者select方法返回有效值，直接重置selectCnt。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"unexpectedSelectorWakeup方法中会在selectCnt大于SELECTOR"},{"type":"text","marks":[{"type":"italic"}],"text":"AUTO"},{"type":"text","text":"REBUILD_THRESHOLD时重建selector。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#9"}]},{"type":"text","text":" 如果是正在关闭状态，则要关闭所有的Channel"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"IO事件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面看一下Eventloop中如何处理IO事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop关键字段"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"Selector unwrappedSelector;\t\t\t\t// JVM中的Selector\nSelector selector;\t\t\t\t\t\t// 优化后的SelectedSelectionKeySetSelector\nSelectedSelectionKeySet selectedKeys;\t// 对(jvm)Selector#selectedKeys进行优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SelectedSelectionKeySetSelector每次调用select前都清除SelectedSelectionKeySet"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SelectedSelectionKeySet使用数组代替原Selector的中的HashSet，提高性能。数组默认大小为1024，不够用时扩展为原大小的2倍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#构造方法 -> NioEventLoop#openSelector"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"private SelectorTuple openSelector() {\n final Selector unwrappedSelector;\n try {\n\t\t// #1\n unwrappedSelector = provider.openSelector();\n } catch (IOException e) {\n throw new ChannelException(\"failed to open a new selector\", e);\n }\n\n if (DISABLE_KEY_SET_OPTIMIZATION) {\n return new SelectorTuple(unwrappedSelector);\n }\n\n ...\n\n final Class> selectorImplClass = (Class>) maybeSelectorImplClass;\n final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();\n\n Object maybeException = AccessController.doPrivileged(new PrivilegedAction