Netty源碼解析 -- 事件循環機制實現原理

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要分享Netty中事件循環機制的實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"源碼分析基於Netty 4.1"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"EventLoop"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面分享服務端和客戶端啓動過程的文章中說過，Netty通過事件循環機制(EventLoop)處理IO事件和異步任務，簡單來說，就是通過一個死循環，不斷處理當前已發生的IO事件和待處理的異步任務。示例如下"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"while(true) {\n\tprocess(selector.select());\n\n\tprocess(getTask());\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種事件循環機制也是一種常用的IO事件處理機制，包括Redis，Mysql都使用了類似的機制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於異步任務，前面文章說過，EventLoop實現了(jvm)Executor的接口，execute方法可以提供異步任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"register，bind，connect等操作，都會提交一個任務給EventLoop處理。如"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"if (eventLoop.inEventLoop()) {\n\tregister0(promise);\t\n} else {\n\teventLoop.execute(new Runnable() {\t\n\t\tpublic void run() {\n\t\t\tregister0(promise);\n\t\t}\n\t});\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面看一下Netty中事件循環機制相關的類。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutor，事件執行器，負責處理事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutorGroup維護了一個EventExecutor鏈表，它繼承了ScheduledExecutorService，execute方法通過next方法選擇一個EventExecutor，並調用EventLoop#execute處理事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（EventExecutor繼承了EventExecutorGroup，可以看做一個特殊的EventExecutorGroup，其execute方法可以提交一個任務任務）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoop，事件循環器，繼承了EventExecutor，通過循環不斷處理註冊於其上的Channel的IO事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoopGroup接口則繼承了EventExecutorGroup，負責調度EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor實現了EventExecutor，它會創建一個新線程，並在該線程上處理事件，可以理解爲單線程處理器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventExecutorGroup實現EventExecutorGroup，可以理解爲多線程處理器（實際上是維護了多個EventExecutor，一個EventExecutor可以理解爲一個線程），newChild方法構造具體的EventExecutor。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventExecutorGroup可以配置EventExecutor數量，即線程數量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventExecutorChooserFactory.EventExecutorChooser負責選擇一個EventExecutor執行實際操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop繼承了SingleThreadEventExecutor，負責處理NIO事件。所以，一個NioEventLoop對象可以看做是一個線程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop也實現了EventLoop接口，它實現了事件循環機制，是Netty核心類。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MultithreadEventLoopGroup繼承了MultithreadEventExecutorGroup，並實現了EventLoopGroup，其newChild方法構造具體的EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoopGroup#newChild會構建NioEventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"EventLoop各實現類關係如下"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0c/0c798a130a5a981c40b262af818ce0b1.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"啓動"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor關鍵字段"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"private final Queue taskQueue;\t// 待處理異步任務\nprivate volatile Thread thread;\t\t\t\t// EventLoop執行線程，即SingleThreadEventExecutor創建的新線程\nprivate final Executor executor;\t\t\t// java.util.concurrent.Executor，負責創建線程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當我們通過execute方法提交任務時，如果還沒有創建執行新線程，會通過SingleThreadEventExecutor#executor一個新線程，並在新線程中調用run方法(run方法由子類實現，負責實現事件循環機制，新線程是EventLoop真正執行線程)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SingleThreadEventExecutor#execute"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public void execute(Runnable task) {\n\t...\n\n\tboolean inEventLoop = inEventLoop();\n\t// #1\n\taddTask(task);\n\t// #2\n\tif (!inEventLoop) {\n\t\tstartThread();\n\t\t// #3\n\t\tif (isShutdown()) {\n\t\t\t...\n\t\t}\n\t}\n\t// #4\n\tif (!addTaskWakesUp && wakesUpForTask(task)) {\n\t\twakeup(inEventLoop);\n\t}\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1"}]},{"type":"text","text":" 添加任務到待處理列表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2"}]},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"inEventLoop方法，判斷當前線程是否爲EventLoop執行線程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"若當前線程非EventLoop執行線程，調用startThread方法啓動一個新的線程，執行run方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏可以理解爲啓動EventLoop。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3"}]},{"type":"text","text":" 如果當前EventLoop已關閉，拒絕任務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4"}]},{"type":"text","text":" 若當前EventLoop線程阻塞正等待IO事件(Selector#select方法)，調用wakeup方法喚醒線程執行該新增任務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"循環機制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#run方法負責實現NIO事件處理機制。"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"protected void run() {\n\tint selectCnt = 0;\n\t// #1\n\tfor (;;) {\n\n\t\t\tint strategy;\n\t\t\t\n\t\t\t\t// #2\n\t\t\t\tstrategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());\n\t\t\t\tswitch (strategy) {\n\t\t\t\tcase SelectStrategy.CONTINUE:\n\t\t\t\t\tcontinue;\n\n\t\t\t\tcase SelectStrategy.BUSY_WAIT:\n\t\t\t\t\t// fall-through to SELECT since the busy-wait is not supported with NIO\n\n\t\t\t\tcase SelectStrategy.SELECT:\n\t\t\t\t\t// #3\n\t\t\t\t\tlong curDeadlineNanos = nextScheduledTaskDeadlineNanos();\n\t\t\t\t\tif (curDeadlineNanos == -1L) {\n\t\t\t\t\t\tcurDeadlineNanos = NONE; // nothing on the calendar\n\t\t\t\t\t}\n\t\t\t\t\tnextWakeupNanos.set(curDeadlineNanos);\n\t\t\t\t\ttry {\n\t\t\t\t\t\t// #4\n\t\t\t\t\t\tif (!hasTasks()) {\n\t\t\t\t\t\t\tstrategy = select(curDeadlineNanos);\n\t\t\t\t\t\t}\n\t\t\t\t\t} finally {\n\t\t\t\t\t\t// #5\n\t\t\t\t\t\tnextWakeupNanos.lazySet(AWAKE);\n\t\t\t\t\t}\n\t\t\t\t\t// fall through\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\t...\n\t\t\t\n\t\t\t// #6\n\t\t\tselectCnt++;\n\t\t\tcancelledKeys = 0;\n\t\t\tneedsToSelectAgain = false;\n\t\t\tfinal int ioRatio = this.ioRatio;\n\t\t\tboolean ranTasks;\n\t\t\t// #7\n\t\t\tif (ioRatio == 100) {\n\t\t\t\ttry {\n\t\t\t\t\tif (strategy > 0) {\n\t\t\t\t\t\tprocessSelectedKeys();\n\t\t\t\t\t}\n\t\t\t\t} finally {\n\t\t\t\t\t// Ensure we always run tasks.\n\t\t\t\t\tranTasks = runAllTasks();\n\t\t\t\t}\n\t\t\t} else if (strategy > 0) {\n\t\t\t\tfinal long ioStartTime = System.nanoTime();\n\t\t\t\ttry {\n\t\t\t\t\tprocessSelectedKeys();\n\t\t\t\t} finally {\n\t\t\t\t\t// Ensure we always run tasks.\n\t\t\t\t\tfinal long ioTime = System.nanoTime() - ioStartTime;\n\t\t\t\t\tranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tranTasks = runAllTasks(0); // This will run the minimum number of tasks\n\t\t\t}\n\t\t\t// #8\n\t\t\tif (ranTasks || strategy > 0) {\n\t\t\t\tif (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {\n\t\t\t\t\tlogger.debug(\"Selector.select() returned prematurely {} times in a row for Selector {}.\",\n\t\t\t\t\t\t\tselectCnt - 1, selector);\n\t\t\t\t}\n\t\t\t\tselectCnt = 0;\n\t\t\t} else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)\n\t\t\t\tselectCnt = 0;\n\t\t\t}\n\t\t\n\t\t\n\t\t\t// #9\n\t\t\tif (isShuttingDown()) {\n\t\t\t\tcloseAll();\n\t\t\t\tif (confirmShutdown()) {\n\t\t\t\t\treturn;\n\t\t\t\t}\n\t\t\t}\n\t\t\n\t}\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了版面整潔，這裏刪除了異常處理代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1"}]},{"type":"text","text":" 可以看到，這裏通過一個死循環不斷處理IO事件和異步任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2"}]},{"type":"text","text":" 如果當前存在待處理的任務，調用selector.selectNow()，這時會跳出switch語句，往下處理事件和任務，否則返回SelectStrategy.SELECT。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3"}]},{"type":"text","text":" curDeadlineNanos，計算延遲任務隊列中第一個任務的到期執行時間（即最晚還能延遲多長時間執行），沒有任務則返回-1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新nextWakeupNanos爲阻塞時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於頻繁調用(jvm)Selector.wakeup會造成性能消耗，NioEventLoop維護了一個喚醒標識nextWakeupNanos。nextWakeupNanos有三種值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NONE -- 執行線程被阻塞;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AWAKE -- 執行線程未阻塞;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其他值 -- 執行線程被超時阻塞，在指定的時間後喚醒;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#wakeup方法中，只有"},{"type":"codeinline","content":[{"type":"text","text":"nextWakeupNanos.getAndSet(AWAKE) != AWAKE"}]},{"type":"text","text":"成功才調用selector.wakeup()方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時如果還沒有任務加入，則執行select，阻塞線程。select方法返回結果作爲新的strategy。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#5"}]},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"lazySet方法，設置值之後其他線程在短期內還是可能讀到舊值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏將nextWakeupNanos設置爲AWAKE，主要是減少wakeup方法中不必要的wakeup操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以使用lazySet方法也沒有問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#6"}]},{"type":"text","text":" selectCnt增加"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舊版本的Java NIO在Linux Epoll實現上存在bug，(jvm)Selector.select方法可能在沒有任何就緒事件的情況下返回，導致CPU空轉，利用率飆升到100%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是，Netty計算select方法重複調用次數selectCnt，並在selectCnt大於SELECTOR"},{"type":"text","marks":[{"type":"italic"}],"text":"AUTO"},{"type":"text","text":"REBUILD_THRESHOLD配置(默認512)時，重建selector，從而規避該問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"幸好在JDK6"},{"type":"text","marks":[{"type":"italic"}],"text":"6u4，JDK7"},{"type":"text","text":"b12已修復該Bug。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#7"}]},{"type":"text","text":" processSelectedKeys方法處理IO事件，runAllTasks方法處理任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ioRatio表示執行IO事件所佔CPU時間百分比，默認50，"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ioTime * (100 - ioRatio) / ioRatio"}]},{"type":"text","text":"，通過ioTime，ioRatio計算處理任務的CPU時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#8"}]},{"type":"text","text":" 如果執行了任務或者select方法返回有效值，直接重置selectCnt。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"unexpectedSelectorWakeup方法中會在selectCnt大於SELECTOR"},{"type":"text","marks":[{"type":"italic"}],"text":"AUTO"},{"type":"text","text":"REBUILD_THRESHOLD時重建selector。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#9"}]},{"type":"text","text":" 如果是正在關閉狀態，則要關閉所有的Channel"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"IO事件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面看一下Eventloop中如何處理IO事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop關鍵字段"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"Selector unwrappedSelector;\t\t\t\t// JVM中的Selector\nSelector selector;\t\t\t\t\t\t// 優化後的SelectedSelectionKeySetSelector\nSelectedSelectionKeySet selectedKeys;\t// 對(jvm)Selector#selectedKeys進行優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SelectedSelectionKeySetSelector每次調用select前都清除SelectedSelectionKeySet"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SelectedSelectionKeySet使用數組代替原Selector的中的HashSet，提高性能。數組默認大小爲1024，不夠用時擴展爲原大小的2倍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NioEventLoop#構造方法 -> NioEventLoop#openSelector"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"private SelectorTuple openSelector() {\n final Selector unwrappedSelector;\n try {\n\t\t// #1\n unwrappedSelector = provider.openSelector();\n } catch (IOException e) {\n throw new ChannelException(\"failed to open a new selector\", e);\n }\n\n if (DISABLE_KEY_SET_OPTIMIZATION) {\n return new SelectorTuple(unwrappedSelector);\n }\n\n ...\n\n final Class> selectorImplClass = (Class>) maybeSelectorImplClass;\n final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();\n\n Object maybeException = AccessController.doPrivileged(new PrivilegedAction