Netty源碼解析 -- FastThreadLocal與HashedWheelTimer

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netty源碼分析系列文章已接近尾聲,本文再來分析Netty中兩個常見組件:FastThreadLoca與HashedWheelTimer。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"源碼分析基於Netty 4.1.52","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"FastThreadLocal","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FastThreadLocal比較簡單。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FastThreadLocal和FastThreadLocalThread是配套使用的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FastThreadLocalThread繼承了Thread,FastThreadLocalThread#threadLocalMap 是一個InternalThreadLocalMap,該InternalThreadLocalMap對象只能用於當前線程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"InternalThreadLocalMap#indexedVariables是一個數組,存放了當前線程所有FastThreadLocal對應的值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而每個FastThreadLocal都有一個index,用於定位InternalThreadLocalMap#indexedVariables。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/96/967a02e1c526cf5295e7af4297fd7536.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FastThreadLocal#get","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public final V get() {\n\t// #1\n\tInternalThreadLocalMap threadLocalMap = InternalThreadLocalMap.get();\n\t// #2\n\tObject v = threadLocalMap.indexedVariable(index);\n\tif (v != InternalThreadLocalMap.UNSET) {\n\t\treturn (V) v;\n\t}\n\t// #3\n\treturn initialize(threadLocalMap);\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1","attrs":{}}],"attrs":{}},{"type":"text","text":" 獲取該線程的InternalThreadLocalMap","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果是FastThreadLocalThread,直接獲取FastThreadLocalThread#threadLocalMap。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"否則,從UnpaddedInternalThreadLocalMap.slowThreadLocalMap獲取該線程InternalThreadLocalMap。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意,UnpaddedInternalThreadLocalMap.slowThreadLocalMap是一個ThreadLocal,這裏實際回退到使用ThreadLocal了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" 每個FastThreadLocal都有一個index。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過該index,獲取InternalThreadLocalMap#indexedVariables中存放的值","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3","attrs":{}}],"attrs":{}},{"type":"text","text":" 找不到值,通過initialize方法構建新對象。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到,FastThreadLocal中連hash算法都不用,通過下標獲取對應的值,複雜度爲log(1),自然很快啦。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"HashedWheelTimer","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HashedWheelTimer是Netty提供的時間輪調度器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間輪是一種充分利用線程資源進行批量化任務調度的調度模型,能夠高效的管理各種延時任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單說,就是將延時任務存放到一個環形隊列中,並通過執行線程定時執行該隊列的任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如,","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"環形隊列上有60個格子,","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"執行線程每秒移動一個格子,則環形隊列每輪可存放1分鐘內的任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在有兩個定時任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"task1,32秒後執行","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"task2,2分25秒後執行","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而執行線程當前位於第6格子","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"則task1放到32+6=38格,輪數爲0","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"task2放到25+6=31個,輪數爲2","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"執行線程將執行當前格子輪數爲0的任務,並將其他任務輪數減1。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/852f9978026c760fd68224666b3ba2b4.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"缺點,時間輪調度器的時間精度不高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲時間輪算法的精度取決於執行線程移動速度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如上面例子中執行線程每秒移動一個格子,則調度精度小於一秒的任務就無法準時調用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HashedWheelTimer關鍵字段","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"// 任務執行器,負責執行任務\nWorker worker = new Worker();\n// 任務執行線程\nThread workerThread;\n// HashedWheelTimer狀態, 0 - init, 1 - started, 2 - shut down\nint workerState;\n// 時間輪隊列,使用數組實現\nHashedWheelBucket[] wheel;\n// 暫存新增的任務\nQueue timeouts = PlatformDependent.newMpscQueue();\n// 已取消任務\nQueue cancelledTimeouts = PlatformDependent.newMpscQueue();","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"添加延遲任務","attrs":{}},{"type":"text","text":" HashedWheelTimer#newTimeout","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public Timeout newTimeout(TimerTask task, long delay, TimeUnit unit) {\n ...\n\n // #1\n start();\n\n // #2\n long deadline = System.nanoTime() + unit.toNanos(delay) - startTime;\n\n ...\n HashedWheelTimeout timeout = new HashedWheelTimeout(this, task, deadline);\n timeouts.add(timeout);\n return timeout;\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1","attrs":{}}],"attrs":{}},{"type":"text","text":" 如果HashedWheelTimer未啓動,則啓動該HashedWheelTimer","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HashedWheelTimer#start方法負責是啓動workerThread線程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" startTime是HashedWheelTimer啓動時間","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"deadline是相對HashedWheelTimer啓動的延遲時間","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"構建HashedWheelTimeout,添加到HashedWheelTimer#timeouts","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"時間輪運行","attrs":{}},{"type":"text","text":" Worker#run","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public void run() {\n\t...\n\n\t// #1\n\tstartTimeInitialized.countDown();\n\n\tdo {\n\t\t// #2\n\t\tfinal long deadline = waitForNextTick();\n\t\tif (deadline > 0) {\n\t\t\t// #3\n\t\t\tint idx = (int) (tick & mask);\n\t\t\tprocessCancelledTasks();\n\t\t\tHashedWheelBucket bucket = wheel[idx];\n\t\t\t// #4\n\t\t\ttransferTimeoutsToBuckets();\n\t\t\t// #5\n\t\t\tbucket.expireTimeouts(deadline);\n\t\t\t// #6\n\t\t\ttick++;\n\t\t}\n\t} while (WORKER_STATE_UPDATER.get(HashedWheelTimer.this) == WORKER_STATE_STARTED);\n\n\t// #7\n\t...\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1","attrs":{}}],"attrs":{}},{"type":"text","text":" HashedWheelTimer#start方法阻塞HashedWheelTimer線程直到Worker啓動完成,這裏解除HashedWheelTimer線程阻塞。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" 計算下一格子開始執行的時間,然後sleep到下次格子開始執行時間","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" tick是從HashedWheelTimer啓動後移動的總格子數,這裏獲取tick對應的格子索引。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Long類型足夠大,這裏並不考慮溢出問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4","attrs":{}}],"attrs":{}},{"type":"text","text":" 將HashedWheelTimer#timeouts的任務遷移到對應的格子中","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#5","attrs":{}}],"attrs":{}},{"type":"text","text":" 處理已到期任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#6","attrs":{}}],"attrs":{}},{"type":"text","text":" 移動到下一個格子","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#7","attrs":{}}],"attrs":{}},{"type":"text","text":" 這裏是HashedWheelTimer#stop後的邏輯處理,取消任務,停止時間輪","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"遷移任務","attrs":{}},{"type":"text","text":" Worker#transferTimeoutsToBuckets","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"private void transferTimeoutsToBuckets() {\n\t// #1\n\tfor (int i = 0; i < 100000; i++) {\n\t\tHashedWheelTimeout timeout = timeouts.poll();\n\t\tif (timeout == null) {\n\t\t\t// all processed\n\t\t\tbreak;\n\t\t}\n\t\tif (timeout.state() == HashedWheelTimeout.ST_CANCELLED) {\n\t\t\tcontinue;\n\t\t}\n\t\t// #2\n\t\tlong calculated = timeout.deadline / tickDuration;\n\t\t// #3\n\t\ttimeout.remainingRounds = (calculated - tick) / wheel.length;\n\t\t// #4\n\t\tfinal long ticks = Math.max(calculated, tick); // Ensure we don't schedule for past.\n\t\t// #5\n\t\tint stopIndex = (int) (ticks & mask);\n\n\t\tHashedWheelBucket bucket = wheel[stopIndex];\n\t\tbucket.addTimeout(timeout);\n\t}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1","attrs":{}}],"attrs":{}},{"type":"text","text":" 注意,每次只遷移100000個任務,以免阻塞線程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" 任務延遲時間/每格時間數, 得到該任務需延遲的總格子移動數","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3","attrs":{}}],"attrs":{}},{"type":"text","text":" (總格子移動數 - 已移動格子數)/每輪格子數,得到輪數","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4","attrs":{}}],"attrs":{}},{"type":"text","text":" 如果任務在timeouts隊列放得太久導致已經過了執行時間,則使用當前tick, 也就是放到當前bucket,以便儘快執行該任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#5","attrs":{}}],"attrs":{}},{"type":"text","text":" 計算tick對應格子索引,放到對應的格子位置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"執行到期任務","attrs":{}},{"type":"text","text":" HashedWheelBucket#expireTimeouts","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"public void expireTimeouts(long deadline) {\n\tHashedWheelTimeout timeout = head;\n\n\twhile (timeout != null) {\n\t\tHashedWheelTimeout next = timeout.next;\n\t\t// #1\n\t\tif (timeout.remainingRounds <= 0) {\n\t\t\t// #2\n\t\t\tnext = remove(timeout);\n\t\t\tif (timeout.deadline <= deadline) {\n\t\t\t\t// #3\n\t\t\t\ttimeout.expire();\n\t\t\t} else {\n\t\t\t\tthrow new IllegalStateException(String.format(\n\t\t\t\t\t\t\"timeout.deadline (%d) > deadline (%d)\", timeout.deadline, deadline));\n\t\t\t}\n\t\t} else if (timeout.isCancelled()) {\n\t\t\tnext = remove(timeout);\n\t\t} else {\n\t\t\t// #4\n\t\t\ttimeout.remainingRounds --;\n\t\t}\n\t\ttimeout = next;\n\t}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#1","attrs":{}}],"attrs":{}},{"type":"text","text":" 選擇輪數小於等於0的任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#2","attrs":{}}],"attrs":{}},{"type":"text","text":" 移除任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#3","attrs":{}}],"attrs":{}},{"type":"text","text":" 修改狀態爲過期,並執行任務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"#4","attrs":{}}],"attrs":{}},{"type":"text","text":" 其他任務輪數減1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ScheduledExecutorService使用堆(DelayedWorkQueue)維護任務,新增任務複雜度爲O(logN)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 HashedWheelTimer 新增任務複雜度爲O(1),所以在任務非常多時, HashedWheelTimer 可以表現出它的優勢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是任務較少甚至沒有任務時,HashedWheelTimer的執行線程都需要不斷移動,也會造成性能消耗。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意,HashedWheelTimer使用同一個線程調用和執行任務,如果某些任務執行時間過久,則影響後續定時任務執行。當然,我們也可以考慮在任務中另起線程執行邏輯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外,如果任務過多,也會導致任務長期滯留在HashedWheelTimer#timeouts中而不能及時執行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果您覺得本文不錯,歡迎關注我的微信公衆號,系列文章持續更新中。您的關注是我堅持的動力!","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5c/5cb935abd5751b075beefdc1bf4914a5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章