如何使用Eclipse內存分析工具定位內存泄露

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文以我司生產環境Java應用內存泄露爲案例進行分析,講解如何使用Eclipse的MAT分析定位問題","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"11月10號晚上8點收到報警郵件,一看是OOM","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打開公司監控系統查看應用各項指標發現JVM中老年代在持續增長(從上次發佈10月30號到11月10號的12天內一直在增長, 存在內存泄露跡象)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e7/e701977a8e0626542d69b400072f0ffe.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從圖中可以看出, 從10月30號發佈到11月10號oom期間11天老年代一直在緩慢上漲, 雖然有下降, 但整體趨勢是上升的,平均每天泄露約50M內存, 說明每次都無法完全釋放乾淨","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲生產環境的JVM添加了 -XX:+HeapDumpOnOutOfMemoryError 參數,該配置會把dump文件的快照保存下來供後續分析排查問題,也可以使用jmap或jcmd等jvm命令進行dump:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"jmap -dump:format=b,file=文件名 [pid]\njcmd pid GC.heap_dump 文件路徑\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分析內存泄露","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"內存泄露和內存溢出的區別:內存泄露從老年代的增長情況看是緩慢上升的, 最終達到老年代上限纔會導致溢出,有些內存泄露可能需要很長的時間發生, 所以說內存泄露更隱蔽, 不像內存溢出那樣容易暴露(內存溢出直接拋出OOM), 而且內存長時間得不到釋放會導致服務性能越來越差、gc時間變長、響應變慢:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/45/45f01082f50ad02171a9785a40d00a5d.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從圖中可以看出在12天裏每天大概泄露(增長) 50M 左右, 這種情況下定位泄露原因需要多次dump採集樣本, 然後和上次的比較分析, 即需要多個dump文件進行比較分析才能精確定位問題。 否則很難看出具體泄露的點, 加上dump文件中大部分是正常的內存使用, 會干擾問題的定位, 增加排查難度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以當時的做法是每天固定時間dump一次, 採集足夠多的樣本, 如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b72f6cf739e9f27da68e8d2f5cd2091d.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外測試環境不好重現的主要原因是不清楚是哪個接口調用引起的, 這個Java服務有多個暴露的api, 而且測試環境不方便壓測,壓測量大了, 底層接口熔斷, 壓測量小看不出泄露跡象, 所以得從dump分析入手, 找到問題所在再去測試環境驗證。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏使用Eclipse的memory analysis tool(MAT)工具進行分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把下載到本地的多個dump文件用mat依次打開(“File → Open Heap Dump”), 如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c5/c5bd7cf620c7b2821d2a1a185c978e63.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如我們要分析這3個dump文件(當然你也可以分析更多個, 這樣會更精準), 打開後, 使用compare basket功能找出內存泄露的差異點:","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"使用 compare basket 功能分析內存泄露","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1> 菜單欄 window → compare basket ,打開比較窗口(如果最下面一欄已經有compare basket則這步不需要),如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4cab29676d5c466499624487cb4d4b69.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2> 依次打開3個dump的dashboard面板, 在下方的 Actions一欄點擊\"histogram\"或\"dominator tree\"生成對應的直方圖或支配樹列表,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7e/7e70e15b4a470e3ad61318804d6b9f2a.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直方圖或支配樹都可以列出堆中存活的所有對象,但二者的維度不同, 直方圖按照類型統計, 支配樹是以對象維度統計。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你對項目代碼比較熟悉, 通過直方圖定位內存泄露會更快,因爲它是按照類型全部平鋪開的,如果這個項目不是你負責的, 建議使用支配樹的方式, 因爲支配樹包含了對象之間的引用關係(支配樹視圖可以展開查看內部引用層級)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3> 我們以支配樹做比對, 在最下面一欄的\"Navigation History (window → navigation history)\"裏(直方圖類似)找到在第2步打開的支配樹dominator tree圖標, 右鍵添加到compare basket, 如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/64/64c1387238d4254875da9a6c44d60a77.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4> 重複上面的2, 3步驟依次把其他的dump文件添加到\"compare basket\"欄, 然後點擊右上角的紅色感嘆號, 生成比較結果,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/83/839e3ad7316dbac4771e57fd386c811b.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(注意比較的dump文件的順序,時間最早的在上面,可以通過右上角的上箭頭↑和下箭頭↓調整順序)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"生成的比對結果如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5e/5e85efdaa5c3f8efe44da024a3867602.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shallow Heap一列後面的序號 #0, #1, #2 分別對應:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一個dump文件佔用的shallow size, 第二個dump文件佔用的shallow size , 第三個dump文件佔用的shallow size","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Retained Heap #0, Retained Heap #1, Retained Heap #2 這3列分別對應:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一個dump文件佔用的retained size, 第二個dump文件佔用的retained size , 第三個dump文件佔用的retained size","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Retained Heap的變化趨勢可以看出:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"紅框 圈出的是內存連續增長的對象, 可以通過右邊紅框的retained heap看出內存變大的趨勢","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綠框 圈出的是沒有變化的對象(至少在這3次比較中沒有變化)","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"藍框 圈出的是內存佔用下降的對象","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一般我們主要關注紅框標出的對象, 因爲這部分發生內存泄露的嫌疑最大","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏先區分兩個概念:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Shallow Size","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對象自身佔用的內存大小,不包括它引用的對象。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對非數組類型的對象,它的大小就是對象與它所有的成員變量大小的總和。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對數組類型的對象,它的大小是數組元素對象的大小總和。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Retained Size","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Retained Size=當前對象大小+當前對象可直接或間接引用到的對象的大小總和。(間接引用的含義:A->B->C, C就是間接引用)","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Retained Size就是當前對象被GC後,從Heap上總共能釋放掉的內存。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲這裏我們比較的是支配樹, 所以按照retained heap倒序排列, 從左到右依次爲: retained heap #0 → retained heap #1 → retained heap #2(以最後一個retained heap #2 倒序, 因爲這個是最後一次dump的內存快照, 這樣可以看出內存泄露的增長趨勢)","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"定位內存泄露","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上一步得出的比較結果, 可以看出\"","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"org.apache.tomcat.util.threads.TaskThread http-nio-8080-exec-*","attrs":{}}],"attrs":{}},{"type":"text","text":"\" 有內存泄露的嫌疑, 查看它的引用關係:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/28/28be28f2b346b07a5f84a3f49d2d330b.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"點擊\"with outgoing references\"後逐層展開第一個對象內部的引用關係(以Retained Heap倒序,主要是看retained size排在前面的對象), 如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c7/c7c920c2e68d6b4c264621cd5da9c1dd.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到TaskThead內部有一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"threadLocal","attrs":{}}],"attrs":{}},{"type":"text","text":", threadLocal內部有一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"concurrentHashMap","attrs":{}}],"attrs":{}},{"type":"text","text":",這個map裏存的是我們的日誌相關對象\"","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"com.*.framework.log.FieldAppendedValue","attrs":{}}],"attrs":{}},{"type":"text","text":"\",從下面幾個map裏的key可以確定是我們記錄到日誌系統(ElasticSearch)的對象, 這些日誌對象主要記錄調用接口的請求報文、響應報文、SOA接口名稱等信息,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/405d810db4d307d2fd0e3ed09ff87082.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但爲什麼日誌對象會佔用這麼多內存?而且這裏看到的只是其中一個taskThread裏,繼續展開RESPONSE_CONTENT的val對象FieldAppendedValue內部引用, 如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ad/ad9376c84b0500d09e42b3d7e0f1a44f.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發現","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"FieldAppendedValue","attrs":{}}],"attrs":{}},{"type":"text","text":"內部維護了一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"CopyOnWriteArrayList","attrs":{}}],"attrs":{}},{"type":"text","text":"對象, 這個list裏竟然存放了10674個值,正常來講不可能一次接口請求會有這麼多的日誌對象, 而且接口請求完記錄到ES後, 這部分內存就應該釋放了纔對。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看CopyOnWriteArrayList內部存儲的內容,如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/76e7b49226891e6d23e095a339e1c40d.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨便打開10675箇中的幾個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"FieldAppendedValue","attrs":{}}],"attrs":{}},{"type":"text","text":", 發現內部存放的都是同一個接口的請求響應報文,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4f/4f8254737bfe5918046f07c14a63a917.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以右鍵copy→ value 把值複製出來查看, 接口報文如下:(響應報文)","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"{\n \"ResponseStatus\": {\n \"Timestamp\": \"/Date(1605583909438+0800)/\",\n \"Ack\": \"Success\",\n \"Errors\": [],\n \"Build\": null,\n \"Version\": null,\n \"Extension\": []\n },\n \"downloadUrl\": \"https://ii066.cn/hFGBEW\"\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面那張","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"concurrentHashMap","attrs":{}}],"attrs":{}},{"type":"text","text":"截圖(key : SOA_METHOD_NAME) 可知這個接口名是: getDownloadLink, 也就是說list裏10675個日誌對象存的都是\"getDownloadLink\"這個接口的報文。而且這只是其中一個TaskThead內部情況, 加上全部20個對象, 20 * 10675 大概是213500個接口報文,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d0/d07e913d143e295e67bde556f6202697.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個接口是什麼鬼?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/59/593c9e70ee6eddbcdff30d47fb598120.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"代碼分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看代碼得知這個接口並沒什麼幺蛾子,只是當時的開發同學在調用這個底層接口時新接入了我們部門封裝的SOA組件公共類:","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"AbstractSimpleHandler.java","attrs":{}}],"attrs":{}},{"type":"text","text":"(這個公共類主要是通過模板方法在調用接口時記錄報文日誌埋點、超時時間設置、mock等功能)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這次出現OOM的這個Java項目之前調用soa接口是自己實現了一套公共方法(早於框架之前實現), 也就是說只有這一個接口使用了新的公共類","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"AbstractSimpleHandler","attrs":{}}],"attrs":{}},{"type":"text","text":",其他的接口調用方式還是原來的方式。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新的工具類","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"AbstractSimpleHandler","attrs":{}}],"attrs":{}},{"type":"text","text":"記錄接口報文的代碼是通過調用","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ELKLogUtils.write()","attrs":{}}],"attrs":{}},{"type":"text","text":"實現的, 這個方法的內部大致邏輯如下:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"Object value = HttpContext.get(BEHAVIOR_LOG);\n if (value == null) {\n value = new ConcurrentHashMap<>();\n HttpContext.add(BEHAVIOR_LOG, value);\n }\n\n\nHttpContext內部維護的是一個ThreadLocal:\n\n\n\npublic class HttpContext {\n\n private static final int CONTEXT_DEFAULT_SIZE = 1 << 6;\n\n private static final ThreadLocal> CONTEXT = new ThreadLocal>() {\n @Override\n protected Map initialValue() {\n return new ConcurrentHashMap<>(CONTEXT_DEFAULT_SIZE);\n }\n };\n\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有調用soa底層接口的報文日誌都是通過ThreadLocal內的map存儲的, 然後統一發送到ES日誌系統。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們都知道theadLocal是線程安全的, 但是一般我們的項目都是部署在Tomcat等web容器裏, tomcat維護了一個http線程池, 就是前面截圖的那個TaskThead Http-nio*線程對象,每次前端app發起請求都會從tomcat的線程池裏取一個線程處理前端的請求, 如果複用的是上一個線程, 那他內部的threadLocal沒有清空, 還是會保存上次的報文信息,這樣的話這次請求又會繼續存放接口報文, 就會越積越多。。。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新接入的組件把接口報文存到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"threadLoacl","attrs":{}}],"attrs":{}},{"type":"text","text":"的代碼是在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"AbstractSimpleHandler.java","attrs":{}}],"attrs":{}},{"type":"text","text":"裏的,而清除threadLoacl的代碼是在另外一個公共類","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"BaseService.java","attrs":{}}],"attrs":{}},{"type":"text","text":"裏做的,也就是說要接入新的公共類除了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"AbstractSimpleHandler.java","attrs":{}}],"attrs":{}},{"type":"text","text":"外,還要接入","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"BaseService.java","attrs":{}}],"attrs":{}},{"type":"text","text":" 這個公共類!","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個也是有歷史原因的, 這個Java項目本身比較早, 那時候還沒有我們部門框架的SOA公共類, 所以自己實現了一套,後來使用新的框架組件調用接口的開發小夥伴沒有調研全面, 少接了一個公共類(BaseService)導致了這一問題發生。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以這個問題的根因是threadllocal使用不當引起的內存泄露","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"弄清楚原因後就好辦了, 解決辦法是在請求完接口後主動調用下框架裏的HttpContext.clear(), 清除下框架內部的threadlocal.map即可,當然後續還是要統一接口的調用方式, 不能兩套工具類並存。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"使用 path to gc root 定位業務代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有另外一個內存泄露的嫌疑是\"","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"com.*.common.utils.ITextRendererPoolManager","attrs":{}}],"attrs":{}},{"type":"text","text":"\", 如上面比對結果的圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/72/720731351e36f0a5dd9b4587821cc68f.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單獨在dominator tree支配樹視圖展開如圖所示:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/64/64bbb82ac233e0de7e555648c05d71a9.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ITextRendererPoolManager","attrs":{}}],"attrs":{}},{"type":"text","text":"內部使用了apache的一個對象緩衝池, 目的可能是爲了對象複用, 繼續展開,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5e/5e0dcde4840ee44ddcc9ef9cd0d4ecfc.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發現是pdf的一個工具類:","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"org.xhtmlrenderer.pdf.ITextRenderer","attrs":{}}],"attrs":{}},{"type":"text","text":", 這個開源的pdf工具是我們項目的郵件功能在發送附件的時候生成pdf文檔時引入的一個第三方jar包,開始懷疑是否是這個開源的pdf工具導致的內存泄露, 但是不清楚這個jar包是在哪裏調用的?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏可以通過\"path to gc root\"查看是誰在引用他, 即我們業務代碼調用的地方,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a6/a6480212fffff25a1f51b7478a1171aa.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏先說下\"path to gc root\"選項的含義:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"with all references : 所有引用, 包括強引用, 弱引用, 軟引用, 虛引用","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"exclude weak reference : 排除弱引用","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"exclude soft reference : 排除軟引用","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"。。。。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們知道軟引用, 弱引用這些在發生full gc時可能會被回收掉(回收時機不同, 具體可自行百度), 目的是不造成內存溢出。一般引起內存溢出的都是強引用,所以你可以選擇\"exclude all ptantom/weak/soft reference\"只查看強引用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但在這個案例中","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"pdf.ITextRenderer","attrs":{}}],"attrs":{}},{"type":"text","text":"是被軟引用引用的(從上圖中可以看出), 雖然說軟引用不會導致溢出, 但可能會引起內存一點點上升(軟引用只有在內存不足發生GC時纔會被回收), 這個跟本地緩存還不一樣, 因爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"shareContext","attrs":{}}],"attrs":{}},{"type":"text","text":"對象沒有達到複用的目的, 而且最重要的是它沒有失效機制,只要沒有達到堆最大值或發生full gc就會一直存在, 這樣的話會拖累JVM的性能,所以我選擇\"with all references\"查看所有類型引用:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4a876d0abefa9916adcd720037ec9743.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發現是被PdfUtil這個類引用, 查看代碼發現PdfUtil是我們自己封裝的一個pdf工具類, 這個工具類把創建pdf的ITextRenderer對象緩存到了iTextRendererPoolManager對象池裏, 這樣下次就不需要再重新創建, 代碼大致如下:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {\n iTextRenderer = iTextRendererPoolManager.borrowObject();\n ......\n iTextRenderer.layout();\n } catch (Exception e) {\n LOGGER.error(e);\n } finally {\n if (iTextRenderer != null) {\n iTextRendererPoolManager.returnObject(iTextRenderer);\n }\n }\n\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是在放回對象池前沒有對ITextRenderer裏面的sharedContext屬性清空, 這樣的話下次從對象池裏如果還是獲取到這個對象,就會對ITextRenderer內部的屬性sharedContext繼續疊加。。。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查了下官方使用手冊發現沒有這樣的用法, 所以導致這個問題的原因應該是我們使用的姿勢不對","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決方法一種是繼續使用對象池,但在放回對象池之前先清除下SharedContext, 或者簡單點不再用對象池,每次new一個, 因爲是在方法內部創建的局部變量, 不會逃逸出方法外, 方法調用完就自動釋放了。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"驗證結果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修復上述兩個問題後在測試環境驗證通過然後發佈上線從12月10號一直截止到今天,大概18天裏內存再沒有泄露跡象, 堆外內存(RSS-JVM內存)也穩定下來,如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/91/9176d4e4dfcbd9782cd6118df726c0b6.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4cb4c87e1eb7a8d073153f1c57870213.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/15/1577dd0ae010ac4345348bfb60dc815a.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此, 內存泄露問題算是告一段落。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看git提交記錄發現這個問題在線上存在有一段時間了(10月30號之前就有泄露跡象),之前一直沒報出來主要是每週都有發版,發佈肯定會重啓清空內存,發佈頻繁也就掩蓋了這個問題,所以這個問題其實是一直存在的","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/15/15e8ee548bbefb9c82e4258db77b13b0.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但測試環境又很難重現出來,很少有應用在測試環境壓測10天以上的,壓測頻率高了,接口容易熔斷。。。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且有些泄露也不是\"真正的泄露\", 比如本地緩存的失效策略設置不合理、寫多讀少、內存佔用持續上升,直到觸發拋棄策略等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實下面的三種情況都屬於廣義上的內存泄露:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"仍然具有GC ROOT根引用但從未在應用程序代碼中使用的對象。這也是傳統意義上的內存泄漏","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"對象太多或太大。意味着沒有足夠的堆可用於執行應用程序,因爲內存中保存了太大的對象樹(例如緩存)","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"臨時對象太多。意味着Java代碼中的處理暫時需要太多內存","attrs":{}}]}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種是大家都熟悉的內存泄露,後兩種多半屬於寫代碼不規範,或業務流程上設計不合理或寫代碼時沒充分考慮緩存的使用場景,所以:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"寫代碼時要加強這方面的意識,包括review的人","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發佈上線後要定時監控,及早發現這類問題","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"MAT工具使用相關事項","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用mat前最好把初始化內存設置大一點,因爲一般生產環境的dump文件都比較大,mat內存大小至少要cover住dump文件的大小,否則打開會報錯,配置文件如圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f3/f39828cc141e128588cf1bf6156bae4b.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如下面堆內存的最大(Xmx)最小(Xms)設置爲4G(具體以你dump文件大小爲準):","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"-startup\nplugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar\n--launcher.library\nplugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.300.v20150602-1417\n-vmargs\n-Xms4g\n-Xmx4g\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外最好設置下顯示單位, 以兆 M或G 爲單位更便於理解,如圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e8/e87e2dda227683d6205126ccd8820d97.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/10/102b56808c155629e69c09eb597c0d56.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你想學好JAVA這門技術,也想在IT行業拿高薪,可以進來看看 ,羣裏有:Java工程化、高性能及分佈式、高性能、深入淺出。高架構。性能調優、Spring,MyBatis,Netty源碼分析和大數據等多個知識點。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你想拿高薪的,想學習的,想就業前景好的,想跟別人競爭能取得優勢的,想進阿里面試但擔心面試不過的,你都可以來,加V:psk12221","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" ","attrs":{}},{"type":"text","text":"(小白和廣告勿擾)","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7f/7faf99e2143d7d4ec3dd745fc6b061bf.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"看完三件事❤️","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你覺得這篇內容對你還蠻有幫助,我想邀請你幫我三個小忙:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"點贊,轉發,有你們的 『點贊和評論』,纔是我創造的動力。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"關注公衆號 『 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"java爛豬皮","attrs":{}},{"type":"text","text":" 』,不定期分享原創知識。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"同時可以期待後續文章ing🚀","attrs":{}}]}],"attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/34/34172ad7f3cc8e0f28bd1fc6ca2d2b68.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文來自公衆號:Java老K,互聯網一線java開發老兵,工作10年有餘,夢想敲一輩子代碼,以夢爲碼,不負韶華。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章