記一次由Arthas引起的Metaspace OOM問題

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如無特殊說明，本文默認基於以下環境敘述：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JDK: OpenJDK 14GA","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"macOS 10.15","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Arthas 3.3.9","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"VisualVM 2.0.2","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從Arthas 3.4.2開始，此問題已經被修復。感謝Arthas團隊對此問題的重視。","attrs":{}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Arthas是一款由阿里巴巴開源的Java應用程序診斷工具，它功能強大，且不需要對原有的應用做任何改動，即可幫助開發者全方位地觀測Java應用程序的運行狀態，特別是在線上服務不便於調試，問題復現概率低的場景下極大地方便了開發人員的調試工作，因此深受集團內外的開發者喜愛，筆者在工作中也經常使用Arthas幫助定位一些服務運行過程中的問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今年8月中旬，在工作中需要使用Arthas的trace命令統計一個有大量get set及多種接口調用的巨大方法，執行trace命令後，Arthas遲遲沒有顯示命令調用成功的提示，同時連接Arthas的終端失去了響應。嘗試重新連接Arthas，再次進行trace，結果卻彈出了trace失敗的提示:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Enhanceerror! exception: \n\njava.lang.InternalError\n\nerror happens when enhancing\n\nclass: null, check arthas log: \n\n/path/to/server-log/arthas.log","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是查看服務器上的Arthas運行日誌，發現日誌中有以下的異常堆棧:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"java"},"content":[{"type":"text","text":"java.lang.InternalError:nullat\n\nsun.instrument.InstrumentationImpl\n\n.retransformClasses0(NativeMethod)\n\nat\n\nsun.instrument.InstrumentationImpl\n\n.retransformClasses(InstrumentationImpl.java:144)\n\nat\n\ncom.taobao.arthas.core.advisor.Enhancer.enh\n\nance(Enhancer.java:368)\n\nat\n\ncom.taobao.arthas.core.command.mon\n\nitor200.EnhancerCommand.enhance(EnhancerCommand.java:149)\n\nat\n\ncom.taobao.arthas.core.comma\n\nnd.monitor200.EnhancerComma\n\nnd.process(EnhancerCommand.java:96)\n\nat\n\ncom.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)\n\nat com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)\n\nat com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl\n\n$ProcessHandler.handle(AnnotatedCommandImpl.java:111)// ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"幾乎同時，筆者收到了監控平臺發出的目標機器Metaspace OOM的告警，查看服務器監控面板，發現當前JVM的Metaspace已經爆滿。回到開發環境，再次嘗試了幾次相同操作，竟然是穩定復現Metaspace OOM。於是開始着手排查這個問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"問題分析","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"初窺Metaspace結構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目標應用運行在集團基於OpenJDK 8深度定製的AliJDK上，查閱相關文檔知，它和普通的OpenJDK一樣，Metaspace是實現爲堆外內存，因此傳統的Dump heap分析前後堆內對象數量變化的思路便行不通了，只能先從Metaspace的存儲結構入手分析。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Metaspace 主要分爲Non-Class space和Class space兩部分。他們的作用分別如下所示:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b8/b82267030185cb24c87402fe7a26f8fd.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•Class space","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存放Klass對象、vtable, itable, 以及記錄類中非靜態成員引用對象的地址的Map，等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•Klass對象是Java的類在JVM層次的運行時數據結構，當類被加載的時候，會產生一個描述當前類的InstanceKlass對象，這些Klass對象會保存在Metaspace的Class space區域。在Java對象的對象頭中有指向對象所屬類的Klass對象的指針。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•vtable 是爲了實現Java中的虛分派功能而存在。HotSpot把Java中的方法都抽象成了Method對象，InstanceKlass中的成員屬性_methods就保存了當前類所有方法對應的Method實例。HotSpot並沒有顯式地把虛函數表設計爲Klass的field，而是提供了一個虛函數表視圖。在.class文件被解析的過程中會計算vtable的大小，在類被連接的時候會真正產生出vtable。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•itable 記錄的是當一個類有實現接口時，接口方法在vtable中的偏移量。在.class文件被解析的過程中會計算itable的大小，在類被連接的時候會真正產生出itable。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/74/74d99ff080475223f7ffba00887bdda4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•Non-class Space","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個區域有很多的東西，下面這些佔用了最多的空間：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•常量池，可變大小（注意是class文件中的常量池的結構化表示，而不是運行時的String常量）；    •每個成員方法的 Metadata：ConstMethod 結構，包含了好幾個可變大小的內部結構，如方法字節碼、局部變量表、異常表、參數信息、方法簽名等；    •運行時數據，用來控制 JIT 的行爲；    •註解數據等等","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"查看診斷命令輸出","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"瞭解Metaspace中主要存儲的數據後，便可以使用診斷命令去查看Metaspace的內存佔用情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於JDK 8，可以使用命令jstat -gc;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 高版本的 JDK (通常在JDK 12以後), 引入了[1]","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"VM.metaspace","attrs":{}}],"attrs":{}},{"type":"text","text":"診斷命令，","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"jcmd VM.metaspace","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以輸出更爲全面的診斷信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先看trace前的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"jstat","attrs":{}}],"attrs":{}},{"type":"text","text":"輸出：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f61ab957b703497235aac5c2daa56d1a.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到MU大約是95MB左右，CCSU大概在14MB左右。由於MU = Non-class Space + Class space, 因此Non-class space大概在80多MB。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用了高版本的JDK，可以使用","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"VM.metaspace","attrs":{}}],"attrs":{}},{"type":"text","text":"命令查看更詳細的結果：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cb/cbf598e588c3cd89f54ed30fe1c2ea93.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到數據符合之前的預期。接下來看一下trace後的診斷信息：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8e4f63deed5eeafa838bc8994edf0ddb.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發現Non-class區大小激增，而Class區大小及已加載的類數量沒有明顯變化。這一現象說明，引起Metaspace OOM的原因很可能是JVM在解析Arthas增強後的類字節碼數據，向Non-class區放入新生成的方法、常量池等數據時申請了大量的Non-class空間導致的。因此，接下來需要分析增強前後字節碼的區別。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"分析Arthas的命令執行過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲增強後的字節碼是由Arthas輸出並注入到JVM的，在分析之前便需要搞清楚Arthas是如何產生增強後的字節碼的。由於本例中的Arthas是以Agent方式運行的，因此直接看源碼，瞭解ArthasAgent的附加過程：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// arthas-agent-attach/src/main/java/com/taobao/arthas/agent/attach/ArthasAgent.javapublic void init() throws IllegalStateException {// ...// 通過反射調用 ArthasBootstrap bootstrap = ArthasBootstrap.getInstance(inst); Class> bootstrapClass = arthasClassLoader.loadClass(ARTHAS_BOOTSTRAP);Object bootstrap = bootstrapClass.getMethod(GET_INSTANCE, Instrumentation.class, Map.class).invoke(null,instrumentation, configMap);boolean isBind = (Boolean) bootstrapClass.getMethod(IS_BIND).invoke(bootstrap);if (!isBind) {StringerrorMsg = \"Arthas server port binding failed! Please check $HOME/logs/arthas/arthas.log for more details.\";thrownew RuntimeException(errorMsg); }// ...}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終會調用到ArthasBootstrap的構造方法：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"private ArthasBootstrap(Instrumentation instrumentation, Map args) throws Throwable {// ... shutdown = new Thread(\"as-shutdown-hooker\") { @Overridepublic void run(){ArthasBootstrap.this.destroy(); } };// ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"這裏使用先前傳入的instrumentation構造類字節碼的transformerManager。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"transformerManager = new TransformerManager(instrumentation);Runtime.getRuntime().addShutdownHook(shutdown); }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跟入","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"TransformManager","attrs":{}}],"attrs":{}},{"type":"text","text":"可以看到註冊類字節碼增強回調函數的代碼：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"public TransformerManager(Instrumentation instrumentation){this.instrumentation = instrumentation; classFileTransformer = new ClassFileTransformer() { @Overridepublicbyte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {// ...// TraceTransformerfor (ClassFileTransformer classFileTransformer : traceTransformers) {byte[] transformResult = classFileTransformer.transform(loader, className, classBeingRedefined,protectionDomain, classfileBuffer);if (transformResult != null) {classfileBuffer = transformResult; } }return classfileBuffer; } }; instrumentation.addTransformer(classFileTransformer, true); }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很巧的是，這裏有一個traceTransformers。對Arthas源碼進行斷點調試，發現trace操作確實會走到此回調方法。於是在此處修改Arthas的代碼，判斷如果待transform的類是會引發OOM的目標類，那就把","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"classfileBuffer","attrs":{}}],"attrs":{}},{"type":"text","text":"和transform完成的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"transformResult","attrs":{}}],"attrs":{}},{"type":"text","text":"都保存到文件。以此方式順利地拿到了增強前後的字節碼。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3f/3fcc4650741ae76ddd3f523807429172.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"分析增強前後的字節碼結構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新生成的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件比老的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件大了很多。將兩個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件拖入IDEA中進行反編譯，查看對應的Java代碼。由於被trace的方法體本身非常龐大，內部具有大量的DTO轉換操作，充斥着大量的get set方法調用，因此Arthas在生成增強的字節碼時在方法調用前後插入了大量的計時代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0a/0a90b9e07c89416d1bbec722b7275254.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不過仔細看，可以發現，雖然看上去代碼中有非常多的字符串，但是實際上很多字符串都是一模一樣的，只是反編譯過程中重複顯示了而已，這一點可以從","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"的文件大小得出結論：雖然新類中多了不少字符串，但是不同的字符串肯定很少，否則","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件中需要耗費大量的空間去保存這些不一樣的字符串，勢必文件大小也會膨脹得厲害；而現在新類的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件才1M左右，與Metaspace OOM時暴漲500MB的表現實在是相去甚遠，因此並不是常量過多引發Metaspace暴漲。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"既然從反編譯的結果中得不到問題的突破口，於是嘗試使用","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"javap -verbose","attrs":{}}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/79/7934a5081fee2756520d115eb76f12be.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對比兩個前後","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"javap","attrs":{}}],"attrs":{}},{"type":"text","text":"工具輸出的信息，發現了兩個令人在意的細節：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 增強後的類常量池區域的內容結構完全變了，增強前的類常量池一開始都只是些方法引用，字符串類型的常量index基本都在400、1200左右。而新的類常量池一開始全是類及字符串常量的index，方法引用、類引用夾雜在字符串常量之間。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. StackMapTable產生了大量的Entries，且有很多Entry是full frame。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/43/438f8f862bf77db9373355c35674051d.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"frame_type常見取值含義","attrs":{}},{"type":"text","text":":","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = SAME ;/ ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"0-63","attrs":{}},{"type":"text","text":" / 與上一個比較位置的局部變量表相同，且操作數棧爲空，這個值也是隱含的 offset_delta","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = SAME_LOCALS_1_STACK_ITEM; / ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"64-127","attrs":{}},{"type":"text","text":" / 當前幀與上一幀有相同的局部變量，操作數棧中的變量數目爲 1，隱式 offset_delta 爲 frame_type – 64","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = SAME_LOCALS_1_STACK_ITEM_EXTENDED; / ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"247","attrs":{}},{"type":"text","text":" /","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = CHOP / ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"248- 250","attrs":{}},{"type":"text","text":" /","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = SAME_FRAME_EXTENDED / ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"251","attrs":{}},{"type":"text","text":" / 局部變量信息和上一個幀相同，且操作數棧爲空","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = APPEND ; / ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"252-254","attrs":{}},{"type":"text","text":" / 當前幀比上一幀多了k個局部變量，且操作數棧爲空，其中 k = frame_type -251","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•frame_type = FULL_FRAME;/ ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"255","attrs":{}},{"type":"text","text":" / 局部變量表和操作數棧做完整記錄","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到StackMapTable的作用基本上是在字節碼驗證期間校驗字節碼合法性的，因此考慮先關閉JVM的字節碼校驗功能，看看排除了StackMapTable的影響後是否能夠減輕Metaspace空間上漲的症狀。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/92/92b778f7f0088c4044330af4abafc8ea.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到關閉字節碼校驗後，確實能夠緩解Metaspace空間上漲的問題，但是關閉JVM的字節碼校驗功能並不見得是一個安全的操作，這使得應用更容易受到非法字節碼的影響：不單單是增加了被惡意的字節碼攻擊應用的風險，而且在應用中爲了實現AOP，也引入了不少的動態生成字節碼的工具；缺乏字節碼校驗能力，同樣也會增加由於字節碼生成工具可能存在的問題而導致不合法的字節碼影響應用穩定的風險。因此，在沒有搞清楚問題根源就簡單地關閉掉字節碼校驗，是弊大於利，得不償失的。有必要進一步分析產生Metaspace OOM問題的原因。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"問題定位","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前爲止，雖然我們已經在字節碼層面上看到了異常的ConstantPool layout以及龐大的StackMapTable，但卻得不到更多的信息來發現問題了。因此只能考慮從JVM層面入手。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於筆者發現Metaspace OOM的問題在普通的JDK上也存在（在macOS上測試了OpenJDK 8及14，在Ubuntu 18上測試了OpenJDK 12，問題均存在），於是下載一份OpenJDK 14的源碼，打開slowdebug模式編譯了一份可進行調試的JDK。我們知道類加載過程中申請Metaspace空間最終會調用到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"share/memory/metaspace/spaceManager.cpp#SpaceManager::get_new_chunk","attrs":{}}],"attrs":{}},{"type":"text","text":"方法:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Metachunk* SpaceManager::get_new_chunk(size_t chunk_word_size) {// Get a chunk from the chunk freelist Metachunk* next = chunk_manager()->chunk_freelist_allocate(chunk_word_size);if (next == NULL) {next = vs_list()->get_new_chunk(chunk_word_size, medium_chunk_bunch()); } Log(gc, metaspace, alloc) log;if (log.is_trace() && next != NULL && SpaceManager::is_humongous(next->word_size())) {log.trace(\" new humongous chunk word size \" PTR_FORMAT, next->word_size()); }returnnext;}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此可以在方法頭部下條件斷點 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"chunk_word_size > 8192","attrs":{}}],"attrs":{}},{"type":"text","text":"，期望能從調用棧中看到消耗Metaspace的“罪魁禍首\"。","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個新產生的普通ClassLoader一開始會拿到4KB大小的chunks，直到申請次數達到一個上限（目前這個上限爲4），接下來Allocator就會”失去耐心“，每次都給這個ClassLoader分配64K大小的chunks。因爲是word_size，所以在筆者的x64 Mac上，一個word的size爲64，64 Kbytes = 65536 bytes = 8192 * 64 / 8，因此設成8192是恰到好處的。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很快，發現了申請大量Metaspace的調用棧：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ce/ce3d12dbe13d70918961a3501fd01c9a.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"逐級跟入調用棧，發現有兩個方法的註釋值得關注：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// We have entries mapped between the newand merged constant pools// so we have to rewrite some constant pool references.// 存在需要在新的及合併後的Constant Pool間映射的Entry，因此我們必須重寫一些Constant Pool的引用。if (!rewrite_cp_refs(scratch_class, THREAD)) {return JVMTI_ERROR_INTERNAL; }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// Rewrite constant pool references in the specific method. This code// was adapted from Rewriter::rewrite_method().void VM_RedefineClasses::rewrite_cp_refs_in_method(methodHandle method,methodHandle *new_method_p, TRAPS) {// ...// the new value needs ldc_w instead of ldc u_char inst_buffer[4]; // max instruction size is4 bytesbcp = (address)inst_buffer;// construct new instruction sequence *bcp = Bytecodes::_ldc_w; bcp++; Bytes::put_Java_u2(bcp, new_index);Relocator rc(method, NULL /* no RelocatorListener needed */);methodHandle m; {PauseNoSafepointVerifier pnsv(&nsv);// ldc is2 bytes and ldc_w is3 bytes// 執行到這一句進入空間分配 m = rc.insert_space_at(bci, 3, inst_buffer, CHECK); }// return the new method so that the caller can update// the containing class *new_method_p = method = m;//switch our bytecode processing loop from the old method// to the new method// ... } // end we need ldc_w instead of ldc } // end if there is a mapped index } break;// ...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個方法的主要作用是重寫指定方法的字節碼在常量池中的引用，從調試信息中可以看到，當前需要重寫的字節碼指令爲ldc, 在老常量池中ldc的常量池引用index爲2，而在新類中爲385，不滿足","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"new_index <= max_jubyte（255）","attrs":{}}],"attrs":{}},{"type":"text","text":"的條件，需要將","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ldc","attrs":{}}],"attrs":{}},{"type":"text","text":"指令擴展爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ldc_w","attrs":{}}],"attrs":{}},{"type":"text","text":"，因此插入新的字節碼指令","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/11/1119ac08afb6f5bedfaaffa92f78f1a6.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/31/31aaefa73236a64346600008f35024cf.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在插入字節碼指令的過程中，JDK會複製一遍當前方法的StackMapTable，","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/04c26b9a814e4ec046418a191936fee9.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個方法的StackMapTable很大，達到了900多KB，因此每擴展一次","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ldc","attrs":{}}],"attrs":{}},{"type":"text","text":"指令到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ldc_w","attrs":{}}],"attrs":{}},{"type":"text","text":"，差不多就需要向Metaspace申請約1MB的空間。老類中的ldc指令只有32個，而新類中的ldc指令多達1054個，再考慮到剛纔從","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"javap -verbose","attrs":{}}],"attrs":{}},{"type":"text","text":"結果中看到的，新類中Constant Pool layout與老類完全不同，這就意味着有很多的ldc指令因爲錯位而需要擴展，考慮到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"max_jubyte","attrs":{}}],"attrs":{}},{"type":"text","text":"的取值爲255，1054/2大約就是500個左右的ldc指令需要擴展。最終便導致了文章開頭的情景：Metaspace激增了約500MB。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a1/a122f21580087ac366604896d6e20456.jpeg","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到這裏，還剩下最後一個問題，爲什麼關掉JVM的字節碼校驗，就不會出現Metaspace激增呢？因爲關閉JVM的字節碼校驗後，ClassFileParser就不會去解析","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":".class","attrs":{}}],"attrs":{}},{"type":"text","text":"文件的StackMapTable部分，進而走不到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"if(m->has_stackmap_table())","attrs":{}}],"attrs":{}},{"type":"text","text":"語句，避免了StackMapTable的複製。這一點也可以從JVM源碼中得到佐證：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// src/hotspot/share/classfile/classFileParser.cpp # parse_stackmap_tablestatic const u1* parse_stackmap_table(const ClassFileStream* const cfs,u4 code_attribute_length,bool need_verify, TRAPS) {// ... // check code_attribute_length first cfs->skip_u1(code_attribute_length, CHECK_NULL);// 關注這一行if (!need_verify && !DumpSharedSpaces) {returnNULL; }return stackmap_table_start;}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果不需要verify且不需要DumpSharedSpaces，那麼parse_stackmap_table會直接返回NULL。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"繼續查看調用棧，整個棧是由","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"VM_RedefineClasses::load_new_class_versions","attrs":{}}],"attrs":{}},{"type":"text","text":"方法一路觸發調用的，","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"jvmtiError VM_RedefineClasses::load_new_class_versions(TRAPS) {// ...for (int i = 0; i < _class_count; i++) { // Create HandleMark so that any handles created while loading newclass// versions are deleted. Constant pools are deallocated while merging// constant poolsHandleMark hm(THREAD); InstanceKlass* the_class = get_ik(_class_defs[i].klass);Symbol* the_class_sym = the_class-&>name(); log_debug(redefine, class, load) (\"loading name=%s kind=%d (avail_mem=\" UINT64_FORMAT \"K)\", the_class->external_name(), _class_load_kind, os::available_memory() >> 10);// 構造了這個ClassFileStream對象↓ClassFileStreamst((u1*)_class_defs[i].class_bytes, _class_defs[i].class_byte_count,\"__VM_RedefineClasses__\", ClassFileStream::verify); // ...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法開頭構造了一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ClassFileStream","attrs":{}}],"attrs":{}},{"type":"text","text":"對象，這個對象的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"verify_stream","attrs":{}}],"attrs":{}},{"type":"text","text":"屬性被設置爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ClassFileStream::verify","attrs":{}}],"attrs":{}},{"type":"text","text":"，而這個值默認是爲true。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在ClassFileParser的構造函數中有設置_need_verify的代碼：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// Figure out whether we can skip format checking (matching classic VM behavior)if (DumpSharedSpaces) { // 沒有啓動參數，爲false // verify == true means it's a 'remote' class (i.e., non-boot class)// Verification decision is based on BytecodeVerificationRemote flag // for those classes. _need_verify = (stream->need_verify()) ? BytecodeVerificationRemote : BytecodeVerificationLocal; }else {// 走到這個分支 _need_verify = Verifier::should_verify_for(_loader_data->class_loader(), stream->need_verify()); }bool Verifier::should_verify_for(oop class_loader, bool should_verify_class) {return (class_loader == NULL || !should_verify_class) ?BytecodeVerificationLocal : BytecodeVerificationRemote;}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"class_loader !=null","attrs":{}}],"attrs":{}},{"type":"text","text":", ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"should_verify_class","attrs":{}}],"attrs":{}},{"type":"text","text":"爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"true","attrs":{}}],"attrs":{}},{"type":"text","text":"，於是走到了取值","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"BytecodeVerificationRemote","attrs":{}}],"attrs":{}},{"type":"text","text":"，而這個值正好就是由","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"-noverify","attrs":{}}],"attrs":{}},{"type":"text","text":"啓動參數決定的。只要在啓動參數中關閉JVM字節碼校驗，那麼","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"BytecodeVerificationRemote","attrs":{}}],"attrs":{}},{"type":"text","text":"就爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"false","attrs":{}}],"attrs":{}},{"type":"text","text":"，最終方法就不會攜帶StackMapTable信息，避免了StackMapTable的複製而導致佔用大量Metaspace空間。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，我們終於搞清楚了導致Metaspace OOM的根源：","attrs":{}},{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"在trace巨大方法時，Arthas產生新類的Constant Pool的Layout發生變化導致ldc指令需要rewrite，新的指令index超過max_jubyte後需要擴展ldc指令爲ldc_w指令，指令擴展過程中需要插入新的字節碼操作符，而插入新的字節碼操作符時又需要複製StackMapTable，而巨大的StackMapTable以及大量的ldc指令需要擴展，最終導致Metaspace空間暴增，引發問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"問題解決","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"既然知道了Metaspace OOM是由StackMapTable的複製引起的，而StackMapTable的複製又是在新舊Constant Pool index需要映射的情況下發生，那有沒有辦法儘可能的保持Constant Pool layout一致，避免這樣的重映射呢？閱讀了Arthas的源碼及其使用的字節碼增強庫bytebuddy的接口方法後，答案是肯定的。於是筆者開始嘗試修改Arthas的代碼，以便儘可能地保持新舊類的Constant Pool Layout一致。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"// com/alibaba/repackage-asm/0.0.7/com/alibaba/deps/org/objectweb/asm/ClassWriter.class","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參數 ClassReader: ClassReader實例用於讀取原始類文件，它將會被用於從原始類中複製完整的常量池、Bootstrap Method以及其他原始類中可複製部分的字節碼。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"com.taobao.arthas.core.advisor.Enhancer","attrs":{}}],"attrs":{}},{"type":"text","text":"類兩處，一處獲取","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ClassReader","attrs":{}}],"attrs":{}},{"type":"text","text":"實例的引用：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// src/main/java/com/taobao/arthas/core/advisor/Enhancer.java// ...if (matchingClasses != null && !matchingClasses.contains(classBeingRedefined)) {returnnull;}ClassNode classNode = new ClassNode(Opcodes.ASM8);// 在AsmUtils中新增方法，返回處理ClassNode的ClassReader。// 此時這個ClassReader中已經保存了原始類的Constant Pool等信息// 保持着這個ClassReader對象，在最後生成字節碼的時候有用ClassReader classReader = AsmUtils.toClassReader(classfileBuffer, classNode);// remove JSR https://github.com/alibaba/arthas/issues/1304classNode = AsmUtils.removeJSRInstructions(classNode);// 生成增強字節碼DefaultInterceptorClassParser defaultInterceptorClassParser = new DefaultInterceptorClassParser();// ...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一處將先前獲取到的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ClassReader","attrs":{}}],"attrs":{}},{"type":"text","text":"實例傳入字節碼生成方法中用於複製常量池","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"// src/main/java/com/taobao/arthas/core/advisor/Enhancer.java// ...// https://github.com/alibaba/arthas/issues/1223if (classNode.version