導致該情況中止原因爲垃圾對象沒有達到閥值 5%，不啓動Mixed。 現有JVM 參數 GC日誌截圖分析 結論

報警名稱: intelligent-rec Exceptions_OutOfMemoryError_Error

現有JVM 參數

`-Xmx20G 
-Xms20G `
-XX:MaxDirectMemorySize=5G 
-XX:+UnlockExperimentalVMOptions 
-XX:G1NewSizePercent=5 
-XX:G1HeapRegionSize=16M 
-XX:G1RSetUpdatingPauseTimePercent=1 
-XX:+ParallelRefProcEnabled 
-XX:MetaspaceSize=1G 
-XX:+UseG1GC
 -XX:MaxGCPauseMillis=100 
-XX:GCPauseIntervalMillis=300 
-XX:G1MixedGCCountTarget=16 
-XX:StringTableSize=4000000 
-XX:+PrintStringTableStatistics 
-XX:+PrintGCDetails 
-XX:+PrintGCDateStamps 
-Xloggc:/data/logs/intelligent-rec-gc-%p-%t.log 
-XX:+PrintGCApplicationConcurrentTime 
-XX:+PrintGCApplicationStoppedTime 
-XX:+HeapDumpOnOutOfMemoryError 
-Djava.net.preferIPv4Stack=true 
`-XX:G1MaxNewSizePercent=50`
`-XX:InitiatingHeapOccupancyPercent=30`
-XX:G1ReservePercent=15 
-XX:+UnlockDiagnosticVMOptions  
-XX:+PrintHeapAtGC

新生代比例有兩個數值指定,下限:-XX:G1NewSizePercent,默認值5%,上限:-XX:G1MaxNewSizePercent 默認值60% 相應老年代最大40%

GC日誌截圖分析

JVM內存分佈

每代分配的大小 allocated

Indicates the allocated size for each generation. This data point is gathered from the GC log, 
thus It may or may not match with the size that is specified by the JVM system properties (i.e. –Xmx, -Xms,…). 
Say you have configured total heap size (i.e. –Xmx) as 2gb, whereas at runtime if JVM has allocated only 1gb, 
then in this report you are going to see the allocated size as 1gb only.

每代分配的大小。這個數據點是從GC日誌中收集的，
因此，它可能與JVM系統屬性(例如-Xmx， -Xms，…)指定的大小匹配，也可能不匹配。
假設您已經將總堆大小(即-Xmx)配置爲2gb，而在運行時，如果JVM只分配了1gb，
然後在這個報告中，您將看到分配的大小僅爲1gb。

每一代的峯值內存利用率 peak,通常它不會超過分配的大小

Peak memory utilization of each generation. Typically it won't exceed the `allocated` size.
 However in few cases, we have seen peak utilization go beyond allocated size as well, especially in G1 GC.

每一代的峯值內存利用率。通常它不會超過分配的大小。然而，在少數情況下，峯值利用率也超過了分配的大小，特別是在G1 GC中。

-XX:G1NewSizePercent=5 -XX:G1MaxNewSizePercent=50 -Xmx20G(年輕代最小爲20*0.05=1，最大爲 20*0.5=10；老年代：20-10=10，20-1=19)

老年代內存徒增

https://zhuanlan.zhihu.com/p/79643783

老區不夠了，這個時候會把young區所有對象不管死活都轉成old區對象，所以總的內存使用量會暴增?

結論存疑，但年輕代晉升級老年代，當survivor空間不足時，會直接進入老年代，可以確認。

兩種可能

1. 可達對象(可能性較大)
2. 不可達對象

Mixed gc停止

12:15 之後mix GC終止 說明垃圾不多 （嚴重）

GC日誌出現 to-space exhausted 或者 to-space overflow 馬上FullGC

FULL gc

Full GC (Allocation Failure

2021-05-23T12:24:47.616+0800: 403254.508: Total time for which application threads were stopped: 0.0385721 seconds, Stopping threads took: 0.0002912 seconds
2021-05-23T12:24:47.616+0800: 403254.508: Application time: 0.0001031 seconds
{Heap before GC invocations=29097 (full 0):
 garbage-first heap   total 20971520K, used 20938752K [0x00000002c0000000, 0x00000002c1002800, 0x00000007c0000000)
  region size 16384K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 115382K, capacity 122379K, committed 123648K, reserved 1157120K
  class space    used 13564K, capacity 14845K, committed 15104K, reserved 1048576K
2021-05-23T12:24:47.625+0800: 403254.517: [Full GC (Allocation Failure)  19G->15G(20G), 38.9985388 secs]
   [Eden: 0.0B(1024.0M)->0.0B(1024.0M) Survivors: 0.0B->0.0B Heap: 20.0G(20.0G)->15.2G(20.0G)], [Metaspace: 115382K->115370K(1157120K)]
Heap after GC invocations=29098 (full 1):
 garbage-first heap   total 20971520K, used 15899924K [0x00000002c0000000, 0x00000002c1002800, 0x00000007c0000000)
  region size 16384K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 115370K, capacity 122361K, committed 123648K, reserved 1157120K
  class space    used 13563K, capacity 14842K, committed 15104K, reserved 1048576K
}
 [Times: user=59.83 sys=0.37, real=39.00 secs]

YGC

105.0711085 secs

{Heap before GC invocations=29071 (full 0):
 garbage-first heap   total 20971520K, used 13940955K [0x00000002c0000000, 0x00000002c1002800, 0x00000007c0000000)
  region size 16384K, 640 young (10485760K), 12 survivors (196608K)
 Metaspace       used 115257K, capacity 122315K, committed 123648K, reserved 1157120K
  class space    used 13556K, capacity 14845K, committed 15104K, reserved 1048576K
2021-05-23T12:21:02.040+0800: 403028.932: [GC pause (G1 Evacuation Pause) (young), 105.0711085 secs]
   [Parallel Time: 105051.7 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 403028933.0, Avg: 403028933.3, Max: 403028933.6, Diff: 0.6]
      [Ext Root Scanning (ms): Min: 12.4, Avg: 12.7, Max: 13.0, Diff: 0.6, Sum: 165.2]
      [Update RS (ms): Min: 5.1, Avg: 5.1, Max: 5.5, Diff: 0.4, Sum: 66.8]
         [Processed Buffers: Min: 22, Avg: 51.3, Max: 89, Diff: 67, Sum: 667]
      [Scan RS (ms): Min: 26.6, Avg: 26.9, Max: 27.0, Diff: 0.4, Sum: 349.6]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 95548.6, Avg: 96411.4, Max: 97892.4, Diff: 2343.8, Sum: 1253348.3]
      [Termination (ms): Min: 7113.7, Avg: 8594.7, Max: 9457.6, Diff: 2343.9, Sum: 111731.2]
         [Termination Attempts: Min: 1443879, Avg: 1480471.4, Max: 1575977, Diff: 132098, Sum: 19246128]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.8]
      [GC Worker Total (ms): Min: 105050.5, Avg: 105051.0, Max: 105051.3, Diff: 0.8, Sum: 1365663.0]
      [GC Worker End (ms): Min: 403133984.2, Avg: 403133984.3, Max: 403133984.4, Diff: 0.3]
   [Code Root Fixup: 0.4 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 1.2 ms]
   [Other: 17.8 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 4.7 ms]
      [Ref Enq: 0.3 ms]
      [Redirty Cards: 8.3 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.8 ms]
   [Eden: 10048.0M(10048.0M)->0.0B(352.0M) Survivors: 192.0M->1280.0M Heap: 13.3G(20.0G)->7754.5M(20.0G)]
Heap after GC invocations=29072 (full 0):
 garbage-first heap   total 20971520K, used 7940573K [0x00000002c0000000, 0x00000002c1002800, 0x00000007c0000000)
  region size 16384K, 80 young (1310720K), 80 survivors (1310720K)
 Metaspace       used 115257K, capacity 122315K, committed 123648K, reserved 1157120K
  class space    used 13556K, capacity 14845K, committed 15104K, reserved 1048576K
}
 [Times: user=1364.20 sys=1.26, real=105.07 secs]

-XX:ParallelGCThreads 加大線程數

並行GC線程數：通過 -XX:ParallelGCThreads來指定，也就是在STW階段工作的GC線程數，其值遵循以下原則：

• 如果用戶顯示指定了ParallelGCThreads，則使用用戶指定的值。
• 否則，需要根據實際的CPU所能夠支持的線程數來計算ParallelGCThreads的值，計算方法見步驟③和步驟④。
• 如果物理CPU所能夠支持線程數小於8，則ParallelGCThreads的值爲CPU所支持的線程數。這裏的閥值爲8，是因爲JVM中調用nof_parallel_worker_threads接口所傳入的switch_pt的值均爲8。
• 如果物理CPU所能夠支持線程數大於8，則ParallelGCThreads的值爲8加上一個調整值，調整值的計算方式爲：物理CPU所支持的線程數減去8所得值的5/8或者5/16，JVM會根據實際的情況來選擇具體是乘以5/8還是5/16。

比如，在64線程的x86 CPU上，如果用戶未指定ParallelGCThreads的值，則默認的計算方式爲：ParallelGCThreads = 8 + (64 - 8) * (5/8) = 8 + 35 = 43。

結論

1. to-space exhausted 或者 to-space overflow 可能有大量對象無法晉升，老年代擔保，直接進入老年代
2. 在12:15 之後mix GC終止,導致該情況中止原因爲垃圾對象沒有達到閥值 5%，不啓動Mixed。如果老年代爲可達對象，情況會比較嚴重! 需要dump 內存進一步分析。

觸發Mixed GC的堆垃圾佔比：通過`-XX:G1HeapWastePercent`指定，默認值5%，也就是在全局標記結束後能夠統計出所有Cset內可被回收的垃圾佔整對的比例值，如果超過5%，那麼就會觸發之後的多輪Mixed GC，如果不超過，那麼會在之後的某次Young GC中重新執行全局併發標記。

3. 年輕代相對較大，存活對象較多時，複製壓力較大。

建議解決方案

修改JVM參數

-XX:HeapDumpPath=/data/logs/gc%p-%t.hprof
-XX:SurvivorRatio=6
-XX:ParallelGCThreads=64
-XX:G1MaxNewSizePercent=30

-XX:ParallelGCThreads=64 該參數影響特別大

-XX:G1NewSizePercent=20（默認5%）

設置年輕代佔用堆最小百分比爲20%來解決這個問題。因爲有了這個設置，即使引用處理耗時變長，Eden區大小也不可能比這個閾值更低，從而避免對象提早晉升。同時，我們還添加了參數-XX:+ParallelRefProcEnabled，從而在Remark階段多線程併發處理引用對象。

減少緩存多餘字段