頻繁操作本地緩存導致YGC耗時過長

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"某天，某位羣友在JVM討論羣裏發來一張GC log的圖片。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中主要的問題是YGC過長，每次耗時約爲200ms。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cb/cbf95fe264bb7a20233f2240a455a33a.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用的JVM參數如下："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"-Xmn2048m -Xms4096m -Xmx4096m -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128M\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"指定年輕代內存爲2g，初始JVM內存爲4g，最大JVM內存爲4g。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個問題引起了羣友們的關注。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從GC log和JVM參數可以看出，GC算法使用默認的Parallel Scanvenge。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到Eden區大小爲1536M，兩個Survivor區大小爲256M。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得出"},{"type":"codeinline","content":[{"type":"text","text":"-XX:SurvivorRatio = 6"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外可以看到在GC時，"},{"type":"codeinline","content":[{"type":"text","text":"desired survivor size 268435456 bytes = 256M"}]},{"type":"text","text":"，得出"},{"type":"codeinline","content":[{"type":"text","text":"-XX:TargetSurvivorRatio = 100"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"默認情況下，"},{"type":"codeinline","content":[{"type":"text","text":"-XX:SurvivorRatio = 8，-XX:SurvivorRatio = 50"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而並未設置這兩個參數，一直懷疑是配置沒有生效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一時沒有想到辦法，有羣友建議試着調整下 MaxGCPauseMills 或者 GCTimeRatio 參數，然後效果都不好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之後的某天，嘗試使用jmap -heap pid打印應用的堆棧信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/46/460dd5605c4333706c27f792d4df75e0.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發現雖然寫着SurvivorRatio = 8，但是E:S0:S1的比例並非是8:1:1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是開始尋找原因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"找到來自R大的回答："}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HotSpot VM裏，ParallelScavenge系的GC（UseParallelGC / UseParallelOldGC）默認行爲是SurvivorRatio如果不顯式設置就沒啥用。顯式設置到跟默認值一樣的值則會有效果。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲ParallelScavenge系的GC最初設計就是默認打開AdaptiveSizePolicy的，它會自動、自適應的調整各種參數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是推薦羣友嘗試使用CMS，讓這些配置固定下來，不做自適應調整。但設置之後，發現YGC效果依舊不好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d33c494cac97bedc5152ab7e67893721.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯示發生4次YGC，耗時1.145s，平均耗時約286ms，情況反而更糟，回頭再次分析GC log。發現日誌中有這麼一行："},{"type":"codeinline","content":[{"type":"text","text":"new threshold 7 （max 15）"}]},{"type":"text","text":"JVM中有個參數爲晉升年齡閾值（"},{"type":"codeinline","content":[{"type":"text","text":"-XX:MaxTenuringThreshold"}]},{"type":"text","text":"），默認值爲15。意思爲在YGC時，超過該年齡的對象會被晉升到老年代。但GC log中顯示該閾值被修改成了7。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在年輕代對象晉升時，有一個判斷條件如下："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"動態年齡判斷，大於等於某個年齡的對象超過了survivor空間一半，大於等於某個年齡的對象直接進入老年代。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得出，在某次YGC時，Survivor區中，年齡超過7的對象佔用了Survivor空間一半以上。而正常情況下，年輕代對象朝生夕死。網絡服務處理請求爲毫秒級，YGC幾秒甚至十幾秒才發生一次,多數年輕對象活不過1代。於是，猜測該羣友使用了本地緩存。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在得到肯定的回覆後，詳細詢問了羣友使用本地緩存的方法。自行實現了一個本地緩存，類似於HashMap。別的服務會每一分鐘推送緩存數據用於同步。在同步的時候不做diff操作，直接put。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"緩存中保存Person類"},{"type":"text","text":"。"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1@Data\n2class Person {\n3\n4 private String name;\n5\n6 private Integer age;\n7}\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存內容可能爲："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1{\n2 \"jjs\":{\n3 \"age\":27,\n4 \"name\":\"jjs\"\n5 }\n6}\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"緩存同步涉及兩種操作：新增和覆蓋。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"兩種操作均直接使用put操作實現，無論當前緩存key是否已經存在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣的操作方法在業務上完全沒有問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但對於GC而言，每次緩存同步需要new很多新的對象，並且這些對象都將一直存活，直到被覆蓋，或者晉升到老年代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些緩存對象首先會被分配到年輕代，在YGC時候，這些對象都會被標記爲存活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得到YGC耗時過長原因一："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"年輕代中有太多存活的對象，增加了標記時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，HashMap是數組加鏈表的結構，使用Node結構用於保存key、value。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HashMap的Node結構如下："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1static class Node implements Map.Entry {\n2 final int hash;\n3 final K key;\n4 V value;\n5 Node next;\n6}\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次put生成的node節點，很可能(hash衝突)被掛在已有node節點的next域上。已有node爲緩存，長期存活，很有可能位於老年代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼，就形成了老年代對象對年輕代對象的引用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在YGC中，需要掃描Card Table中的dirty區域來識別被老年代對象引用的年輕代對象。正常情況下，這種情形並不多。但在本文例子中，會大量存在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得到YGC耗時過長原因二："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"YGC又需要花費大量的時間在掃描Card Table上,總結原因是操作本地緩存太頻繁導致了YGC耗時過長。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回顧YGC的大致過程："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/63b888c87a7ac471c96ef689dc106836.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從根節點開始掃描年輕代對象，直到掃描到下個引用爲非年輕代對象。(可以避免YGC掃描整個堆。)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"掃描老年代dirty區域，即可掃描到被老年代對象引用的年輕代對象。(老年代被分爲不同的塊，Card Table字節數組中每個字節表示老年代中的一塊。新分配對象時，觸發寫屏障，存在有老年代對象引用年輕代對象時，將對應的卡表設置爲dirty。)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將Eden和From區中的對象複製到To區。如果To區已滿，則直接複製到老年代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"YGC耗時過長問題的排查還是應該從兩個點出發："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"YGC時存活的年輕代對象太多。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"老年代對象引用年輕代對象的情況太多。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改代碼需要一定的時間，羣友採用了一種短期的辦法。"},{"type":"text","marks":[{"type":"strong"}],"text":"修改了推送的週期。原來每一分鐘推送一次"},{"type":"text","text":"。YGC下降到18-25ms，但在緩存推送時，YGC時間仍然達到200ms。兩次緩存推送之間的對象都符合朝生夕死的弱分代假設，YGC時間正常。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"後續修改思考/建議："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分批推送緩存，並且在接到推送的緩存時做diff操作，儘量修改已有對象而非新建對象。此舉可減少長壽對象生成。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即使使用分批推送，在應用啓動時，還是需要全量加載緩存。仍舊會面臨應用剛啓動時，YGC耗時過長的問題。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"重新規劃應用。因爲經常變化的數據並不適合放在緩存中。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用Redis緩存。Redis的響應時間爲毫秒級，甚至只需幾毫秒，並且無需考慮分佈式下緩存同步問題。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用CMS垃圾回收算法。因爲默認和Parallel Scanvenge算法配合的老年代回收算法是Serial Old。該算法需要標記清理壓縮，STW時間較長。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"看完三件事❤️"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你覺得這篇內容對你還蠻有幫助，我想邀請你幫我三個小忙："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"點贊，轉發，有你們的 『點贊和評論』，纔是我創造的動力。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"關注公衆號 『 "},{"type":"text","marks":[{"type":"strong"}],"text":"java爛豬皮"},{"type":"text","text":" 』，不定期分享原創知識。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"同時可以期待後續文章ing🚀"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/8518f1f13ab07e46122c84420bbf39a8.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者：阿菜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"出處：https://club.perfma.com/article/1578279"}]}]}]}