記一次真實的JVM性能調優過程

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近對負責的項目進行了一次性能優化，其中包括對 JVM 參數的調整，算是進行了一次簡單的 JVM 調優，JVM 參數調整之後，服務的整體性能有 5% 左右的提升，還算不錯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先介紹一下項目的基本情況：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"項目是一個高 QPS 壓力的 web 服務，單機 QPS 一直維持在 1.5K 以上，由於舊機器的”拖累”，配置的堆大小是 8G，其中 young 區是 4G，垃圾回收器用的是 parNew + CMS。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"舊狀","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是查看當前 GC 的情況，主要是使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"jstat","attrs":{}}],"attrs":{}},{"type":"text","text":" 查看 GC 的概況，再查看 gc log，分析單次 gc 的詳細狀況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"jstat -gcutil pid 1000","attrs":{}}],"attrs":{}},{"type":"text","text":" 每隔一秒打印一次 gc 統計信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/482f59488d6ff8b4bf897a71d2867c01.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，單次 gc 平均耗時是 60ms 左右，還算可以接受，但 YGC 非常頻繁，基本上每秒一次，有的時候還會一秒兩次，在一秒兩次的時候，服務對業務響應時長的壓力就會變得很大。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接着查看 gc log，打印 gc log 需要在 JVM 啓動參數裏添加以下參數：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-XX:+PrintGCDateStamps","attrs":{}}],"attrs":{}},{"type":"text","text":"：打印 gc 發生的時間戳。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-XX:+PrintTenuringDistribution","attrs":{}}],"attrs":{}},{"type":"text","text":"：打印 gc 發生時的分代信息。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-XX:+PrintGCApplicationStoppedTime","attrs":{}}],"attrs":{}},{"type":"text","text":"：打印 gc 停頓時長","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-XX:+PrintGCApplicationConcurrentTime","attrs":{}}],"attrs":{}},{"type":"text","text":"：打印 gc 間隔的服務運行時長","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-XX:+PrintGCDetails","attrs":{}}],"attrs":{}},{"type":"text","text":"：打印 gc 詳情，包括 gc 前/內存等。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"-Xloggc:../gclogs/gc.log.date","attrs":{}}],"attrs":{}},{"type":"text","text":"：指定 gc log 的路徑","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看到的 gc log 形如：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a2/a2e58d625d5ead0c18f3e9c6596b0e0e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單次 GC 方面並不能直接看出問題，但可以看到 gc 前有很多次 18ms 左右的停頓。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分析和調整","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"YGC 頻繁","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直接查看 gc log 並不直觀，我們可以借用一些可視化工具來幫助我們分析， ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"[gceasy](https://gceasy.io/)","attrs":{}}],"attrs":{}},{"type":"text","text":" 是個挺不錯的網站，我們把 gc log 上傳上去後， gceasy 可以幫助我們生成各個維度的圖表幫助分析。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看 gceasy 生成的報告，發現我們服務的 gc 吞吐量是 95%，它指的是 JVM 運行業務代碼的時長佔 JVM 總運行時長的比例，這個比例確實有些低了，運行 100 分鐘就有 5 分鐘在執行 gc。幸好這些 GC 中絕大多數都是 YGC，單次時長可控且分佈平均，這使得我們服務還能平穩運行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決這個問題要麼是減少對象的創建，要麼就增大 young 區。前者不是一時半會兒都解決的，需要查找代碼裏可能有問題的點，分步優化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而後者雖然改一下配置就行，但以我們對 GC 最直觀的印象來說，增大 young 區，YGC 的時長也會迅速增大。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實這點不必太過擔心，我們知道 YGC 的耗時是由 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"GC 標記 + GC 複製","attrs":{}}],"attrs":{}},{"type":"text","text":" 組成的，相對於 GC 複製，GC 標記是非常快的。而 young 區內大多數對象的生命週期都非常短，如果將 young 區增大一倍，GC 標記的時長會提升一倍，但到 GC 發生時被標記的對象大部分已經死亡， GC 複製的時長肯定不會提升一倍，所以我們可以放心增大 young 區大小。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於低內存舊機器都被換掉了，我把堆大小調整到了 12G，young 區保留爲 8G。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"分代調整","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了 GC 太頻繁之外，GC 後各分代的平均大小也需要調整。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/04/04d1975979779f267253a6f1e161c0d9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們知道 GC 的提升機制，每次 GC 後，JVM 存活代數大於 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"MaxTenuringThreshold","attrs":{}}],"attrs":{}},{"type":"text","text":" 的對象提升到老年代。當然，JVM 還有動態年齡計算的規則：按照年齡從小到大對其所佔用的大小進行累積，當累積的某個年齡大小超過了 survivor 區的一半時，取這個年齡和 MaxTenuringThreshold 中更小的一個值，作爲新的晉升年齡閾值，但看各代總的內存大小，是達不到 survivor 區的一半的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/84/8439f6fbff6f7b9a9a66ab82c58af9fd.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以這十五個分代內的對象會一直在兩個 survivor 區之間來回複製，再觀察各分代的平均大小，可以看到，四代以上的對象已經有一半都會保留到老年區了，所以可以將這些對象直接提升到老年代，以減少對象在兩個 survivor 區之間複製的性能開銷。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我把 MaxTenuringThreshold 的值調整爲 4，將存活超過四代的對象直接提升到老年代。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"偏向鎖停頓","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有一個問題是 gc log 裏有很多 18ms 左右的停頓，有時候連續有十多條，雖然每次停頓時長不長，但連續多次累積的時間也非常可觀。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個問題在我之前的文章有講過，這裏就不贅述了。1.8 之後 JVM 對鎖進行了優化，添加了偏向鎖的概念，避免了很多不必要的加鎖操作，但偏向鎖一旦遇到鎖競爭，取消鎖需要進入 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"safe point","attrs":{}}],"attrs":{}},{"type":"text","text":"，導致 STW。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決方式很簡單，JVM 啓動參數裏添加 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"-XX:-UseBiasedLocking","attrs":{}}],"attrs":{}},{"type":"text","text":" 即可。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"結果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調整完 JVM 參數後先是對服務進行壓測，發現性能確實有提升，也沒有發生嚴重的 GC 問題，之後再把調整好的配置放到線上機器進行灰度，同時收集 gc log，再次進行分析。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於 young 區大小翻倍了，所以 YGC 的頻率減半了，GC 的吞量提升到了 97.75%。 平均 GC 時長略有上升，從 60ms 左右提升到了 66ms，還是挺符合預期的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於 CMS 在進行 GC 時也會清理 young 區，CMS 的時長也受到了影響，CMS 的最終標記和併發清理階段耗時增加了，也比較正常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外我還統計了對業務的影響，之前因爲 GC 導致超時的請求大大減少了。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"小結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總之，這是一次挺成功的 GC 調整，讓我對 GC 有了更深的理解，但由於沒有深入到 old 區，之前學習到的 CMS 相關的知識還沒有複習到。不過性能優化並不是一朝一夕的事，需要時刻關注問題，及時做出調整。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我把自己的調優經驗整理成了一本PDF，也收集了一些關於調優的書籍的PDF，都分享給大夥，需要的同學可以點擊","attrs":{}},{"type":"link","attrs":{"href":"https://u.wechat.com/MMP-94sa6p25rlWtZmr06ys","title":"","type":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"JVM調優筆記與學習書籍","attrs":{}}]},{"type":"text","text":"領取。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於本文有什麼疑問可以在下面留言交流，如果您覺得本文對您有幫助，歡迎關注。","attrs":{}}]}]}