開篇

要想排查jvm的線上問題，必先熟悉常用命令。
主要有下列這些：
jvm相關的命令：jinfo jmap jstat jstack jps
linux系統相關命令:top ps print diff free netstat wc
文檔查看相關命令:less more vi vim
高級命令:sed awk grep egrep sort head

熟悉上面這些命令才能在生產環境排查問題時遊刃有餘，當然還需要經驗的積累，才能變成找問題工程師。

linux相關

查看系統運行的最大句柄數 open files

[root@4mfb6 root]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256766
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655350
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192

查看當前系統運行的句柄數
1是pid

[root@4mfb6 root]# ll /proc/1/fd | wc -l
335

查看當前系統運行的線程數

[root@4mfb6 root]# ll /proc/1/task/ | wc -l
630```

# JVM相關命令
## jps
查看當前系統的java進程id和進程名
> jps
```java
[root@localhost ~]$ jps
21381 BrokerStartup
24696 Jps
20668 NamesrvStartup

jinfo

查看jvm的所有運行時參數（相當於 jinfo -sysprops -flags pid）

jinfo pid;

查看某個參數的值

jinfo -flag name pid

[root@localhost ~]$ jinfo -flag MaxMetaspaceSize 17246
-XX:MaxMetaspaceSize=335544320

生效或者失效某個參數

jinfo -flag [+|-] name pid

[root@localhost ~]$ jinfo -flag -PrintGCTimeStamps 17246
[root@localhost ~]$ jinfo -flag +PrintGCTimeStamps 17246

運行時設置jvm的參數值（我試了幾個參數都不能被修改，放棄）

jinfo -flag MaxMetaspaceSize =1000000000 pid

[root@localhost ~]$ jinfo -flag MaxMetaspaceSize=1000000000 17246
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'MaxMetaspaceSize' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
        
[root@localhost ~]$ jinfo -flag MaxHeapSize=5000000000 17246
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'MaxHeapSize' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
       
[root@localhost ~]$ jinfo -flag NumberOfGCLogFiles=50 17246
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'NumberOfGCLogFiles' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)

只查看jvm相關的參數

jinfo -flags pid

[root@localhost ~]$ jinfo -flags 17246
Attaching to process ID 17246, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
Non-default VM flags: -XX:CICompilerCount=2 -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:GCLogFileSize=31457280 -XX:InitialHeapSize=4294967296 -XX:MaxHeapSize=4294967296 -XX:MaxMetaspaceSize=335544320 -XX:MaxNewSize=2147483648 -XX:MetaspaceSize=134217728 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=2147483648 -XX:NumberOfGCLogFiles=5 -XX:OldPLABSize=16 -XX:OldSize=2147483648 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8 -XX:+UseCMSCompactAtFullCollection -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseFastUnorderedTimeStamps -XX:+UseGCLogFileRotation -XX:-UseLargePages -XX:-UseParNewGC
Command line:  -Xms4g -Xmx4g -Xmn2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:SurvivorRatio=8 -XX:-UseParNewGC -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc_%p_%t.log -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m -XX:-OmitStackTraceInFastThrow -XX:-UseLargePages -Djava.ext.dirs=/home/root/jdk1.8.0_112/jre/lib/ext:/home/root/kafka_2.11-1.0.0/bin/rocketmq-all-4.7.0-bin-release/bin/../lib

查看系統的參數

jinfo -sysprops pid

[root@localhost ~]$ jinfo -sysprops 17246
Attaching to process ID 17246, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
java.runtime.name = Java(TM) SE Runtime Environment
java.vm.version = 25.112-b15
sun.boot.library.path = /home/root/jdk1.8.0_112/jre/lib/amd64
rocketmq.remoting.version = 353
java.vendor.url = http://java.oracle.com/
java.vm.vendor = Oracle Corporation
path.separator = :
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
sun.os.patch.level = unknown
...

jstack

查看當前jvm主進程的所有線程。

jstack pid

[root@localhost ~]$ jstack 17246
2020-05-30 23:11:59
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):

"Attach Listener" #28 daemon prio=9 os_prio=0 tid=0x00007f6930001000 nid=0x462f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

Attach Listener：線程名
#28：看着像是線程的編號，第一個啓動的是1
daemon：表示是jvm的守護進程，當主進程關閉，jvm會整個關閉。
prio=9：是java的線程優先級，thread類的一個屬性
os_prio=0：是操作系統的線程優先級
tid=0x00007f6930001000：是java線程的id，thread類的一個屬性。
nid=0x462f：是操作系統級別的線程id，16進制。（linux下16進制和10進制互轉命令,print %d 0x462f; print %x 17246）
waiting on condition [0x0000000000000000]：具體是等待在哪裏
java.lang.Thread.State: RUNNABLE：線程的狀態。全部狀態在Thread的State類（NEW、RUNNING、BLOCKED、WAITING、TIME_WAITING、TERMINATED）中。

查看當前jvm主進程的所有線程。附加打印本地方法棧

jstack -m pid

----------------- 17967 -----------------
0x0000003d02c0eb2d      __libc_accept + 0x2d
0x00007f696cbf8bae      _ZN14AttachListener7dequeueEv + 0x9e
0x00007f696cbf7aba      _ZL28attach_listener_thread_entryP10JavaThreadP6Thread + 0x19a
0x00007f696d3505f3      _ZN10JavaThread17thread_main_innerEv + 0x103
0x00007f696d35073c      _ZN10JavaThread3runEv + 0x11c
0x00007f696d201138      _ZL10java_startP6Thread + 0x108

查看當前jvm主進程的所有線程。附加打印具體鎖信息

jstack -l pid

"Attach Listener" #28 daemon prio=9 os_prio=0 tid=0x00007f6930001000 nid=0x462f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

jstat

可以看到jstat的選項較多。但是大的是三塊gc、class、compiler

[root@localhost ~]$ jstat -options
-class
-compiler
-gc
-gccapacity
-gccause
-gcmetacapacity
-gcnew
-gcnewcapacity
-gcold
-gcoldcapacity
-gcutil
-printcompilation

查看jvm加載的class信息

jstat -class pid

[root@localhost ~]$ jstat -class 17246
Loaded  Bytes  Unloaded  Bytes     Time
  2683  5201.9        0     0.0       1.21

Loaded ：加載class的個數
Bytes：加載class的字節數
Unloaded ：卸載class的個數
Bytes：卸載class的字節數
Time：加載和卸載的總耗時

查看java類編譯器編譯的信息（感覺作用不大，除非可以知道JIT的編譯情況）

jstat -compiler pid

[root@localhost ~]$ jstat -compiler 17246
Compiled Failed Invalid   Time   FailedType FailedMethod
    1271      0       0     1.73          0

查看gc的情況，堆內存以字節的形式展示

jstat -gc pid [interval] [count]

查看gc的情況，堆內存以百分比的形式展示

jstat -gcutil pid [interval] [count]

查看gc的原因

jstat -gccause pid [interval] [count]

sh-4.2# jstat -gccause 1 1000 50                                                                                                                                                                                 
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT    LGCC                 GCC                                                                                                                                                                                                              
 60.94   0.00  99.09  49.62  95.21  92.84    706   74.757     0    0.000   74.757 Allocation Failure   No GC                                                                                                       
 60.94   0.00  99.98  49.62  95.21  92.84    706   74.757     0    0.000   74.757 Allocation Failure   No GC                                                                                                       
  0.00  67.19   1.84  49.64  95.21  92.84    707   74.866     0    0.000   74.866 Allocation Failure   No GC                                                                                                       
  0.00  67.19   1.90  49.64  95.21  92.84    707   74.866     0    0.000   74.866 Allocation Failure   No GC

jmap

一共有-heap、-histo、-dump、 -clsstats、finalizerinfo幾種選項，而且前四個都比較重要

類似linux的pmap命令

jmap pid

[root@localhost ~]$ jmap 21381
Attaching to process ID 21381, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
0x0000000000400000      7K      /home/root/jdk1.8.0_112/bin/java
0x0000003d02400000      157K    /lib64/ld-2.12.so
0x0000003d02800000      1885K   /lib64/libc-2.12.so
0x0000003d02c00000      143K    /lib64/libpthread-2.12.so

打印jvm堆的詳細信息，包括各分代的初始，已使用，空閒的內存大小。以及關於堆的jvm參數值

jmap -heap pid

[root@localhost ~]$ jmap -heap 21381
Attaching to process ID 21381, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15

using thread-local object allocation.
Garbage-First (G1) GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 1073741824 (1024.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 637534208 (608.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 16777216 (16.0MB)

Heap Usage:
G1 Heap:
   regions  = 64
   capacity = 1073741824 (1024.0MB)
   used     = 27190200 (25.93059539794922MB)
   free     = 1046551624 (998.0694046020508MB)
   2.5322847068309784% used
G1 Young Generation:
Eden Space:
   regions  = 0
   capacity = 50331648 (48.0MB)
   used     = 0 (0.0MB)
   free     = 50331648 (48.0MB)
   0.0% used
Survivor Space:
   regions  = 1
   capacity = 16777216 (16.0MB)
   used     = 16777216 (16.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 2
   capacity = 1006632960 (960.0MB)
   used     = 10412984 (9.930595397949219MB)
   free     = 996219976 (950.0694046020508MB)
   1.0344370206197102% used

7807 interned Strings occupying 701616 bytes.

查看當前（存活）堆中對象的分佈情況。常常用來查找內存泄漏問題

jmap -histo[:live] pid

[root@localhost ~]$ jmap -histo 21381 | head -n 15

 num     #instances         #bytes  class name
----------------------------------------------
   1:         16152       21883608  [B
   2:          2904        8052288  [I
   3:         79506        4452336  org.apache.rocketmq.store.schedule.ScheduleMessageService$DeliverDelayedMessageTimerTask
   4:         38040        3697960  [C
   5:         93669        2997408  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
   6:         82557        1320912  java.lang.Object
   7:         31793         763032  java.lang.String
   8:          3293         377312  java.lang.Class
   9:          5238         339224  [Ljava.lang.Object;
  10:          4239         229944  [Ljava.lang.String;
  11:          4366         209568  java.util.HashMap
  12:          6418         154032  java.util.concurrent.CopyOnWriteArrayList$COWIterator

num:編號，默認字節數從大到小排序
#instances：對象數
#bytes：類實例化對象所佔的內存大小
class name:類名稱

查看類加載的情況

jmap -clstats pid

[root@localhost ~]$ jmap -clstats 21381
class_loader    classes bytes   parent_loader   alive?  type
<bootstrap>     1793    3250520   null          live    <internal>
0x00000000fc000000      1       880     0x00000000c02779f8      dead    sun/reflect/DelegatingClassLoader@0x0000000100009df8
0x00000000fc000380      1       1471      null          dead    sun/reflect/DelegatingClassLoader@0x0000000100009df8

查看當前的finalize隊列中的個數

jmap -finalizerinfo pid

[root@localhost ~]$ jmap -finalizerinfo 21381
Attaching to process ID 21381, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
Number of objects pending for finalization: 0

dump當前jvm的內存堆。dump下來的文件可以使用MemoryAnalyzer分析。也可以使用jvisualvm來分析。（注意此操作可能導致jvm進程掛掉）

jmap -dump:format=b,file=filename pid

[root@localhost ~]$ jmap -dump:format=b,file=20200531.hp 21381
Dumping heap to /home/root/20200531.hp ...
Heap dump file created

JVM問題排查之常用命令

開篇

linux相關

jinfo

jstack

jstat

jmap

手撕RocketMQ源碼——ConsumeMessageService

GEO類型的命令

手撕RocketMQ源碼——DefaultMQPushConsumer

List類型的命令

Sort Set類型的命令

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結