Linux運維排查問題

[mart_cd@BJYF-Client-15244 ~]$ top
top - 14:07:25 up 322 days, 21:16, 5 users, load average: 0.07, 0.53, 0.84
Tasks: 623 total, 1 running, 620 sleeping, 1 stopped, 1 zombie
%Cpu(s): 0.2 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65565628 total, 32825828 free, 6729156 used, 26010644 buff/cache
KiB Swap: 16777212 total, 14644392 free, 2132820 used. 58094756 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
56590 mart_cd 20 0 20.973g 2.310g 6244 S 7.0 3.7 2175:26 java
101114 mart_cd 20 0 7552872 202144 4448 S 0.3 0.3 1222:16 java
101280 mart_cd 20 0 3445440 60496 1712 S 0.3 0.1 417:37.02 java
1 root 20 0 56384 17660 1388 S 0.0 0.0 5:34.37 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:04.10 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 4:18.04 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
8 root rt 0 0 0 0 S 0.0 0.0 0:29.16 migration/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18
29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20
31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21
32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22
33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23
34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/24
35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/25
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/26
37 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/27
38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/28
39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/29
40 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/30
41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/31
42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/32
43 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/33
44 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/34

1. jps -lvm
   92530 org.apache.hadoop.util.RunJar /software/servers/hive-cli-1.0-SNAPSHOT.jar org.apache.hadoop.hive.cli.CliDriver -Xmx2048m -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/software/servers/druid/mart_cd/hive/conf/parquet-logging.properties -Dhadoop.log.dir=/data0/hadoop-logs/ -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/software/servers/hadoop-2.7.1 -Dhadoop.id.str=mart_cd -Dhadoop.root.logger=INFO,console -Djava.library.path=/software/servers/hadoop-2.7.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/software/servers/druid/mart_cd/hive/conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender
   94228 sun.tools.jps.Jps -lvm -Dapplication.home=/software/servers/jdk1.7.0_67 -Xms8m
   101280 com.jd.update.client.timer.ServiceNT -Dfile.encoding=UTF-8 -Xmx256m -Djava.library.path=../lib -Dwrapper.key=KXQPoyBZElItMyTN -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=101275 -Dwrapper.version=3.5.12 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1

3.jstack -l 92530
[aaa@fasg4 ~]$ jstack -l 92530
2017-07-04 14:10:59
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):

“Attach Listener” daemon prio=10 tid=0x00007f5a340e9800 nid=0x170a8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“org.apache.hadoop.hdfs.PeerCache@55e563” daemon prio=10 tid=0x00007f5a46429000 nid=0x238fa sleeping[0x00007f5a26ca9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255)
at org.apache.hadoop.hdfs.PeerCache.access000(PeerCache.java:46)atorg.apache.hadoop.hdfs.PeerCache 1.run(PeerCache.java:124)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

“NonBlockingInputStreamThread” daemon prio=10 tid=0x00007f5a46176000 nid=0x16a17 in Object.wait() [0x00007f5a269a6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000782b0f550> (a jline.internal.NonBlockingInputStream)
at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
- locked <0x0000000782b0f550> (a jline.internal.NonBlockingInputStream)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

“client DomainSocketWatcher” daemon prio=10 tid=0x00007f5a45610000 nid=0x16a05 runnable [0x00007f5a27588000]
java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at org.apache.hadoop.net.unix.DomainSocketWatcher.access900(DomainSocketWatcher.java:52)atorg.apache.hadoop.net.unix.DomainSocketWatcher 2.run(DomainSocketWatcher.java:511)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

“Service Thread” daemon prio=10 tid=0x00007f5a440db800 nid=0x169dc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“C2 CompilerThread1” daemon prio=10 tid=0x00007f5a440d9000 nid=0x169db waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“C2 CompilerThread0” daemon prio=10 tid=0x00007f5a440d6000 nid=0x169da waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“Signal Dispatcher” daemon prio=10 tid=0x00007f5a440d4000 nid=0x169d9 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“Finalizer” daemon prio=10 tid=0x00007f5a440ad000 nid=0x169d8 in Object.wait() [0x00007f5a2f869000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x0000000780a95348> (a java.lang.ref.ReferenceQueueLock)atjava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)atjava.lang.ref.Finalizer FinalizerThread.run(Finalizer.java:209)

Locked ownable synchronizers:
- None

“Reference Handler” daemon prio=10 tid=0x00007f5a440a5000 nid=0x169d7 in Object.wait() [0x00007f5a2f96a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.ReferenceReferenceHandler.run(Reference.java:133)−locked<0x0000000780a860f8>(ajava.lang.ref.Reference Lock)

Locked ownable synchronizers:
- None

“main” prio=10 tid=0x00007f5a44015800 nid=0x169be runnable [0x00007f5a4c3fa000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.read0(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:210)
at jline.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:169)
- locked <0x0000000782b0f550> (a jline.internal.NonBlockingInputStream)
at jline.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:137)
at jline.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:246)
at jline.internal.InputStreamReader.read(InputStreamReader.java:261)
- locked <0x0000000782b0f550> (a jline.internal.NonBlockingInputStream)
at jline.internal.InputStreamReader.read(InputStreamReader.java:198)
- locked <0x0000000782b0f550> (a jline.internal.NonBlockingInputStream)
at jline.console.ConsoleReader.readCharacter(ConsoleReader.java:2145)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2349)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2269)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:944)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:895)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:810)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Locked ownable synchronizers:
- None

“VM Thread” prio=10 tid=0x00007f5a440a0800 nid=0x169d6 runnable

“GC task thread#0 (ParallelGC)” prio=10 tid=0x00007f5a4402c000 nid=0x169bf runnable

“GC task thread#1 (ParallelGC)” prio=10 tid=0x00007f5a4402e000 nid=0x169c0 runnable

“GC task thread#2 (ParallelGC)” prio=10 tid=0x00007f5a4402f800 nid=0x169c1 runnable

“GC task thread#3 (ParallelGC)” prio=10 tid=0x00007f5a44031800 nid=0x169c2 runnable

“GC task thread#4 (ParallelGC)” prio=10 tid=0x00007f5a44033800 nid=0x169c3 runnable

“GC task thread#5 (ParallelGC)” prio=10 tid=0x00007f5a44035800 nid=0x169c4 runnable

“GC task thread#6 (ParallelGC)” prio=10 tid=0x00007f5a44037800 nid=0x169c5 runnable

“GC task thread#7 (ParallelGC)” prio=10 tid=0x00007f5a44039800 nid=0x169c6 runnable

“GC task thread#8 (ParallelGC)” prio=10 tid=0x00007f5a4403b800 nid=0x169c7 runnable

“GC task thread#9 (ParallelGC)” prio=10 tid=0x00007f5a4403d000 nid=0x169c8 runnable

“GC task thread#10 (ParallelGC)” prio=10 tid=0x00007f5a4403f000 nid=0x169c9 runnable

“GC task thread#11 (ParallelGC)” prio=10 tid=0x00007f5a44041000 nid=0x169ca runnable

“GC task thread#12 (ParallelGC)” prio=10 tid=0x00007f5a44043000 nid=0x169cb runnable

“GC task thread#13 (ParallelGC)” prio=10 tid=0x00007f5a44045000 nid=0x169cc runnable

“GC task thread#14 (ParallelGC)” prio=10 tid=0x00007f5a44047000 nid=0x169cd runnable

“GC task thread#15 (ParallelGC)” prio=10 tid=0x00007f5a44048800 nid=0x169ce runnable

“GC task thread#16 (ParallelGC)” prio=10 tid=0x00007f5a4404a800 nid=0x169cf runnable

“GC task thread#17 (ParallelGC)” prio=10 tid=0x00007f5a4404c800 nid=0x169d0 runnable

“GC task thread#18 (ParallelGC)” prio=10 tid=0x00007f5a4404e800 nid=0x169d1 runnable

“GC task thread#19 (ParallelGC)” prio=10 tid=0x00007f5a44050800 nid=0x169d2 runnable

“GC task thread#20 (ParallelGC)” prio=10 tid=0x00007f5a44052800 nid=0x169d3 runnable

“GC task thread#21 (ParallelGC)” prio=10 tid=0x00007f5a44054800 nid=0x169d4 runnable

“GC task thread#22 (ParallelGC)” prio=10 tid=0x00007f5a44056000 nid=0x169d5 runnable

“VM Periodic Task Thread” prio=10 tid=0x00007f5a440ef000 nid=0x169dd waiting on condition

JNI global references: 292