java系统运行缓慢，CPU 100%，以及Full GC次数过多问题的排查工具和思路

一.排查思路

推荐参考: https://my.oschina.net/zhangxufeng/blog/3017521

 

二.排查工具使用

寻找服务进程(如2823)

[root@zuo demo]# jps

查看服务每个内存区域情况

[root@zuo demo]# jmap -heap 2823 Attaching to process ID 5187, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.161-b12 using thread-local object allocation. Parallel GC with 4 thread(s) Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 1572864000 (1500.0MB) NewSize = 349175808 (333.0MB) MaxNewSize = 524288000 (500.0MB) OldSize = 699400192 (667.0MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 (0.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 187170816 (178.5MB) used = 171007032 (163.08501434326172MB) free = 16163784 (15.414985656738281MB) 91.36415369370404% used From Space: capacity = 4718592 (4.5MB) used = 4278352 (4.0801544189453125MB) free = 440240 (0.4198455810546875MB) 90.67009819878473% used To Space: capacity = 108527616 (103.5MB) used = 0 (0.0MB) free = 108527616 (103.5MB) 0.0% used PS Old Generation capacity = 1048576000 (1000.0MB) used = 94086448 (89.72782897949219MB) free = 954489552 (910.2721710205078MB) 8.97278289794922% used

查看该进程对象空间占用排名看有没有业务大对象排在前面

[root@zuo demo]# jmap -histo:live 2823 > aaa.txt [root@zuo demo]# more aaa.txt num #instances #bytes class name ---------------------------------------------- 1: 189797 20947312 [C 2: 88129 7755352 java.lang.reflect.Method 3: 188451 6030432 java.util.concurrent.ConcurrentHashMap$Node 4: 187165 4491960 java.lang.String 5: 1052 4035808 [J 6: 83755 3350200 java.util.LinkedHashMap$Entry 7: 58419 2804112 org.aspectj.weaver.reflect.ShadowMatchImpl 8: 47515 2608432 [Ljava.lang.Object; 9: 46272 2591232 java.util.LinkedHashMap 10: 13806 2197104 [B 11: 19004 2141400 java.lang.Class 12: 30251 2099096 [Ljava.util.HashMap$Node; 13: 1083 1989080 [Ljava.util.concurrent.ConcurrentHashMap$Node; 14: 58419 1869408 org.aspectj.weaver.patterns.ExposedState 15: 43429 1389728 java.util.HashMap$Node 16: 8325 1323768 [I 17: 42141 1011384 java.util.ArrayList 18: 40578 973872 org.springframework.core.MethodClassKey 19: 52418 838688 java.lang.Object 20: 36074 803976 [Ljava.lang.Class; 21: 17623 563936 java.util.LinkedList 22: 19220 517112 [Ljava.lang.String; 23: 15732 503424 java.lang.ref.WeakReference 24: 5909 425448 org.springframework.core.annotation.AnnotationAttributes

查看GC/内存占用/cpu占用/死锁检测等[jconsole]

 

查看该进程下所有线程的运行情况[top]

看哪个线程占用内存或者cpu高?

[root@zuo demo]# top -H -p 2823 top - 17:50:39 up 8 days, 4:37, 7 users, load average: 0.04, 0.04, 0.09 Threads: 101 total, 0 running, 101 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.1 st KiB Mem : 14203320 total, 355044 free, 12963976 used, 884300 buff/cache KiB Swap: 0 total, 0 free, 0 used. 917336 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5187 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.04 java 5191 root 20 0 5462568 1.934g 14652 S 0.0 14.3 2:44.18 java 5192 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.53 java 5193 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.70 java 5194 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.32 java 5195 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.41 java 5202 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:07.31 java 5203 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.03 java 5204 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.25 java 5205 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java 5206 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java 5207 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java 5208 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:53.65 java 5209 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:51.13 java 5210 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:14.52 java

 

查看该进程下所有线程运行信息[jstack]

看看有没死锁信息;

找找cpu或内存高的进程在做什么?

[root@zuo demo]# jstack 2823 > bbb.txt [root@zuo demo]# more bbb.txt 2020-07-01 17:53:01 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode): "Attach Listener" #155 daemon prio=9 os_prio=0 tid=0x00007fd600209000 nid=0x184d waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "RMI TCP Connection(24)-192.168.18.135" #153 daemon prio=5 os_prio=0 tid=0x00007fd5ec013800 nid=0x17fe runnable [0x00007fd5a16e0000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x00000000a79d1e20> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:83) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$240/1511977249.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RMI TCP Connection(20)-192.168.18.135" #151 daemon prio=5 os_prio=0 tid=0x00007fd5ec011000 nid=0x17fa runnable [0x00007fd5a1fe7000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x00000000a79d63d8> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:83) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)

该命令是查看进程2823下所有线程此刻的运行信息,但是线程号nid是16进制的,需要通过linux命令进行转换,看占用资源高的进行具体是在运行什么

printf %x 十六进制数  --> 输出十进制

printf %d 十进制数 --> 输出十六进制