Java程序崩溃原因分析：错误日志分析及解决（Cannot allocate memory）

绝地灬酷狼 2022-09-02 11:34 4990阅读 0赞

## 一、背景 ##

周末仓内派单程序奔溃，程序启动时间：2021-07-22 10:38:19；程序奔溃时间：2021-07-25 11:12:27。

2021-07-25 11:30左右群里有反映收不到拣货派单结果，经查询程序崩溃了。程序于2021-07-25 12:06:49重新启动。启动时添加了JVM的GC垃圾回收监控，`-Xloggc:./wse-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause`。

目前该参内派单程序刚上线不久，业务量不大，为了便于及时修改项目相关算法及配置，单点部署在34这台机器上，等项目稳定后迁移至docker环境并分布式部署。所以目前属于单点没有监控的状态。

相关分析报告情况如下。

## 二、相关环境情况 ##

### 2.1.机器物理内存和核数 ###

64G/16C

[ops@dis-algo34 pick-task-assign-job]$ free -m           
                  total        used        free      shared  buff/cache   available
    Mem:          64264       17561       37012           9        9690       46106
    Swap:             0           0           0
    
    [ops@dis-algo34 pick-task-assign-job]$ top
    top - 16:16:33 up 95 days,  4:40,  1 user,  load average: 0.05, 0.16, 0.26
    Tasks: 220 total,   1 running, 215 sleeping,   4 stopped,   0 zombie
    %Cpu0  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu4  :  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu6  :  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu7  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu8  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu9  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu10 :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu11 :  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu12 :  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu13 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu14 :  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu15 :  0.3 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 65806676 total, 38360376 free, 17231528 used, 10214772 buff/cache
    KiB Swap:        0 total,        0 free,        0 used. 47969452 avail Mem

### 2.2.仓内派单java程序程序启动参数 ###

我们使用的是jdk1.8，程序启动参数如下:

java -Dspring.profiles.active=tce -Djava.library.path=/data/cplex -jar app.jar

由于启动时没有指定具体的堆栈大小，使用默认配置，查看默认配置：

[ops@dis-algo34 pick-task-assign-job]$ java -XX:+PrintCommandLineFlags -version
    -XX:InitialHeapSize=1052906816 -XX:MaxHeapSize=16846509056 -XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC 
    java version "1.8.0_251"
    Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
    Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
    [ops@dis-algo34 pick-task-assign-job]$

初始堆内存InitialHeapSize：1052906816字节=1G ；最大堆内存MaxHeapSize：16846509056字节=16G。JVM初始分配的内存默认是物理内存的1/64；JVM最大分配的内存默认是物理内存的1/4。

默认空余堆内存小于40%时，JVM就会增大堆直到-Xmx的最大限制；空余堆内存大于70%时，JVM会减少堆直到-Xms的最小限制。因此服务器一般设置-Xms、-Xmx相等以避免在每次GC后调整堆的大小。使用-XX:-UseAdaptiveSizePolicy 避免堆大小的自适应扩缩容。

### 2.3.机器使用情况 ###

目前该机器是公用的，上面部署了很多其他程序。

派单程序崩溃时其他程序的资源使用情况如下：

![image-20210726162020979][]

pid=28179的程序内存一直是超过40%，cpu的瞬时使用率也很高。这个程序一直占用40%以上的内存可能也是派单程序奔溃的外部原因之一。

## 三、日志分析 ##

程序上线运行了一周多的时间，周六突然程序挂掉了，日志的最后几行显示：

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000513200000, 3075473408, 0) failed; error='Cannot allocate memory' (errno=12)
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 3075473408 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /data/Projects/pick-task-assign-job/hs_err_pid5421.log

上面报错：Cannot allocate memory，不能分配内存；“内存不足，Java运行时环境无法继续。本地内存分配3075473408字节（3G）失败。错误信息存储在/data/Projects/pick-task-assign-job/hs\_err\_pid5421.log文件中”。

JVM出现的致命错误，会在默认工作目录下面生成了hs\_err\_pid.log 日志文件。其中包含了导致JVM crash 的重要信息（可以在虚拟参数中添加 -XX:ErrorFie=./hss\_err\_pid.log指定生成路径。）

hs\_err\_pid.log 日志文件包含：日志头文件，导致crash的线程信息，所有的线程信息，安全点和锁信息，堆信息，本地代码缓存，编译事件，gc相关记录，jvm内存映射，jvm启动参数，服务器信息。下面简单看下hs\_err\_pid5421.log文件：

[ops@dis-algo34 pick-task-assign-job]$ cat hs_err_pid5421.log 
    
    # 1.首先是日志头信息。这里已经详细告诉我们原因了：进程号5421,由于物理内存不足，已经出现Out of Memory Error错误。同时提供了解决方案：减轻机器内存负载，设置Xmx /xms减少java堆内存空间大小，如果是64位的操作系统建议使用64bit java，减少java线程数量，减少每个线程堆栈大小-Xss。
    
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 3075473408 bytes for committing reserved memory.
    # Possible reasons:
    # The system is out of physical RAM or swap space
    # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
    # Possible solutions:
    # Reduce memory load on the system
    # Increase physical memory or swap space
    # Check if swap backing store is full
    # Decrease Java heap size (-Xmx/-Xms)
    # Decrease number of Java threads
    # Decrease Java thread stack sizes (-Xss)
    # Set larger code cache with -XX:ReservedCodeCacheSize=
    # JVM is running with Zero Based Compressed Oops mode in which the Java heap is
    # placed in the first 32GB address space. The Java Heap base address is the
    # maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
    # to set the Java Heap base and to place the Java Heap above 32GB virtual address.
    # This output file may be truncated or incomplete.
    #
    # Out of Memory Error (os_linux.cpp:2749), pid=5421, tid=0x00007ff48d6d6700
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_251-b08) (build 1.8.0_251-b08)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.251-b08 mixed mode linux-amd64 compressed oops)
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    #
    
    #2.导致程序crash的线程信息。导致程序奔溃的线程id=5442，线程类型是：VMThread，这种类型的线程是jvm内部线程。还有其他类型的线程，如JavaThread表示执行的是java线程，GCTaskThread是执行gc的线程，CompilerThread是用来调用JITing实时编译装卸class，WatcherThread是jvm周期性任务调度的线程。
    ---------------  T H R E A D  ---------------
    
    Current thread (0x00007ff4e4272000):  VMThread [stack: 0x00007ff48d5d7000,0x00007ff48d6d7000] [id=5442]
    
    Stack: [0x00007ff48d5d7000,0x00007ff48d6d7000],  sp=0x00007ff48d6d53f0,  free space=1016k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0xad9dc5]  VMError::report_and_die()+0x2e5
    V  [libjvm.so+0x4e1d67]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67 # 内存不足导致分配失败这就是JVM发生致命的原因所在
    V  [libjvm.so+0x912cb0]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x100
    V  [libjvm.so+0x90a2df]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x1f
    V  [libjvm.so+0x98f0e6]  PSVirtualSpace::expand_by(unsigned long)+0x56
    V  [libjvm.so+0x97e3bb]  PSOldGen::expand_by(unsigned long) [clone .part.28]+0x1b
    V  [libjvm.so+0x97e578]  PSOldGen::expand(unsigned long) [clone .part.29]+0xa8
    V  [libjvm.so+0x97f121]  PSOldGen::resize(unsigned long)+0xe1
    V  [libjvm.so+0x98525c]  PSParallelCompact::invoke_no_policy(bool) [clone .part.118]+0xa2c
    V  [libjvm.so+0x98d075]  PSScavenge::invoke()+0x215
    V  [libjvm.so+0x93cbf8]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x68 #内存分配失败
    V  [libjvm.so+0xadb913]  VM_ParallelGCFailedAllocation::doit()+0x93#内存分配失败
    V  [libjvm.so+0xae0b66]  VM_Operation::evaluate()+0x46
    V  [libjvm.so+0xadee9d]  VMThread::evaluate_operation(VM_Operation*) [clone .constprop.44]+0xcd
    V  [libjvm.so+0xadf483]  VMThread::loop()+0x3a3
    V  [libjvm.so+0xadf858]  VMThread::run()+0x78
    V  [libjvm.so+0x9102e2]  java_start(Thread*)+0x102
    #下面这句：ParallelGCFailedAllocation，表示执行Parallel垃圾回收过程中内存分派失败，这可能导致的内存溢出。
    VM_Operation (0x00007ff27a5f1ba0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007fefa1290000
    
    
    #3.下面是程序奔溃时正在运行的线程信息，其中双引号中是线程名称。最后一个线程就是导致程序奔溃的线程
    ---------------  P R O C E S S  ---------------
    #3.1.java线程
    Java Threads: ( => current thread )
      0x00007ff1ba676800 JavaThread "Unknown thread" [_thread_blocked, id=24432, stack(0x00007ff281a9d000,0x00007ff28229d000)]
      0x00007ff306e4d000 JavaThread "Unknown thread" [_thread_blocked, id=24431, stack(0x00007ff27b5f6000,0x00007ff27bdf6000)]
      0x00007ff2ffb8b000 JavaThread "Unknown thread" [_thread_blocked, id=24430, stack(0x00007ff27adf5000,0x00007ff27b5f5000)]
      0x00007ff30204d000 JavaThread "Unknown thread" [_thread_blocked, id=24429, stack(0x00007ff27a5f4000,0x00007ff27adf4000)]
      0x00007fefa1290000 JavaThread "Unknown thread" [_thread_blocked, id=24428, stack(0x00007ff279df3000,0x00007ff27a5f3000)]
      0x00007fefc871a800 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=24450, stack(0x00007ff282cdd000,0x00007ff282dde000)]
      0x00007fee88687000 JavaThread "ForkJoinPool.commonPool-worker-13" daemon [_thread_blocked, id=5498, stack(0x00007ff287eff000,0x00007ff288000000)]
      0x00007fedbfaef000 JavaThread "ForkJoinPool.commonPool-worker-4" daemon [_thread_blocked, id=32156, stack(0x00007ff285b09000,0x00007ff285c0a000)]
      0x00007fee9eb0f800 JavaThread "ForkJoinPool.commonPool-worker-11" daemon [_thread_in_native, id=705, stack(0x00007ff282bdc000,0x00007ff282cdd000)]
      0x00007ff4cb461800 JavaThread "ForkJoinPool.commonPool-worker-2" daemon [_thread_blocked, id=30106, stack(0x00007ff2840ef000,0x00007ff2841f0000)]
      0x00007ff356726000 JavaThread "ForkJoinPool.commonPool-worker-5" daemon [_thread_blocked, id=13133, stack(0x00007ff28229d000,0x00007ff28239e000)]
      0x00007ff3c8077800 JavaThread "lettuce-eventExecutorLoop-1-16" daemon [_thread_blocked, id=28265, stack(0x00007ff2839ea000,0x00007ff283aeb000)]
      0x00007ff3c8077000 JavaThread "lettuce-eventExecutorLoop-1-15" daemon [_thread_blocked, id=28235, stack(0x00007ff48a177000,0x00007ff48a278000)]
      0x00007ff3c805b000 JavaThread "lettuce-eventExecutorLoop-1-14" daemon [_thread_blocked, id=25593, stack(0x00007ff282dde000,0x00007ff282edf000)]
      0x00007ff3c805a800 JavaThread "lettuce-eventExecutorLoop-1-13" daemon [_thread_blocked, id=25569, stack(0x00007ff282edf000,0x00007ff282fe0000)]
      0x00007ff3c8059800 JavaThread "lettuce-eventExecutorLoop-1-12" daemon [_thread_blocked, id=22888, stack(0x00007ff282fe0000,0x00007ff2830e1000)]
      0x00007ff3c8081000 JavaThread "lettuce-eventExecutorLoop-1-11" daemon [_thread_blocked, id=22872, stack(0x00007ff2832e3000,0x00007ff2833e4000)]
      0x00007ff3c8080800 JavaThread "lettuce-eventExecutorLoop-1-10" daemon [_thread_blocked, id=19918, stack(0x00007ff2830e1000,0x00007ff2831e2000)]
      0x00007ff3c805d000 JavaThread "lettuce-eventExecutorLoop-1-9" daemon [_thread_blocked, id=19905, stack(0x00007ff2831e2000,0x00007ff2832e3000)]
      0x00007ff3c805c800 JavaThread "lettuce-eventExecutorLoop-1-8" daemon [_thread_blocked, id=16974, stack(0x00007ff2833e4000,0x00007ff2834e5000)]
      0x00007ff3c805e000 JavaThread "lettuce-eventExecutorLoop-1-7" daemon [_thread_blocked, id=16965, stack(0x00007ff2836e7000,0x00007ff2837e8000)]
      0x00007ff3c8051000 JavaThread "lettuce-eventExecutorLoop-1-6" daemon [_thread_blocked, id=14162, stack(0x00007ff2834e5000,0x00007ff2835e6000)]
      0x00007ff3c8050000 JavaThread "lettuce-eventExecutorLoop-1-5" daemon [_thread_blocked, id=14161, stack(0x00007ff2835e6000,0x00007ff2836e7000)]
      0x00007ff3c807e800 JavaThread "lettuce-eventExecutorLoop-1-4" daemon [_thread_blocked, id=11346, stack(0x00007ff2837e8000,0x00007ff2838e9000)]
      0x00007ff3c807b000 JavaThread "lettuce-eventExecutorLoop-1-3" daemon [_thread_blocked, id=11345, stack(0x00007ff2838e9000,0x00007ff2839ea000)]
      0x00007ff3c8079800 JavaThread "lettuce-eventExecutorLoop-1-2" daemon [_thread_blocked, id=8628, stack(0x00007ff283aeb000,0x00007ff283bec000)]
      0x00007ff3c800e800 JavaThread "lettuce-eventExecutorLoop-1-1" daemon [_thread_blocked, id=8627, stack(0x00007ff283bec000,0x00007ff283ced000)]
      0x00007ff334064800 JavaThread "lettuce-timer-3-1" [_thread_blocked, id=8607, stack(0x00007ff283ced000,0x00007ff283dee000)]
      0x00007ff3e4008800 JavaThread "logback-8" daemon [_thread_blocked, id=6327, stack(0x00007ff283fee000,0x00007ff2840ef000)]
      0x00007ff2cc002000 JavaThread "logback-7" daemon [_thread_blocked, id=6212, stack(0x00007ff2841f0000,0x00007ff2842f1000)]
      0x00007ff3b0004800 JavaThread "ConsumeMessageThread_20" [_thread_blocked, id=6200, stack(0x00007ff2847f6000,0x00007ff2848f7000)]
      0x00007ff3b8003000 JavaThread "ConsumeMessageThread_19" [_thread_blocked, id=6177, stack(0x00007ff2842f1000,0x00007ff2843f2000)]
      0x00007ff310001800 JavaThread "logback-6" daemon [_thread_blocked, id=6162, stack(0x00007ff2843f2000,0x00007ff2844f3000)]
      0x00007ff288028000 JavaThread "wse-pool-19" [_thread_blocked, id=6139, stack(0x00007ff2844f3000,0x00007ff2845f4000)]
      0x00007ff288026800 JavaThread "wse-pool-18" [_thread_blocked, id=6136, stack(0x00007ff2845f4000,0x00007ff2846f5000)]
      0x00007ff3c800b800 JavaThread "ConsumeMessageThread_18" [_thread_blocked, id=6130, stack(0x00007ff2846f5000,0x00007ff2847f6000)]
      0x00007ff288025000 JavaThread "wse-pool-17" [_thread_blocked, id=5971, stack(0x00007ff2848f7000,0x00007ff2849f8000)]
      0x00007ff288023000 JavaThread "wse-pool-16" [_thread_blocked, id=5970, stack(0x00007ff2849f8000,0x00007ff284af9000)]
      0x00007ff288021000 JavaThread "wse-pool-15" [_thread_blocked, id=5969, stack(0x00007ff284af9000,0x00007ff284bfa000)]
      0x00007ff288020000 JavaThread "wse-pool-14" [_thread_blocked, id=5968, stack(0x00007ff284bfa000,0x00007ff284cfb000)]
      0x00007ff414004800 JavaThread "logback-5" daemon [_thread_blocked, id=5966, stack(0x00007ff285e0a000,0x00007ff285f0b000)]
      0x00007ff3d8001000 JavaThread "ConsumeMessageThread_17" [_thread_blocked, id=5958, stack(0x00007ff284cfb000,0x00007ff284dfc000)]
      0x00007ff3e0002000 JavaThread "ConsumeMessageThread_20" [_thread_blocked, id=5955, stack(0x00007ff284dfc000,0x00007ff284efd000)]
      0x00007ff3c0004800 JavaThread "ConsumeMessageThread_16" [_thread_blocked, id=5953, stack(0x00007ff284efd000,0x00007ff284ffe000)]
      0x00007ff28801e800 JavaThread "wse-pool-13" [_thread_blocked, id=5868, stack(0x00007ff284ffe000,0x00007ff2850ff000)]
      0x00007ff288010000 JavaThread "wse-pool-12" [_thread_blocked, id=5867, stack(0x00007ff2850ff000,0x00007ff285200000)]
      0x00007ff3a0003000 JavaThread "ConsumeMessageThread_15" [_thread_blocked, id=5865, stack(0x00007ff285200000,0x00007ff285301000)]
      0x00007ff3ac003000 JavaThread "ConsumeMessageThread_19" [_thread_blocked, id=5861, stack(0x00007ff285301000,0x00007ff285402000)]
      0x00007ff3b4003000 JavaThread "ConsumeMessageThread_14" [_thread_blocked, id=5844, stack(0x00007ff285402000,0x00007ff285503000)]
      0x00007ff28800f000 JavaThread "wse-pool-11" [_thread_blocked, id=5842, stack(0x00007ff285503000,0x00007ff285604000)]
      0x00007ff288001800 JavaThread "wse-pool-10" [_thread_blocked, id=5841, stack(0x00007ff285604000,0x00007ff285705000)]
      0x00007ff2cc001000 JavaThread "logback-4" daemon [_thread_blocked, id=5840, stack(0x00007ff285705000,0x00007ff285806000)]
      0x00007ff3c8008800 JavaThread "ConsumeMessageThread_18" [_thread_blocked, id=5839, stack(0x00007ff285806000,0x00007ff285907000)]
      0x00007ff3b0003000 JavaThread "ConsumeMessageThread_13" [_thread_blocked, id=5830, stack(0x00007ff285907000,0x00007ff285a08000)]
      0x00007ff3e0003800 JavaThread "ConsumeMessageThread_17" [_thread_blocked, id=5826, stack(0x00007ff285a08000,0x00007ff285b09000)]
      0x00007ff288007000 JavaThread "wse-pool-9" [_thread_blocked, id=5807, stack(0x00007ff2808fd000,0x00007ff2809fe000)]
      0x00007ff288018000 JavaThread "wse-pool-8" [_thread_blocked, id=5806, stack(0x00007ff2809fe000,0x00007ff280aff000)]
      0x00007ff3cc006800 JavaThread "ConsumeMessageThread_16" [_thread_blocked, id=5784, stack(0x00007ff27e5fb000,0x00007ff27e6fc000)]
      0x00007ff314bf4000 JavaThread "kafka-producer-network-thread | producer-1" daemon [_thread_in_native, id=5764, stack(0x00007ff2f82e9000,0x00007ff2f83ea000)]
      0x00007ff314bae000 JavaThread "Druid-ConnectionPool-Destroy-519492428" daemon [_thread_blocked, id=5763, stack(0x00007ff2f85ea000,0x00007ff2f86eb000)]
      0x00007ff314bad000 JavaThread "Druid-ConnectionPool-Create-519492428" daemon [_thread_blocked, id=5762, stack(0x00007ff2f86eb000,0x00007ff2f87ec000)]
      0x00007ff314baa800 JavaThread "Druid-ConnectionPool-Destroy-1052253947" daemon [_thread_blocked, id=5761, stack(0x00007ff2f87ec000,0x00007ff2f88ed000)]
      0x00007ff314ba9800 JavaThread "Druid-ConnectionPool-Create-1052253947" daemon [_thread_blocked, id=5760, stack(0x00007ff2f88ed000,0x00007ff2f89ee000)]
      0x00007ff3a0001000 JavaThread "ConsumeMessageThread_12" [_thread_blocked, id=5748, stack(0x00007ff2f8dee000,0x00007ff2f8eef000)]
      0x00007ff288016000 JavaThread "wse-pool-7" [_thread_blocked, id=5744, stack(0x00007ff2f90ef000,0x00007ff2f91f0000)]
      0x00007ff288014800 JavaThread "wse-pool-6" [_thread_blocked, id=5743, stack(0x00007ff2f91f0000,0x00007ff2f92f1000)]
      0x00007ff288012800 JavaThread "wse-pool-5" [_thread_blocked, id=5742, stack(0x00007ff2f92f1000,0x00007ff2f93f2000)]
      0x00007ff288011000 JavaThread "wse-pool-4" [_thread_blocked, id=5741, stack(0x00007ff2f93f2000,0x00007ff2f94f3000)]
      0x00007ff414002800 JavaThread "logback-3" daemon [_thread_blocked, id=5739, stack(0x00007ff2f94f3000,0x00007ff2f95f4000)]
      0x00007ff3ac001000 JavaThread "ConsumeMessageThread_15" [_thread_blocked, id=5738, stack(0x00007ff2f95f4000,0x00007ff2f96f5000)]
      0x00007ff3a8001000 JavaThread "ConsumeMessageThread_11" [_thread_blocked, id=5727, stack(0x00007ff2f96f5000,0x00007ff2f97f6000)]
      0x00007ff3b4001000 JavaThread "ConsumeMessageThread_14" [_thread_blocked, id=5720, stack(0x00007ff2f97f6000,0x00007ff2f98f7000)]
      0x00007ff28800d800 JavaThread "wse-pool-3" [_thread_blocked, id=5703, stack(0x00007ff2f98f7000,0x00007ff2f99f8000)]
      0x00007ff28800b800 JavaThread "wse-pool-2" [_thread_blocked, id=5702, stack(0x00007ff2f99f8000,0x00007ff2f9af9000)]
      0x00007ff3b0001000 JavaThread "ConsumeMessageThread_13" [_thread_blocked, id=5697, stack(0x00007ff2f9af9000,0x00007ff2f9bfa000)]
      0x00007ff2d4021000 JavaThread "Druid-ConnectionPool-Destroy-669284403" daemon [_thread_blocked, id=5689, stack(0x00007ff2f9bfa000,0x00007ff2f9cfb000)]
      0x00007ff2d4020000 JavaThread "Druid-ConnectionPool-Create-669284403" daemon [_thread_blocked, id=5688, stack(0x00007ff2f9cfb000,0x00007ff2f9dfc000)]
      0x00007ff2d4015000 JavaThread "Druid-ConnectionPool-Destroy-85748029" daemon [_thread_blocked, id=5687, stack(0x00007ff2f9dfc000,0x00007ff2f9efd000)]
      0x00007ff2d4014000 JavaThread "Druid-ConnectionPool-Create-85748029" daemon [_thread_blocked, id=5686, stack(0x00007ff2f9efd000,0x00007ff2f9ffe000)]
      0x00007ff2c8008000 JavaThread "Druid-ConnectionPool-Destroy-294111720" daemon [_thread_blocked, id=5683, stack(0x00007ff3080f0000,0x00007ff3081f1000)]
      0x00007ff2c8007000 JavaThread "Druid-ConnectionPool-Create-294111720" daemon [_thread_blocked, id=5682, stack(0x00007ff3081f1000,0x00007ff3082f2000)]
      0x00007ff288009800 JavaThread "wse-pool-1" [_thread_blocked, id=5661, stack(0x00007ff3082f2000,0x00007ff3083f3000)]
      0x00007ff288008800 JavaThread "wse-pool-0" [_thread_blocked, id=5660, stack(0x00007ff3083f3000,0x00007ff3084f4000)]
      0x00007ff414001000 JavaThread "logback-2" daemon [_thread_blocked, id=5659, stack(0x00007ff3084f4000,0x00007ff3085f5000)]
      0x00007ff4e4009800 JavaThread "DestroyJavaVM" [_thread_blocked, id=5422, stack(0x00007ff4ea15a000,0x00007ff4ea25a000)]
      0x00007ff4e4e8c800 JavaThread "http-nio-8080-Acceptor-0" daemon [_thread_in_native, id=5649, stack(0x00007ff3087f5000,0x00007ff3088f6000)]
      0x00007ff4e4c83800 JavaThread "http-nio-8080-ClientPoller-1" daemon [_thread_blocked, id=5648, stack(0x00007ff3088f6000,0x00007ff3089f7000)]
      0x00007ff4e5cb1000 JavaThread "http-nio-8080-ClientPoller-0" daemon [_thread_blocked, id=5647, stack(0x00007ff3089f7000,0x00007ff308af8000)]
      0x00007ff4e47a8800 JavaThread "http-nio-8080-exec-10" daemon [_thread_blocked, id=5646, stack(0x00007ff308af8000,0x00007ff308bf9000)]
      0x00007ff4e4954000 JavaThread "http-nio-8080-exec-9" daemon [_thread_blocked, id=5645, stack(0x00007ff308bf9000,0x00007ff308cfa000)]
      0x00007ff4e4bd7800 JavaThread "http-nio-8080-exec-8" daemon [_thread_blocked, id=5644, stack(0x00007ff308cfa000,0x00007ff308dfb000)]
      0x00007ff4e5018000 JavaThread "http-nio-8080-exec-7" daemon [_thread_blocked, id=5643, stack(0x00007ff308dfb000,0x00007ff308efc000)]
      0x00007ff4e516f800 JavaThread "http-nio-8080-exec-6" daemon [_thread_blocked, id=5642, stack(0x00007ff308efc000,0x00007ff308ffd000)]
      0x00007ff4e49b1800 JavaThread "http-nio-8080-exec-5" daemon [_thread_blocked, id=5641, stack(0x00007ff3180f0000,0x00007ff3181f1000)]
      0x00007ff4e56cb800 JavaThread "http-nio-8080-exec-4" daemon [_thread_blocked, id=5640, stack(0x00007ff3181f1000,0x00007ff3182f2000)]
      0x00007ff4e4f40000 JavaThread "http-nio-8080-exec-3" daemon [_thread_blocked, id=5639, stack(0x00007ff3182f2000,0x00007ff3183f3000)]
      0x00007ff4e4d20000 JavaThread "http-nio-8080-exec-2" daemon [_thread_blocked, id=5638, stack(0x00007ff3183f3000,0x00007ff3184f4000)]
      0x00007ff4e4d1f000 JavaThread "http-nio-8080-exec-1" daemon [_thread_blocked, id=5637, stack(0x00007ff3184f4000,0x00007ff3185f5000)]
      0x00007ff4e4988800 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_blocked, id=5636, stack(0x00007ff3185f5000,0x00007ff3186f6000)]
      0x00007ff4e4fbf000 JavaThread "scheduling-1" [_thread_blocked, id=5635, stack(0x00007ff3186f6000,0x00007ff3187f7000)]
      0x00007ff2b0001800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5633, stack(0x00007ff3187f7000,0x00007ff3188f8000)]
      0x00007ff2a0002000 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5632, stack(0x00007ff3188f8000,0x00007ff3189f9000)]
      0x00007ff298001800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5631, stack(0x00007ff3189f9000,0x00007ff318afa000)]
      0x00007ff2a8009800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5630, stack(0x00007ff318afa000,0x00007ff318bfb000)]
      0x00007ff290004000 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5629, stack(0x00007ff318bfb000,0x00007ff318cfc000)]
      0x00007ff4e5e95000 JavaThread "kafkaListenerFeature-4-C-1" [_thread_blocked, id=5628, stack(0x00007ff318efc000,0x00007ff318ffd000)]
      0x00007ff4e5e93800 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5627, stack(0x00007ff3280f2000,0x00007ff3281f3000)]
      0x00007ff4e5e78800 JavaThread "kafkaListenerFeature-3-C-1" [_thread_blocked, id=5626, stack(0x00007ff3281f3000,0x00007ff3282f4000)]
      0x00007ff4e5e77000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5625, stack(0x00007ff3282f4000,0x00007ff3283f5000)]
      0x00007ff4e5e5c000 JavaThread "kafkaListenerFeature-2-C-1" [_thread_blocked, id=5624, stack(0x00007ff3283f5000,0x00007ff3284f6000)]
      0x00007ff4e5e5a000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5623, stack(0x00007ff3284f6000,0x00007ff3285f7000)]
      0x00007ff4e5e3f800 JavaThread "kafkaListenerFeature-1-C-1" [_thread_blocked, id=5622, stack(0x00007ff3285f7000,0x00007ff3286f8000)]
      0x00007ff4e5e3f000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5621, stack(0x00007ff3286f8000,0x00007ff3287f9000)]
      0x00007ff4e5e24000 JavaThread "kafkaListenerFeature-0-C-1" [_thread_blocked, id=5620, stack(0x00007ff3287f9000,0x00007ff3288fa000)]
      0x00007ff4e5e19000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5619, stack(0x00007ff3288fa000,0x00007ff3289fb000)]
      0x00007ff4e5132000 JavaThread "quartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=5617, stack(0x00007ff328bfb000,0x00007ff328cfc000)]
      0x00007ff4e5993800 JavaThread "quartzScheduler_Worker-10" [_thread_blocked, id=5616, stack(0x00007ff328efc000,0x00007ff328ffd000)]
      0x00007ff4e5991800 JavaThread "quartzScheduler_Worker-9" [_thread_blocked, id=5615, stack(0x00007ff36c0f2000,0x00007ff36c1f3000)]
      0x00007ff4e53f0800 JavaThread "quartzScheduler_Worker-8" [_thread_blocked, id=5614, stack(0x00007ff36c1f3000,0x00007ff36c2f4000)]
      0x00007ff4e53ef000 JavaThread "quartzScheduler_Worker-7" [_thread_blocked, id=5613, stack(0x00007ff36c2f4000,0x00007ff36c3f5000)]
      0x00007ff4e5518800 JavaThread "quartzScheduler_Worker-6" [_thread_blocked, id=5612, stack(0x00007ff36c3f5000,0x00007ff36c4f6000)]
      0x00007ff4e5517800 JavaThread "quartzScheduler_Worker-5" [_thread_blocked, id=5611, stack(0x00007ff36c4f6000,0x00007ff36c5f7000)]
      0x00007ff4e5972800 JavaThread "quartzScheduler_Worker-4" [_thread_blocked, id=5610, stack(0x00007ff36c5f7000,0x00007ff36c6f8000)]
      0x00007ff4e5971800 JavaThread "quartzScheduler_Worker-3" [_thread_blocked, id=5609, stack(0x00007ff36c6f8000,0x00007ff36c7f9000)]
      0x00007ff4e4a51800 JavaThread "quartzScheduler_Worker-2" [_thread_blocked, id=5608, stack(0x00007ff36c7f9000,0x00007ff36c8fa000)]
      0x00007ff4e4a50800 JavaThread "quartzScheduler_Worker-1" [_thread_blocked, id=5607, stack(0x00007ff36c8fa000,0x00007ff36c9fb000)]
      0x00007ff37c19a000 JavaThread "Druid-ConnectionPool-Destroy-1470358122" daemon [_thread_blocked, id=5606, stack(0x00007ff36cbfb000,0x00007ff36ccfc000)]
      0x00007ff37c1a4800 JavaThread "Druid-ConnectionPool-Create-1470358122" daemon [_thread_blocked, id=5605, stack(0x00007ff36ccfc000,0x00007ff36cdfd000)]
      0x00007ff35c04f000 JavaThread "Abandoned connection cleanup thread" daemon [_thread_blocked, id=5603, stack(0x00007ff4542b8000,0x00007ff4543b9000)]
      0x00007ff38c3a9800 JavaThread "lettuce-epollEventLoop-4-16" daemon [_thread_blocked, id=5567, stack(0x00007ff4547b9000,0x00007ff4548ba000)]
      0x00007ff38c3a7000 JavaThread "lettuce-epollEventLoop-4-15" daemon [_thread_blocked, id=5566, stack(0x00007ff4548ba000,0x00007ff4549bb000)]
      0x00007ff38c3a5000 JavaThread "lettuce-epollEventLoop-4-14" daemon [_thread_blocked, id=5565, stack(0x00007ff4549bb000,0x00007ff454abc000)]
      0x00007ff38c3a3000 JavaThread "lettuce-epollEventLoop-4-13" daemon [_thread_blocked, id=5564, stack(0x00007ff4700c8000,0x00007ff4701c9000)]
      0x00007ff38c3a1800 JavaThread "lettuce-epollEventLoop-4-12" daemon [_thread_blocked, id=5563, stack(0x00007ff4701c9000,0x00007ff4702ca000)]
      0x00007ff38c39f800 JavaThread "lettuce-epollEventLoop-4-11" daemon [_thread_blocked, id=5562, stack(0x00007ff4702ca000,0x00007ff4703cb000)]
      0x00007ff38c39d800 JavaThread "lettuce-epollEventLoop-4-10" daemon [_thread_blocked, id=5561, stack(0x00007ff4703cb000,0x00007ff4704cc000)]
      0x00007ff38c39b800 JavaThread "lettuce-epollEventLoop-4-9" daemon [_thread_blocked, id=5560, stack(0x00007ff4704cc000,0x00007ff4705cd000)]
      0x00007ff38c39a000 JavaThread "lettuce-epollEventLoop-4-8" daemon [_thread_blocked, id=5559, stack(0x00007ff4705cd000,0x00007ff4706ce000)]
      0x00007ff38c398000 JavaThread "lettuce-epollEventLoop-4-7" daemon [_thread_blocked, id=5558, stack(0x00007ff4706ce000,0x00007ff4707cf000)]
      0x00007ff38c396000 JavaThread "lettuce-epollEventLoop-4-6" daemon [_thread_blocked, id=5557, stack(0x00007ff4707cf000,0x00007ff4708d0000)]
      0x00007ff38c394800 JavaThread "lettuce-epollEventLoop-4-5" daemon [_thread_blocked, id=5556, stack(0x00007ff4708d0000,0x00007ff4709d1000)]
      0x00007ff38c392800 JavaThread "lettuce-epollEventLoop-4-4" daemon [_thread_blocked, id=5555, stack(0x00007ff4709d1000,0x00007ff470ad2000)]
      0x00007ff38c391000 JavaThread "lettuce-epollEventLoop-4-3" daemon [_thread_blocked, id=5554, stack(0x00007ff470ad2000,0x00007ff470bd3000)]
      0x00007ff38c38f000 JavaThread "lettuce-epollEventLoop-4-2" daemon [_thread_blocked, id=5553, stack(0x00007ff470bd3000,0x00007ff470cd4000)]
      0x00007ff38c336000 JavaThread "lettuce-epollEventLoop-4-1" daemon [_thread_blocked, id=5552, stack(0x00007ff471cd5000,0x00007ff471dd6000)]
      0x00007ff3d400a000 JavaThread "ConsumeMessageThread_8" [_thread_blocked, id=5551, stack(0x00007ff4723e4000,0x00007ff4724e5000)]
      0x00007ff3cc004800 JavaThread "ConsumeMessageThread_9" [_thread_blocked, id=5550, stack(0x00007ff4724e5000,0x00007ff4725e6000)]
      0x00007ff3e0009000 JavaThread "ConsumeMessageThread_10" [_thread_blocked, id=5549, stack(0x00007ff4725e6000,0x00007ff4726e7000)]
      0x00007ff3c8006800 JavaThread "ConsumeMessageThread_7" [_thread_blocked, id=5548, stack(0x00007ff4726e7000,0x00007ff4727e8000)]
      0x00007ff3c8004800 JavaThread "ConsumeMessageThread_6" [_thread_blocked, id=5547, stack(0x00007ff4727e8000,0x00007ff4728e9000)]
      0x00007ff3d8005000 JavaThread "ConsumeMessageThread_5" [_thread_blocked, id=5546, stack(0x00007ff4728e9000,0x00007ff4729ea000)]
      0x00007ff3d0005000 JavaThread "ConsumeMessageThread_4" [_thread_blocked, id=5545, stack(0x00007ff4729ea000,0x00007ff472aeb000)]
      0x00007ff3c0003000 JavaThread "ConsumeMessageThread_3" [_thread_blocked, id=5544, stack(0x00007ff472aeb000,0x00007ff472bec000)]
      0x00007ff3b8001000 JavaThread "ConsumeMessageThread_2" [_thread_blocked, id=5543, stack(0x00007ff472bec000,0x00007ff472ced000)]
      0x00007ff3bc001000 JavaThread "ConsumeMessageThread_1" [_thread_blocked, id=5542, stack(0x00007ff472ced000,0x00007ff472dee000)]
      0x00007ff3e0006800 JavaThread "ConsumeMessageThread_12" [_thread_blocked, id=5541, stack(0x00007ff472dee000,0x00007ff472eef000)]
      0x00007ff3d4008800 JavaThread "ConsumeMessageThread_10" [_thread_blocked, id=5540, stack(0x00007ff472eef000,0x00007ff472ff0000)]
      0x00007ff3a4008000 JavaThread "ConsumeMessageThread_1" [_thread_blocked, id=5539, stack(0x00007ff472ff0000,0x00007ff4730f1000)]
      0x00007ff3e0005000 JavaThread "ConsumeMessageThread_3" [_thread_blocked, id=5538, stack(0x00007ff4730f1000,0x00007ff4731f2000)]
      0x00007ff3c8003000 JavaThread "ConsumeMessageThread_9" [_thread_blocked, id=5537, stack(0x00007ff4731f2000,0x00007ff4732f3000)]
      0x00007ff3cc003000 JavaThread "ConsumeMessageThread_11" [_thread_blocked, id=5536, stack(0x00007ff4732f3000,0x00007ff4733f4000)]
      0x00007ff3d8003000 JavaThread "ConsumeMessageThread_7" [_thread_blocked, id=5535, stack(0x00007ff4733f4000,0x00007ff4734f5000)]
      0x00007ff3cc001000 JavaThread "ConsumeMessageThread_5" [_thread_blocked, id=5534, stack(0x00007ff4734f5000,0x00007ff4735f6000)]
      0x00007ff3d0004000 JavaThread "ConsumeMessageThread_2" [_thread_blocked, id=5533, stack(0x00007ff4735f6000,0x00007ff4736f7000)]
      0x00007ff3d4006800 JavaThread "ConsumeMessageThread_6" [_thread_blocked, id=5532, stack(0x00007ff4736f7000,0x00007ff4737f8000)]
      0x00007ff3c8001000 JavaThread "ConsumeMessageThread_8" [_thread_blocked, id=5531, stack(0x00007ff4737f8000,0x00007ff4738f9000)]
      0x00007ff3c0001000 JavaThread "ConsumeMessageThread_4" [_thread_blocked, id=5530, stack(0x00007ff4738f9000,0x00007ff4739fa000)]
      0x00007ff3f00e3800 JavaThread "NettyClientPublicExecutor_16" [_thread_blocked, id=5529, stack(0x00007ff4739fa000,0x00007ff473afb000)]
      0x00007ff3dc00d800 JavaThread "NettyClientPublicExecutor_15" [_thread_blocked, id=5528, stack(0x00007ff473afb000,0x00007ff473bfc000)]
      0x00007ff3e4006800 JavaThread "NettyClientPublicExecutor_14" [_thread_blocked, id=5527, stack(0x00007ff473bfc000,0x00007ff473cfd000)]
      0x00007ff3ec008800 JavaThread "NettyClientPublicExecutor_13" [_thread_blocked, id=5526, stack(0x00007ff473cfd000,0x00007ff473dfe000)]
      0x00007ff3f00e1800 JavaThread "NettyClientPublicExecutor_12" [_thread_blocked, id=5525, stack(0x00007ff473dfe000,0x00007ff473eff000)]
      0x00007ff3dc00b800 JavaThread "NettyClientPublicExecutor_11" [_thread_blocked, id=5524, stack(0x00007ff473eff000,0x00007ff474000000)]
      0x00007ff3dc00a000 JavaThread "NettyClientPublicExecutor_10" [_thread_blocked, id=5523, stack(0x00007ff4780ee000,0x00007ff4781ef000)]
      0x00007ff3e4004800 JavaThread "NettyClientPublicExecutor_9" [_thread_blocked, id=5522, stack(0x00007ff4781ef000,0x00007ff4782f0000)]
      0x00007ff3ec006800 JavaThread "NettyClientPublicExecutor_8" [_thread_blocked, id=5521, stack(0x00007ff4782f0000,0x00007ff4783f1000)]
      0x00007ff3f00e0800 JavaThread "NettyClientPublicExecutor_7" [_thread_blocked, id=5520, stack(0x00007ff4783f1000,0x00007ff4784f2000)]
      0x00007ff3e4003000 JavaThread "NettyClientPublicExecutor_6" [_thread_blocked, id=5519, stack(0x00007ff4784f2000,0x00007ff4785f3000)]
      0x00007ff3ec005000 JavaThread "NettyClientPublicExecutor_5" [_thread_blocked, id=5518, stack(0x00007ff4785f3000,0x00007ff4786f4000)]
      0x00007ff3f0001800 JavaThread "NettyClientPublicExecutor_4" [_thread_blocked, id=5517, stack(0x00007ff4786f4000,0x00007ff4787f5000)]
      0x00007ff3dc001000 JavaThread "NettyClientPublicExecutor_3" [_thread_blocked, id=5516, stack(0x00007ff4787f5000,0x00007ff4788f6000)]
      0x00007ff3fc030000 JavaThread "NettyClientWorkerThread_4" [_thread_blocked, id=5515, stack(0x00007ff4798f7000,0x00007ff4799f8000)]
      0x00007ff3e4001000 JavaThread "NettyClientPublicExecutor_2" [_thread_blocked, id=5514, stack(0x00007ff4799f8000,0x00007ff479af9000)]
      0x00007ff3fc02e800 JavaThread "NettyClientWorkerThread_3" [_thread_blocked, id=5513, stack(0x00007ff47aafa000,0x00007ff47abfb000)]
      0x00007ff3ec003000 JavaThread "NettyClientPublicExecutor_1" [_thread_blocked, id=5512, stack(0x00007ff47abfb000,0x00007ff47acfc000)]
      0x00007ff3fc008000 JavaThread "NettyClientWorkerThread_2" [_thread_blocked, id=5511, stack(0x00007ff47bcfd000,0x00007ff47bdfe000)]
      0x00007ff4e4415800 JavaThread "RebalanceService" [_thread_blocked, id=5510, stack(0x00007ff47bdfe000,0x00007ff47beff000)]
      0x00007ff4e5c08000 JavaThread "PullMessageService" [_thread_blocked, id=5509, stack(0x00007ff47beff000,0x00007ff47c000000)]
      0x00007ff3fc016800 JavaThread "NettyClientWorkerThread_1" [_thread_blocked, id=5495, stack(0x00007ff4822d8000,0x00007ff4823d9000)]
      0x00007ff4e5c05000 JavaThread "NettyClientSelector_1" [_thread_blocked, id=5494, stack(0x00007ff4823d9000,0x00007ff4824da000)]
      0x00007ff4e5b17000 JavaThread "MQClientFactoryScheduledThread" [_thread_blocked, id=5493, stack(0x00007ff4824da000,0x00007ff4825db000)]
      0x00007ff4e52f8000 JavaThread "ClientHouseKeepingService" daemon [_thread_blocked, id=5492, stack(0x00007ff4825db000,0x00007ff4826dc000)]
      0x00007ff4e57c2000 JavaThread "container-0" [_thread_blocked, id=5485, stack(0x00007ff4830dc000,0x00007ff4831dd000)]
      0x00007ff408003800 JavaThread "Catalina-utility-2" [_thread_blocked, id=5484, stack(0x00007ff4831dd000,0x00007ff4832de000)]
      0x00007ff4e57b4000 JavaThread "Catalina-utility-1" [_thread_blocked, id=5483, stack(0x00007ff4832de000,0x00007ff4833df000)]
      0x00007ff4e4dc6000 JavaThread "logback-1" daemon [_thread_blocked, id=5465, stack(0x00007ff489354000,0x00007ff489455000)]
      0x00007ff4e42de000 JavaThread "Service Thread" daemon [_thread_blocked, id=5458, stack(0x00007ff48a579000,0x00007ff48a67a000)]
      0x00007ff4e42d3000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=5457, stack(0x00007ff48a67b000,0x00007ff48a77b000)]
      0x00007ff4e42d1000 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=5456, stack(0x00007ff48a77c000,0x00007ff48a87c000)]
      0x00007ff4e42cf000 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=5455, stack(0x00007ff48a87d000,0x00007ff48a97d000)]
      0x00007ff4e42cc800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=5454, stack(0x00007ff48a97e000,0x00007ff48aa7e000)]
      0x00007ff4e42ca800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=5453, stack(0x00007ff48cacc000,0x00007ff48cbcc000)]
      0x00007ff4e42c8800 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5452, stack(0x00007ff48cbcd000,0x00007ff48cccd000)]
      0x00007ff4e42c6000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=5451, stack(0x00007ff48ccce000,0x00007ff48cdce000)]
      0x00007ff4e42bc000 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5450, stack(0x00007ff48cdcf000,0x00007ff48cecf000)]
      0x00007ff4e42ba000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=5449, stack(0x00007ff48ced0000,0x00007ff48cfd0000)]
      0x00007ff4e42b7800 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=5448, stack(0x00007ff48cfd1000,0x00007ff48d0d1000)]
      0x00007ff4e42b6000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=5447, stack(0x00007ff48d0d2000,0x00007ff48d1d2000)]
      0x00007ff4e42b3000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=5446, stack(0x00007ff48d1d3000,0x00007ff48d2d3000)]
      0x00007ff4e42b1000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5445, stack(0x00007ff48d2d3000,0x00007ff48d3d4000)]
      0x00007ff4e4280000 JavaThread "Finalizer" daemon [_thread_blocked, id=5444, stack(0x00007ff48d3d4000,0x00007ff48d4d5000)]
      0x00007ff4e427b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=5443, stack(0x00007ff48d4d5000,0x00007ff48d5d6000)]
    #3.2.其它线程。这些事JVM内部线程
    #1.虚拟机线程VMThread（这个线程5442就是导致程序奔溃的线程）；
    #2.WatcherThread：jvm周期性任务调度的线程
    Other Threads:
    =>0x00007ff4e4272000 VMThread [stack: 0x00007ff48d5d7000,0x00007ff48d6d7000] [id=5442]
      0x00007ff4e42e1000 WatcherThread [stack: 0x00007ff48a479000,0x00007ff48a579000] [id=5459]
    
    #4.程序奔溃时虚拟机状态信息：at safepoint表示所有线程都因为虚拟机等待状态而阻塞，等待一个虚拟机操作完成。这里是因为在垃圾回收，所以会阻塞等待垃圾回收完成。
    VM state:at safepoint (normal execution) 
    
    #5.程序奔溃时锁信息 
    VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
    [0x00007ff4e4006770] ExpandHeap_lock - owner thread: 0x00007ff4e4272000
    [0x00007ff4e4006df0] Threads_lock - owner thread: 0x00007ff4e4272000
    [0x00007ff4e40072f0] Heap_lock - owner thread: 0x00007fefa1290000
    
    heap address: 0x00000003d3c00000, size: 16068 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
    Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
    Compressed class space size: 1073741824 Address: 0x00000007c0000000
    #6.程序奔溃时堆信息，内存的详细使用情况
    Heap:
     PSYoungGen      total 4890112K, used 143712K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000) #年轻代共4G，用了1.1G
      eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
      from space 198656K, 72% used [0x000000079bd80000,0x00000007a49d8120,0x00000007a7f80000)
      to   space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
     ParOldGen       total 5232640K, used 5232632K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000) #老年代5G 用了5G
      object space 5232640K, 99% used [0x00000003d3c00000,0x00000005131fe1b8,0x0000000513200000) #空间使用99%
     Metaspace       used 101388K, capacity 106705K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13051K, committed 13312K, reserved 1048576K
    
    Card table byte_map: [0x00007ff4d2470000,0x00007ff4d43d3000] byte_map_base: 0x00007ff4d05d2000
    
    Marking Bits: (ParMarkBitMap*) 0x00007ff4e9623fe0
     Begin Bits: [0x00007ff4949e0000, 0x00007ff4a44f0000)
     End Bits:   [0x00007ff4a44f0000, 0x00007ff4b4000000)
    
    Polling page: 0x00007ff4ea266000
    #7.本地代码缓存。这是一块用于编译和保存本地代码的内存；注意是本地代码，它和Metaspace（元空间）是不一样的，元空间是用来存放jvm和java类的元数据的。
    CodeCache: size=245760Kb used=73103Kb max_used=74041Kb free=172656Kb
     bounds [0x00007ff4d4793000, 0x00007ff4d90d3000, 0x00007ff4e3793000]
     total_blobs=19194 nmethods=18292 adapters=810
     compilation: enabled
    #8.编译事件
    Compilation events (10 events):
    Event: 260825.612 Thread 0x00007ff4e42d1000 nmethod 25775 0x00007ff4d4e2c810 code [0x00007ff4d4e2c9a0, 0x00007ff4d4e2cca8]
    Event: 260825.613 Thread 0x00007ff4e42d3000 nmethod 25774 0x00007ff4d6bb1ed0 code [0x00007ff4d6bb20c0, 0x00007ff4d6bb2898]
    Event: 260865.508 Thread 0x00007ff4e42cc800 25776       3       sun.misc.Cleaner::create (19 bytes)
    Event: 260865.508 Thread 0x00007ff4e42d1000 25778  s    3       sun.misc.Cleaner::add (26 bytes)
    Event: 260865.508 Thread 0x00007ff4e42d1000 nmethod 25778 0x00007ff4d4e2c250 code [0x00007ff4d4e2c3c0, 0x00007ff4d4e2c710]
    Event: 260865.509 Thread 0x00007ff4e42cc800 nmethod 25776 0x00007ff4d6c90590 code [0x00007ff4d6c90740, 0x00007ff4d6c90df0]
    Event: 260865.509 Thread 0x00007ff4e42cc800 25777       3       sun.misc.Cleaner::<init> (24 bytes)
    Event: 260865.509 Thread 0x00007ff4e42cc800 nmethod 25777 0x00007ff4d52874d0 code [0x00007ff4d5287640, 0x00007ff4d5287950]
    Event: 261186.686 Thread 0x00007ff4e42b6000 25779       4       java.net.SocketTimeoutException::<init> (6 bytes)
    Event: 261186.689 Thread 0x00007ff4e42b6000 nmethod 25779 0x00007ff4d6c67ed0 code [0x00007ff4d6c68040, 0x00007ff4d6c683a8]
    #9.程序奔溃时最近几次的GC垃圾回收相关记录：10次fullGC，eden取全部回收，OldGen回收后还有98%-99%是被占用的，而且full GC出现的频率很高，可能有内存泄漏的情况
    GC Heap History (10 events):
    Event: 260986.480 GC heap after
    Heap after GC invocations=5466 (full 6):
     PSYoungGen      total 4808704K, used 189670K [0x0000000671400000, 0x00000007a2d80000, 0x00000007c0000000)
      eden space 4613632K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078ad80000)
      from space 195072K, 97% used [0x0000000796d00000,0x00000007a26398f0,0x00000007a2b80000)
      to   space 196096K, 0% used [0x000000078ad80000,0x000000078ad80000,0x0000000796d00000)
     ParOldGen       total 5232640K, used 5167250K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 98% used  [0x00000003d3c00000,0x000000050f2248d0,0x0000000513200000) #第5466次调用fullGC后，内存使用98%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    }
    Event: 261046.853 GC heap before
    { Heap before GC invocations=5467 (full 6):
     PSYoungGen      total 4808704K, used 4803302K [0x0000000671400000, 0x00000007a2d80000, 0x00000007c0000000)
      eden space 4613632K, 100% used [0x0000000671400000,0x000000078ad80000,0x000000078ad80000)
      from space 195072K, 97% used [0x0000000796d00000,0x00000007a26398f0,0x00000007a2b80000)
      to   space 196096K, 0% used [0x000000078ad80000,0x000000078ad80000,0x0000000796d00000)
     ParOldGen       total 5232640K, used 5167250K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 98% used  [0x00000003d3c00000,0x000000050f2248d0,0x0000000513200000) #第5467次调用fullGC前，内存使用98%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    Event: 261046.916 GC heap after
    Heap after GC invocations=5467 (full 6):
     PSYoungGen      total 4809728K, used 189466K [0x0000000671400000, 0x00000007a5600000, 0x00000007c0000000)
      eden space 4613632K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078ad80000)
      from space 196096K, 96% used [0x000000078ad80000,0x00000007966868c0,0x0000000796d00000)
      to   space 197120K, 0% used [0x0000000799580000,0x0000000799580000,0x00000007a5600000)
     ParOldGen       total 5232640K, used 5181087K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used  [0x00000003d3c00000,0x000000050ffa7d38,0x0000000513200000) #第5467次调用fullGC后，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    }
    Event: 261107.680 GC heap before
    { Heap before GC invocations=5468 (full 6):
     PSYoungGen      total 4809728K, used 4803098K [0x0000000671400000, 0x00000007a5600000, 0x00000007c0000000)
      eden space 4613632K, 100% used [0x0000000671400000,0x000000078ad80000,0x000000078ad80000)
      from space 196096K, 96% used [0x000000078ad80000,0x00000007966868c0,0x0000000796d00000)
      to   space 197120K, 0% used [0x0000000799580000,0x0000000799580000,0x00000007a5600000)
     ParOldGen       total 5232640K, used 5181087K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x000000050ffa7d38,0x0000000513200000)#第5468次调用fullGC前，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    Event: 261107.740 GC heap after
    Heap after GC invocations=5468 (full 6):
     PSYoungGen      total 4851200K, used 187546K [0x0000000671400000, 0x00000007a5880000, 0x00000007c0000000)
      eden space 4654080K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078d500000)
      from space 197120K, 95% used [0x0000000799580000,0x00000007a4ca6af0,0x00000007a5600000)
      to   space 197120K, 0% used [0x000000078d500000,0x000000078d500000,0x0000000799580000)
     ParOldGen       total 5232640K, used 5196567K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x0000000510ec5cd0,0x0000000513200000) #第5468次调用fullGC后，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    }
    Event: 261187.245 GC heap before
    { Heap before GC invocations=5469 (full 6):
     PSYoungGen      total 4851200K, used 4841626K [0x0000000671400000, 0x00000007a5880000, 0x00000007c0000000)
      eden space 4654080K, 100% used [0x0000000671400000,0x000000078d500000,0x000000078d500000)
      from space 197120K, 95% used [0x0000000799580000,0x00000007a4ca6af0,0x00000007a5600000)
      to   space 197120K, 0% used [0x000000078d500000,0x000000078d500000,0x0000000799580000)
     ParOldGen       total 5232640K, used 5196567K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used  [0x00000003d3c00000,0x0000000510ec5cd0,0x0000000513200000) #第5469次调用fullGC前，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    Event: 261187.288 GC heap after
    Heap after GC invocations=5469 (full 6):
     PSYoungGen      total 4851200K, used 191201K [0x0000000671400000, 0x00000007a7f80000, 0x00000007c0000000)
      eden space 4654080K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078d500000)
      from space 197120K, 96% used [0x000000078d500000,0x0000000798fb87b0,0x0000000799580000)
      to   space 198656K, 0% used [0x000000079bd80000,0x000000079bd80000,0x00000007a7f80000)
     ParOldGen       total 5232640K, used 5210187K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x0000000511c12da0,0x0000000513200000)#第5469次调用fullGC后，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    }
    Event: 261248.905 GC heap before
    { Heap before GC invocations=5470 (full 6):
     PSYoungGen      total 4851200K, used 4845281K [0x0000000671400000, 0x00000007a7f80000, 0x00000007c0000000)
      eden space 4654080K, 100% used [0x0000000671400000,0x000000078d500000,0x000000078d500000)
      from space 197120K, 96% used [0x000000078d500000,0x0000000798fb87b0,0x0000000799580000)
      to   space 198656K, 0% used [0x000000079bd80000,0x000000079bd80000,0x00000007a7f80000)
     ParOldGen       total 5232640K, used 5210187K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x0000000511c12da0,0x0000000513200000) #第5470次调用fullGC前，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    Event: 261248.948 GC heap after
    Heap after GC invocations=5470 (full 6):
     PSYoungGen      total 4890112K, used 194517K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000)
      eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
      from space 198656K, 97% used [0x000000079bd80000,0x00000007a7b75650,0x00000007a7f80000)
      to   space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
     ParOldGen       total 5232640K, used 5218201K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x00000005123e65a0,0x0000000513200000) #第5470次调用fullGC后，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    }
    Event: 261248.948 GC heap before
    { Heap before GC invocations=5471 (full 7):
     PSYoungGen      total 4890112K, used 194517K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000)
      eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
      from space 198656K, 97% used [0x000000079bd80000,0x00000007a7b75650,0x00000007a7f80000)
      to   space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
     ParOldGen       total 5232640K, used 5218201K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
      object space 5232640K, 99% used [0x00000003d3c00000,0x00000005123e65a0,0x0000000513200000) #第5471次调用fullGC前，内存使用99%
     Metaspace       used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
      class space    used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
    
    #10.程序奔溃时非优化事件
    Deoptimization events (10 events):
    Event: 183256.499 Thread 0x00007ff4e5b17000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d8ead114 method=ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy.isTriggeringEvent(Ljava/io/File;Ljava/lang/Object;)Z @ 31
    Event: 188056.499 Thread 0x00007ff3fc016800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d8d1db04 method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; @ 147
    Event: 189591.336 Thread 0x00007ff38c391000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d7f684cc method=io.netty.channel.epoll.EpollEventLoop.epollWait(Z)I @ 29
    Event: 212562.434 Thread 0x00007ff38c3a9800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
    Event: 213358.334 Thread 0x00007ff38c39d800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
    Event: 213475.092 Thread 0x00007ff38c39d800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
    Event: 214383.434 Thread 0x00007ff38c392800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
    Event: 215716.499 Thread 0x00007ff4e5c05000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d76096e0 method=io.netty.channel.nio.NioEventLoop.select(Z)V @ 122
    Event: 238192.333 Thread 0x00007ff38c396000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d7f684cc method=io.netty.channel.epoll.EpollEventLoop.epollWait(Z)I @ 29
    Event: 238605.046 Thread 0x00007ff37c1a4800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d5882778 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 6
    
    Classes redefined (0 events):
    No events
    #11.程序奔溃时内部异常。可以看到主要是socket超时异常
    Internal exceptions (10 events):
    Event: 260986.649 Thread 0x00007ff288025000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000067aab6e98) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261011.428 Thread 0x00007ff28800d800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x0000000702733ab0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261016.425 Thread 0x00007ff288018000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x0000000703e66e30) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261096.428 Thread 0x00007ff288018000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000073f56ef88) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261101.368 Thread 0x00007ff28801e800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000073fe3e198) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261126.806 Thread 0x00007ff288009800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a9b43330) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261131.428 Thread 0x00007ff288014800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006ce14bd38) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261186.685 Thread 0x00007ff288025000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000077f1c1780) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261191.457 Thread 0x00007ff288007000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a3c88db8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    Event: 261191.720 Thread 0x00007ff288007000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a3cc1310) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
    
    #12.程序奔溃时，最近10个事件。最后一个是JVM内存分配失败
    Events (10 events):
    Event: 261248.889 Thread 0x00007ff3237a2800 Thread exited: 0x00007ff3237a2800
    Event: 261248.889 Thread 0x00007ff306e4d000 Thread exited: 0x00007ff306e4d000
    Event: 261248.891 Thread 0x00007ff332d49800 Thread exited: 0x00007ff332d49800
    Event: 261248.894 Thread 0x00007fefa1290000 Thread exited: 0x00007fefa1290000
    Event: 261248.904 Thread 0x00007fefa1290000 Thread added: 0x00007fefa1290000
    Event: 261248.904 Thread 0x00007ff30204d000 Thread added: 0x00007ff30204d000
    Event: 261248.904 Thread 0x00007ff2ffb8b000 Thread added: 0x00007ff2ffb8b000
    Event: 261248.904 Thread 0x00007ff306e4d000 Thread added: 0x00007ff306e4d000
    Event: 261248.904 Thread 0x00007ff1ba676800 Thread added: 0x00007ff1ba676800
    Event: 261248.904 Executing VM operation: ParallelGCFailedAllocation
    
    #13.动态库信息。这些信息是虚拟机奔溃时的虚拟内存列表区域。它可以告诉奔溃时哪些类库正在被使用，位置在哪里，还有堆栈和守护等信息
    Dynamic libraries:
    3d3c00000-513200000 rw-p 00000000 00:00 0 
    5ca700000-671400000 ---p 00000000 00:00 0 
    671400000-7a8300000 rw-p 00000000 00:00 0 
    7a8300000-7c0000000 ---p 00000000 00:00 0 
    7c0000000-7c0d00000 rw-p 00000000 00:00 0 
    7c0d00000-800000000 ---p 00000000 00:00 0 
    562a3e8a8000-562a3e8a9000 r-xp 00000000 fd:01 920956                     /usr/java/jdk1.8.0_251/bin/java
    562a3eaa8000-562a3eaa9000 r--p 00000000 fd:01 920956                     /usr/java/jdk1.8.0_251/bin/java
    562a3eaa9000-562a3eaaa000 rw-p 00001000 fd:01 920956                     /usr/java/jdk1.8.0_251/bin/java
    562a40a50000-562a43108000 rw-p 00000000 00:00 0                          [heap]
    7fec54000000-7fec5778e000 rw-p 00000000 00:00 0 
    7ff28229d000-7ff2822a0000 ---p 00000000 00:00 0 
    7ff2822a0000-7ff2823de000 rw-p 00000000 00:00 0 
    7ff2823de000-7ff28259e000 ---p 00000000 00:00 0 
    7ff28259e000-7ff28279e000 rw-p 00000000 00:00 0 
    7ff28279e000-7ff2827d6000 r-xp 00000000 fd:01 919143                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
    7ff2827d6000-7ff2829d5000 ---p 00038000 fd:01 919143                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
    7ff2829d5000-7ff2829da000 r--p 00037000 fd:01 919143                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
    7ff2829da000-7ff2829dc000 rw-p 0003c000 fd:01 919143                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
    7ff2829dc000-7ff282bdc000 rw-p 00000000 00:00 0 
    7ff282bdc000-7ff282bdf000 ---p 00000000 00:00 0 
    7ff282bdf000-7ff282cdd000 rw-p 00000000 00:00 0 
    7ff282cdd000-7ff282ce0000 ---p 00000000 00:00 0 
    7ff282ce0000-7ff282dde000 rw-p 00000000 00:00 0 
    7ff282dde000-7ff282de1000 ---p 00000000 00:00 0 
    7ff282de1000-7ff282edf000 rw-p 00000000 00:00 0  
    7ff285b09000-7ff285b0c000 ---p 00000000 00:00 0 
    7ff285b0c000-7ff285e0a000 rw-p 00000000 00:00 0 
    7ff285e0a000-7ff285e0d000 ---p 00000000 00:00 0 
    7ff285e0d000-7ff285f0b000 rw-p 00000000 00:00 0 
    7ff285f0b000-7ff285f20000 r-xp 00000000 fd:01 3937                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
    7ff285f20000-7ff28611f000 ---p 00015000 fd:01 3937                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
    7ff28611f000-7ff286120000 r--p 00014000 fd:01 3937                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
    7ff286120000-7ff286121000 rw-p 00015000 fd:01 3937                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
    7ff286121000-7ff28620a000 r-xp 00000000 fd:01 8779                       /usr/lib64/libstdc++.so.6.0.19
    7ff28620a000-7ff28640a000 ---p 000e9000 fd:01 8779                       /usr/lib64/libstdc++.so.6.0.19
    7ff28640a000-7ff286412000 r--p 000e9000 fd:01 8779                       /usr/lib64/libstdc++.so.6.0.19
    7ff286412000-7ff286414000 rw-p 000f1000 fd:01 8779                       /usr/lib64/libstdc++.so.6.0.19
    7ff286414000-7ff286429000 rw-p 00000000 00:00 0 
    7ff286429000-7ff287b3b000 r-xp 00000000 fd:10 1075364526                 /data/cplex/libcplex1260remotejni.so
    7ff287b3b000-7ff287d3b000 ---p 01712000 fd:10 1075364526                 /data/cplex/libcplex1260remotejni.so
    7ff287d3b000-7ff287d3c000 r--p 01712000 fd:10 1075364526                 /data/cplex/libcplex1260remotejni.so
    7ff287d3c000-7ff287ef6000 rw-p 01713000 fd:10 1075364526                 /data/cplex/libcplex1260remotejni.so
    7ff287ef6000-7ff287eff000 rw-p 00000000 00:00 0 
    7ff287eff000-7ff287f02000 ---p 00000000 00:00 0 
    7ff4549be000-7ff455abd000 rw-p 00000000 00:00 0 
    7ff455abd000-7ff45c000000 r--p 00000000 fd:01 4236                       /usr/lib/locale/locale-archive
    7ff45c000000-7ff45fffe000 rw-p 00000000 00:00 0 
    7ff45fffe000-7ff460000000 ---p 00000000 00:00 0 
    7ff460000000-7ff464000000 rw-p 00000000 00:00 0 
    7ff471cd5000-7ff471cd8000 ---p 00000000 00:00 0 
    7ff471cd8000-7ff471fd6000 rw-p 00000000 00:00 0 
    7ff471fd6000-7ff471fe3000 r-xp 00000000 fd:01 20664                      /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
    7ff471fe3000-7ff4721e2000 ---p 0000d000 fd:01 20664                      /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
    7ff4721e2000-7ff4721e4000 rw-p 0000c000 fd:01 20664                      /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
    7ff4721e4000-7ff4723e4000 rw-p 00000000 00:00 0 
    7ff4723e4000-7ff4723e7000 ---p 00000000 00:00 0 
    7ff4837df000-7ff4839df000 rw-p 00000000 00:00 0 
    7ff4839df000-7ff4839f5000 r-xp 00000000 fd:01 7744                       /usr/lib64/libresolv-2.17.so
    7ff4839f5000-7ff483bf5000 ---p 00016000 fd:01 7744                       /usr/lib64/libresolv-2.17.so
    7ff483bf5000-7ff483bf6000 r--p 00016000 fd:01 7744                       /usr/lib64/libresolv-2.17.so
    7ff483bf6000-7ff483bf7000 rw-p 00017000 fd:01 7744                       /usr/lib64/libresolv-2.17.so
    7ff483bf7000-7ff483bf9000 rw-p 00000000 00:00 0 
    7ff483bf9000-7ff483bff000 r-xp 00000000 fd:01 3925                       /usr/lib64/libnss_dns-2.17.so
    7ff483bff000-7ff483dfe000 ---p 00006000 fd:01 3925                       /usr/lib64/libnss_dns-2.17.so
    7ff483dfe000-7ff483dff000 r--p 00005000 fd:01 3925                       /usr/lib64/libnss_dns-2.17.so
    7ff483dff000-7ff483e00000 rw-p 00006000 fd:01 3925                       /usr/lib64/libnss_dns-2.17.so
    7ff483e00000-7ff484000000 rw-p 00000000 00:00 0 
    7ff484000000-7ff487bee000 rw-p 00000000 00:00 0 
    7ff487bee000-7ff488000000 ---p 00000000 00:00 0 
    7ff4880e7000-7ff4882e7000 rw-p 00000000 00:00 0 
    7ff4882e7000-7ff4884e7000 rw-p 00000000 00:00 0 
    7ff4884e7000-7ff4886e7000 rw-p 00000000 00:00 0 
    7ff4886e7000-7ff4888e7000 rw-p 00000000 00:00 0 
    7ff4888e7000-7ff4888f8000 r-xp 00000000 fd:01 919104                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
    7ff4888f8000-7ff488af7000 ---p 00011000 fd:01 919104                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
    7ff488af7000-7ff488af8000 r--p 00010000 fd:01 919104                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
    7ff488af8000-7ff488af9000 rw-p 00011000 fd:01 919104                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
    7ff488af9000-7ff488cf9000 rw-p 00000000 00:00 0 
    7ff488cf9000-7ff488ef9000 rw-p 00000000 00:00 0 
    7ff488f54000-7ff489154000 rw-p 00000000 00:00 0 
    7ff489154000-7ff489354000 rw-p 00000000 00:00 0 
    7ff489354000-7ff489357000 ---p 00000000 00:00 0 
    7ff489357000-7ff489655000 rw-p 00000000 00:00 0 
    7ff489655000-7ff489956000 rw-p 00000000 00:00 0 
    7ff489956000-7ff48996c000 r-xp 00000000 fd:01 919167                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
    7ff48996c000-7ff489b6b000 ---p 00016000 fd:01 919167                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
    7ff489b6b000-7ff489b6c000 r--p 00015000 fd:01 919167                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
    7ff489b6c000-7ff489b6d000 rw-p 00016000 fd:01 919167                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
    7ff489b6d000-7ff489b76000 r-xp 00000000 fd:01 919116                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
    7ff489b76000-7ff489d75000 ---p 00009000 fd:01 919116                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
    7ff489d75000-7ff489d76000 r--p 00008000 fd:01 919116                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
    7ff489d76000-7ff489d77000 rw-p 00009000 fd:01 919116                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
    7ff489d77000-7ff489f77000 rw-p 00000000 00:00 0 
    7ff493a02000-7ff494000000 ---p 00000000 00:00 0 
    7ff494049000-7ff4940df000 r--s 0110d000 fd:01 918937                     /usr/java/jdk1.8.0_251/jre/lib/ext/jfxrt.jar
    7ff4940df000-7ff4b4000000 rw-p 00000000 00:00 0 
    7ff4ced5c000-7ff4d0000000 ---p 00000000 00:00 0 
    7ff4d0013000-7ff4d0063000 rw-p 00000000 00:00 0 
    7ff4d0063000-7ff4d023e000 r--s 03dbf000 fd:01 919174                     /usr/java/jdk1.8.0_251/jre/lib/rt.jar
    7ff4d023e000-7ff4d0378000 rw-p 00000000 00:00 0 
    7ff4d90d3000-7ff4e3793000 ---p 00000000 00:00 0 
    7ff4e3793000-7ff4e37ae000 r-xp 00000000 fd:01 919135                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
    7ff4e37ae000-7ff4e39ad000 ---p 0001b000 fd:01 919135                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
    7ff4e39ad000-7ff4e39ae000 r--p 0001a000 fd:01 919135                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
    7ff4e39ae000-7ff4e39af000 rw-p 0001b000 fd:01 919135                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
    7ff4e39af000-7ff4e39bb000 r-xp 00000000 fd:01 5760                       /usr/lib64/libnss_files-2.17.so
    7ff4e39bb000-7ff4e3bba000 ---p 0000c000 fd:01 5760                       /usr/lib64/libnss_files-2.17.so
    7ff4e3bba000-7ff4e3bbb000 r--p 0000b000 fd:01 5760                       /usr/lib64/libnss_files-2.17.so
    7ff4e3bbb000-7ff4e3bbc000 rw-p 0000c000 fd:01 5760                       /usr/lib64/libnss_files-2.17.so
    7ff4e3bbc000-7ff4e3bc2000 rw-p 00000000 00:00 0 
    7ff4e3bc2000-7ff4e3bee000 r-xp 00000000 fd:01 919146                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
    7ff4e3bee000-7ff4e3dee000 ---p 0002c000 fd:01 919146                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
    7ff4e3dee000-7ff4e3def000 r--p 0002c000 fd:01 919146                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
    7ff4e3def000-7ff4e3df1000 rw-p 0002d000 fd:01 919146                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
    7ff4e3df1000-7ff4e3dfe000 r-xp 00000000 fd:01 919147                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
    7ff4e3dfe000-7ff4e3ffd000 ---p 0000d000 fd:01 919147                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
    7ff4e3ffd000-7ff4e3fff000 r--p 0000c000 fd:01 919147                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
    7ff4e3fff000-7ff4e4000000 rw-p 0000e000 fd:01 919147                     /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
    7ff4e4000000-7ff4e8000000 rw-p 00000000 00:00 0 
    7ff4e8005000-7ff4e800a000 r--s 00040000 fd:01 918942                     /usr/java/jdk1.8.0_251/jre/lib/ext/sunjce_provider.jar
    7ff4e800a000-7ff4e800e000 r--s 00041000 fd:01 918941                     /usr/java/jdk1.8.0_251/jre/lib/ext/sunpkcs11.jar
    7ff4e800e000-7ff4e8011000 r--s 0001a000 fd:01 919204                     /usr/java/jdk1.8.0_251/jre/lib/jce.jar
    7ff4e8011000-7ff4e8024000 r--s 0034b000 fd:01 919049                     /usr/java/jdk1.8.0_251/jre/lib/resources.jar
    7ff4e8024000-7ff4e8029000 r--s 000a8000 fd:01 918933                     /usr/java/jdk1.8.0_251/jre/lib/jsse.jar
    7ff4e8029000-7ff4e8045000 r--s 00393000 fd:01 918936                     /usr/java/jdk1.8.0_251/jre/lib/ext/cldrdata.jar
    7ff4e8045000-7ff4e8046000 ---p 00000000 00:00 0 
    7ff4e8046000-7ff4e8146000 rw-p 00000000 00:00 0 
    7ff4e8146000-7ff4e814d000 r-xp 00000000 fd:01 7747                       /usr/lib64/librt-2.17.so
    7ff4e814d000-7ff4e834c000 ---p 00007000 fd:01 7747                       /usr/lib64/librt-2.17.so
    7ff4e834c000-7ff4e834d000 r--p 00006000 fd:01 7747                       /usr/lib64/librt-2.17.so
    7ff4e834d000-7ff4e834e000 rw-p 00007000 fd:01 7747                       /usr/lib64/librt-2.17.so
    7ff4e834e000-7ff4e844f000 r-xp 00000000 fd:01 3938                       /usr/lib64/libm-2.17.so
    7ff4e844f000-7ff4e864e000 ---p 00101000 fd:01 3938                       /usr/lib64/libm-2.17.so
    7ff4e864e000-7ff4e864f000 r--p 00100000 fd:01 3938                       /usr/lib64/libm-2.17.so
    7ff4e864f000-7ff4e8650000 rw-p 00101000 fd:01 3938                       /usr/lib64/libm-2.17.so
    7ff4e8650000-7ff4e9340000 r-xp 00000000 fd:01 919113                     /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
    7ff4e9340000-7ff4e953f000 ---p 00cf0000 fd:01 919113                     /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
    7ff4e953f000-7ff4e95d5000 r--p 00cef000 fd:01 919113                     /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
    7ff4e95d5000-7ff4e9606000 rw-p 00d85000 fd:01 919113                     /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
    7ff4e9606000-7ff4e9641000 rw-p 00000000 00:00 0 
    7ff4e9641000-7ff4e9805000 r-xp 00000000 fd:01 3908                       /usr/lib64/libc-2.17.so
    7ff4e9805000-7ff4e9a04000 ---p 001c4000 fd:01 3908                       /usr/lib64/libc-2.17.so
    7ff4e9a04000-7ff4e9a08000 r--p 001c3000 fd:01 3908                       /usr/lib64/libc-2.17.so
    7ff4e9a08000-7ff4e9a0a000 rw-p 001c7000 fd:01 3908                       /usr/lib64/libc-2.17.so
    7ff4e9a0a000-7ff4e9a0f000 rw-p 00000000 00:00 0 
    7ff4e9a0f000-7ff4e9a11000 r-xp 00000000 fd:01 3928                       /usr/lib64/libdl-2.17.so
    7ff4e9a11000-7ff4e9c11000 ---p 00002000 fd:01 3928                       /usr/lib64/libdl-2.17.so
    7ff4e9c11000-7ff4e9c12000 r--p 00002000 fd:01 3928                       /usr/lib64/libdl-2.17.so
    7ff4e9c12000-7ff4e9c13000 rw-p 00003000 fd:01 3928                       /usr/lib64/libdl-2.17.so
    7ff4e9c13000-7ff4e9c2a000 r-xp 00000000 fd:01 920384                     /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
    7ff4e9c2a000-7ff4e9e29000 ---p 00017000 fd:01 920384                     /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
    7ff4e9e29000-7ff4e9e2a000 r--p 00016000 fd:01 920384                     /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
    7ff4e9e2a000-7ff4e9e2b000 rw-p 00017000 fd:01 920384                     /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
    7ff4e9e2b000-7ff4e9e42000 r-xp 00000000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
    7ff4e9e42000-7ff4ea041000 ---p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
    7ff4ea041000-7ff4ea042000 r--p 00016000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
    7ff4ea042000-7ff4ea043000 rw-p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
    7ff4ea043000-7ff4ea047000 rw-p 00000000 00:00 0 
    7ff4ea047000-7ff4ea069000 r-xp 00000000 fd:01 4297                       /usr/lib64/ld-2.17.so
    7ff4ea069000-7ff4ea06a000 rw-p 00000000 00:00 0 
    7ff4ea06a000-7ff4ea074000 r--s 00116000 fd:01 918946                     /usr/java/jdk1.8.0_251/jre/lib/ext/localedata.jar
    7ff4ea074000-7ff4ea080000 r--s 04612000 fd:10 1132777118                 /data/Projects/pick-task-assign-job/app.jar
    7ff4ea080000-7ff4ea151000 rw-p 00000000 00:00 0 
    7ff4ea151000-7ff4ea159000 rw-s 00000000 fd:01 266446                     /tmp/hsperfdata_ops/5421
    7ff4ea159000-7ff4ea15d000 ---p 00000000 00:00 0 
    7ff4ea15d000-7ff4ea25e000 rw-p 00000000 00:00 0 
    7ff4ea25e000-7ff4ea260000 r--s 0000d000 fd:01 918944                     /usr/java/jdk1.8.0_251/jre/lib/ext/sunec.jar
    7ff4ea260000-7ff4ea265000 r--s 002f9000 fd:01 919048                     /usr/java/jdk1.8.0_251/jre/lib/charsets.jar
    7ff4ea265000-7ff4ea266000 rw-p 00000000 00:00 0 
    7ff4ea266000-7ff4ea267000 ---p 00000000 00:00 0 
    7ff4ea267000-7ff4ea268000 rw-p 00000000 00:00 0 
    7ff4ea268000-7ff4ea269000 r--p 00021000 fd:01 4297                       /usr/lib64/ld-2.17.so
    7ff4ea269000-7ff4ea26a000 rw-p 00022000 fd:01 4297                       /usr/lib64/ld-2.17.so
    7ff4ea26a000-7ff4ea26b000 rw-p 00000000 00:00 0 
    7ffc4c2fa000-7ffc4c31b000 rw-p 00000000 00:00 0                          [stack]
    7ffc4c3e0000-7ffc4c3e2000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
    
    #14.程序启动时指定的参数信息
    VM Arguments:
    jvm_args: -Dspring.profiles.active=tce -Djava.library.path=/data/cplex 
    java_command: app.jar
    java_class_path (initial): app.jar
    Launcher Type: SUN_STANDARD
    
    #15.程序运行的环境信息
    Environment Variables:
    JAVA_HOME=/usr/java/default
    JRE_HOME=/usr/java/default/jre
    CLASSPATH=.:/usr/java/default/lib:/usr/java/default/jre/lib
    PATH=/data/anaconda3/bin:/data/anaconda3/condabin:/usr/java/default/bin:/usr/local/swig-4.0.2/bin:/usr/local/go/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ucloud_bi/spark/bin:/home/ops/.local/bin:/home/ops/bin
    SHELL=/bin/bash
    
    #16.信号处理信息。
    Signal Handlers:
    SIGSEGV: [libjvm.so+0xada700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGBUS: [libjvm.so+0xada700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGFPE: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGPIPE: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGXFSZ: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGILL: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
    SIGUSR2: [libjvm.so+0x90e110], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
    SIGHUP: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
    SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
    SIGTERM: [libjvm.so+0x90e730], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    SIGQUIT: [libjvm.so+0x90e730], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
    
    #17.下面是机器的相关信息。包括操作系统的版本、内存、cpu等信息
    ---------------  S Y S T E M  ---------------
    
    OS:CentOS Linux release 7.6.1810 (Core) 
    
    uname:Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64
    libc:glibc 2.17 NPTL 2.17 
    rlimit: STACK 8192k, CORE 0k, NPROC 1000000, NOFILE 65536, AS infinity
    load average:4.02 3.61 3.14
    #内存信息
    /proc/meminfo:
    MemTotal:       65806676 kB
    MemFree:          443840 kB
    MemAvailable:     389336 kB
    Buffers:            3896 kB
    Cached:           278636 kB
    SwapCached:            0 kB
    Active:         64351060 kB
    Inactive:         153520 kB
    Active(anon):   64228544 kB
    Inactive(anon):     3252 kB
    Active(file):     122516 kB
    Inactive(file):   150268 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    Dirty:              5416 kB
    Writeback:             0 kB
    AnonPages:      64221128 kB
    Mapped:            36820 kB
    Shmem:              9964 kB
    Slab:             279032 kB
    SReclaimable:     204580 kB
    SUnreclaim:        74452 kB
    KernelStack:       10400 kB
    PageTables:       139052 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:    32903336 kB
    Committed_AS:   51608244 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:      118528 kB
    VmallocChunk:   34359611132 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:     10240 kB
    CmaTotal:              0 kB
    CmaFree:               0 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:      178024 kB
    DirectMap2M:     9259008 kB
    DirectMap1G:    59768832 kB
    #cpu信息。包括每颗cpu信息，这里是16核。
    container (cgroup) information:
    container_type: cgroupv1
    cpu_cpuset_cpus: 0-15
    cpu_memory_nodes: 0
    active_processor_count: 16
    cpu_quota: -1
    cpu_period: 100000
    cpu_shares: -1
    memory_limit_in_bytes: -1
    memory_and_swap_limit_in_bytes: -1
    memory_soft_limit_in_bytes: -1
    memory_usage_in_bytes: 66050940928
    memory_max_usage_in_bytes: 0
    
    
    CPU:total 16 (initial active 16) (16 cores per cpu, 1 threads per core) family 6 model 85 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, tsc, bmi1, bmi2, adx
    #cpu详情
    /proc/cpuinfo:
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 85
    model name      : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
    stepping        : 5
    microcode       : 0x1
    cpu MHz         : 2095.072
    cache size      : 28160 KB
    physical id     : 0
    siblings        : 16
    core id         : 0
    cpu cores       : 16
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
    bogomips        : 4190.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 46 bits physical, 48 bits virtual
    power management:
    
    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 85
    model name      : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
    stepping        : 5
    microcode       : 0x1
    cpu MHz         : 2095.072
    cache size      : 28160 KB
    physical id     : 0
    siblings        : 16
    core id         : 1
    cpu cores       : 16
    apicid          : 1
    initial apicid  : 1
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
    bogomips        : 4190.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 46 bits physical, 48 bits virtual
    power management:
    。。。。。。。。。。
    processor       : 15
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 85
    model name      : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
    stepping        : 5
    microcode       : 0x1
    cpu MHz         : 2095.072
    cache size      : 28160 KB
    physical id     : 0
    siblings        : 16
    core id         : 15
    cpu cores       : 16
    apicid          : 15
    initial apicid  : 15
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
    bogomips        : 4190.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 46 bits physical, 48 bits virtual
    power management:
    
    
    #程序奔溃时机器内存信息。空闲物理内存只有443M了。swap没有可用的了。
    Memory: 4k page, physical 65806676k(443840k free), swap 0k(0k free)
    
    vm_info: Java HotSpot(TM) 64-Bit Server VM (25.251-b08) for linux-amd64 JRE (1.8.0_251-b08), built on Mar 11 2020 23:20:16 by "java_re" with gcc 7.3.0
    #程序奔溃的时间
    time: Sun Jul 25 11:12:33 2021
    timezone: CST
    elapsed time: 261254 seconds (3d 0h 34m 14s)
    
    [ops@dis-algo34 pick-task-assign-job]$ 
    #总结一下：这个问题主要是指线上服务器部署了许多程序，有些邮件统计是每天定时启动，当这些定时脚本启动时，就占用了机器特别多内存。导致内存负载过高，操作系统就会干掉一些进程，所以解决的办法就是把一些程序移到其它机器或者升级内存空间，也可以采用日志头文件中介绍的方法。

**上面日志的最有用部分有两处**。一是：\#9.程序奔溃时最近几次的GC垃圾回收相关记录：fullGC，eden取全部回收，OldGen回收后还有98%-99%是被占用的，而且full GC出现的频率很高，可能有内存泄漏的情况。二是：日志最下面：程序奔溃时机器内存信息，可以看出这时机器扩容已经没有内存了。

重新启动程序，两天后，查看所有的FullGC执行的情况

(base) [ops@dis-algo34 pick-task-assign-job]$ cat wse-gc-2021-07-25_12-21-39.log.0.current  | grep -i full
    2021-07-25T12:21:40.877+0800: 1.304: [Full GC (Metadata GC Threshold) [PSYoungGen: 6080K->0K(558592K)] [ParOldGen: 104K->5893K(286208K)] 6184K->5893K(844800K), [Metaspace: 21004K->21003K(1069056K)], 0.0237441 secs] [Times: user=0.17 sys=0.01, real=0.02 secs] 
    2021-07-25T12:21:42.576+0800: 3.002: [Full GC (Metadata GC Threshold) [PSYoungGen: 8676K->0K(971264K)] [ParOldGen: 7173K->15275K(355840K)] 15849K->15275K(1327104K), [Metaspace: 34880K->34880K(1081344K)], 0.0753956 secs] [Times: user=0.64 sys=0.01, real=0.08 secs] 
    2021-07-25T12:21:46.124+0800: 6.550: [Full GC (Metadata GC Threshold) [PSYoungGen: 10232K->0K(984064K)] [ParOldGen: 24879K->33351K(440832K)] 35111K->33351K(1424896K), [Metaspace: 58110K->58091K(1101824K)], 0.1923452 secs] [Times: user=1.85 sys=0.03, real=0.19 secs] 
    2021-07-25T12:23:00.712+0800: 81.138: [Full GC (Metadata GC Threshold) [PSYoungGen: 8344K->0K(2166784K)] [ParOldGen: 72684K->69273K(593920K)] 81028K->69273K(2760704K), [Metaspace: 94305K->94126K(1136640K)], 0.2455757 secs] [Times: user=1.93 sys=0.06, real=0.25 secs] 
    2021-07-25T18:11:21.406+0800: 20981.833: [Full GC (Ergonomics) [PSYoungGen: 2544K->0K(224256K)] [ParOldGen: 593362K->484582K(1138688K)] 595906K->484582K(1362944K), [Metaspace: 97269K->96856K(1138688K)], 0.5887687 secs] [Times: user=5.21 sys=0.13, real=0.59 secs] 
    2021-07-26T10:39:42.090+0800: 80282.516: [Full GC (Ergonomics) [PSYoungGen: 53947K->6613K(1707008K)] [ParOldGen: 1136516K->1138656K(2050048K)] 1190464K->1145269K(3757056K), [Metaspace: 100956K->100897K(1142784K)], 1.4794690 secs] [Times: user=19.05 sys=0.17, real=1.48 secs] 
    2021-07-27T08:45:41.681+0800: 159842.107: [Full GC (Ergonomics) [PSYoungGen: 50648K->879K(2814464K)] [ParOldGen: 2047436K->2049748K(3226624K)] 2098085K->2050627K(6041088K), [Metaspace: 108690K->108660K(1148928K)], 1.9388877 secs] [Times: user=21.61 sys=0.29, real=1.94 secs] 
    (base) [ops@dis-algo34 pick-task-assign-job]$

观察到所有的fullGC后内存的回收效率比较低。比如最后一次，回收前堆内存是2098085K，回收后堆内存是2050627K；这种情况可能有两个原因，一个是内存泄漏，一个是回收时又生产了很多对象。

## 四、程序内存分析 ##

从上面的GC部分日志分析发现，系统奔溃时的几次Full GC日志中，full GC出现的频率很高，基本60s出现一次，而且GC回收后堆内存任然有98%-99%的占用，可能有内存泄漏的情况。

所以项目重新启动并运行一段时间后，使用jvm内存调优工具Arthas，dump下内存并使用MAT分析（dump内存时会阻塞程序，不能线上dump），结果如下。

dump操作：

(base) [ops@dis-algo34 pick-task-assign-job]$ java -jar arthas-boot.jar 
    [INFO] arthas-boot version: 3.5.3
    [INFO] Process 30482 already using port 3658
    [INFO] Process 30482 already using port 8563
    [INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
    * [1]: 30482 app.jar
    1
    [INFO] arthas home: /home/ops/.arthas/lib/3.5.3/arthas
    [INFO] The target process already listen port 3658, skip attach.
    [INFO] arthas-client connect 127.0.0.1 3658
      ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.                           
     /  O  \ |  .--. ''--.  .--'| '--' | / O \ '   .-' | .-. || '--'.'   |  |   |  .--.  ||  .-.  |`. `-.                          
    |  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-' | `--' `--'`--' '--' `--' `--' `--'`--' `--'`-----'                          
                                                                                    
    
    wiki       https://arthas.aliyun.com/doc                                        
    tutorials  https://arthas.aliyun.com/doc/arthas-tutorials.html                  
    version    3.5.3                                                                
    main_class                                                                      
    pid        30482                                                                
    time       2021-07-26 12:44:02                                                  
    
    [arthas@30482]$ 
    [arthas@30482]$ 
    [arthas@30482]$ 
    [arthas@30482]$ heapdump
    Dumping heap to /tmp/heapdump2021-07-26-16-454750796091920514123.hprof ... #文件所在目录
    Heap dump file created
    [arthas@30482]$ 
    
    [ops@dis-algo34 tmp]$ ll -th /tmp/heapdump2021-07-26-16-454750796091920514123.hprof 
    -rw------- 1 ops ops 4.8G Jul 26 16:46 /tmp/heapdump2021-07-26-16-454750796091920514123.hprof  #查看内存dump文件大小 超过4G，windows的文件系统不支持超过4G的文件，只能在linux下分析了。
    [ops@dis-algo34 tmp]$

下载解压MAT，并修改配置

[ops@dis-algo34 tmp]$ wget http://iso.mirrors.ustc.edu.cn/eclipse/mat/1.8/rcp/MemoryAnalyzer-1.8.0.20180604-linux.gtk.x86_64.zip
    [ops@dis-algo34 tmp]$ unzip MemoryAnalyzer-1.8.0.20180604-linux.gtk.x86_64.zip
    [ops@dis-algo34 mat]$ cd mat
    [ops@dis-algo34 mat]$ vim MemoryAnalyzer.ini  # 修改MAT的内存大小， 注意这个大小要根据你dump文件大小来的，如果dump文件是5GB那么 这里最好配>5GB 否则会报MAT内存不足的异常
    [ops@dis-algo34 mat]$ ./ParseHeapDump.sh ../heapdump2021-07-26-16-454750796091920514123.hprof  org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components 
    #如果启动报错，则修改ParseHeapDump.sh 
     [ops@dis-algo34 mat]$ vim ParseHeapDump.sh 
    #!/bin/sh
    #注释掉下面这句
    #"$(dirname -- "$0")"/MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$@"
    #添加下面这句
    /usr/java/jdk1.8.0_251/bin/java -Xmx5g -Xms5g -jar plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar -consoleLog -consolelog -application org.eclipse.mat.api.parse "$@"
    #重新运行
    [ops@dis-algo34 mat]$ ./ParseHeapDump.sh ../heapdump2021-07-26-16-454750796091920514123.hprof  org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components
    Task: Parsing heapdump2021-07-26-16-454750796091920514123.hprof
    [Task: Parsing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
    [
    Subtask: Scanning /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
    [
    [INFO] Detected compressed references, because with uncompressed 64-bit references the array at 0x671443ae0 would overlap the array at 0x671443ac8
    [....................
    [INFO] Wrote threads call stacks to /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.threads
    [....................
    [INFO] Heap /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof contains 81,886,109 objects
    [....................
    Subtask: Extracting objects from /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
    [................................................................................Task: Removing unreachable objects
    [
    Subtask: Searching for unreachable objects
    [
    Subtask: Marking reachable objects
    [.
    Subtask: Re-indexing objects
    [.
    [INFO] Removed 51,559,853 unreachable objects using 1,892,483,968 bytes
    [..
    Subtask: Re-indexing classes
    [...
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.idx.index
    [....
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.o2c.index
    [.....
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.a2s.index
    [......
    Subtask: Re-indexing outbound index
    [.......
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.inbound.index
    [........
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.outbound.index
    [.........
    Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.o2hprof.index
    [.........Task: Calculating Dominator Tree
    [
    Subtask: Dominator Tree calculation
    [
    Subtask: Depth-first search
    [..............................
    Subtask: Computing dominators
    [.........................................................
    Subtask: Calculate retained sizes
    [......................................................................
    Subtask: Create dominators index file
    [..........................................................................................]
    
    #最终生成三个报告文件
    [ops@dis-algo34 dump-analysis]$ ll -th
    total 4.9G
    -rw-r--r-- 1 root root 280K Jul 26 17:32 heapdump2021-07-26-16-454750796091920514123_Top_Components.zip #内存中的对象数量排名
    -rw-r--r-- 1 root root  46K Jul 26 17:31 heapdump2021-07-26-16-454750796091920514123_System_Overview.zip#综合分析
    -rw-r--r-- 1 root root  57K Jul 26 17:31 heapdump2021-07-26-16-454750796091920514123_Leak_Suspects.zip  #内存泄漏分析

下载到本地解压查看。

内存中的对象占比

![image-20210726180129544][]

![image-20210726180235119][]

![9b84a5d8bdc7165b9966f6b9dc760c31.png][]

从上面的分析来看，**ilog.cplex.CplexI**存在内存泄漏的可能，需要review代码。

Cplex是IBM开发用于运筹学最优解求解的求解器，这部分是一个同事写的，写完后集成到项目中的。经过代码分析，发现Cplex实例化后，使用完了没有调用end()方法，怀疑这就是内存泄漏的问题。

## 五、问题原因总结及解决 ##

经过上面分析，**复盘崩溃时的场景**：1.项目启动时，未配置JVM内存分派，使用默认配置；默认配置是，初始堆大小是1G（默认物理内存的1/64），最大堆内存是16G（默认是物理内存的1/4）；堆内分年轻代和老年代，比例默认是1:2，所以当程序不断扩容后，年轻代最多占用5.33G，老年代最多占用10.66G。2.由于程序刚启动时使用的内存较小，随着程序的运行时间变成，开始执行minor gc和full gc做垃圾回收，由于存在cplex内存泄漏，每次都会有一部分内存因为泄漏回收不了，而且泄漏的内存越来越多，当已分配的堆内存不够用时，就扩容，直到16G堆内存全部分配完，程序出现奔溃。 3.但是根据日志，奔溃时，程序的内存内没有达到16G，所以JVM继续分配内存，此时正好有其他程序那一段时间内存使用很多(超过40%)，导致没有可用的物理内存分配了，最终程序奔溃。

### 5.1 外部原因及解决 ###

目前该机器上运行的程序比较多。可能有些程序使用内存过大，或者瞬时的内存使用过大，导致当前这个java程序在分派内存时，内存不够用，导致程序奔溃。

解决办法：

1.  程序的所有人在启动程序时，最好对程序使用的资源做限制，并且经常查看项目运行的情况，做到在程序崩溃前预知风险，提前解决问题。
2.  最好能有相关报警。
3.  合理的将程序部署在四台机器上，不要把自己的所有服务都部署在同一台机器上。

### 5.2. 内部原因 ###

通过内存分析，发现有内存泄漏的可能；同时项目启动时最好预估内存使用情况并指定内存大小，尽量减少系统的自动扩缩容。

解决办法：

1.  review代码，查看cplex部分是否有内存泄漏的情况。经过代码review发现，cplex使用完后没有调用end方法。修改代码后重新发布，再dump下内存，发现已经没有cplex相关的对象了。
2.  项目启动时指定堆内存的初始值和最大值，并保持相等，这样就不用自动扩容了： **-Xms3072m -Xmx3072m**；关闭堆内存的自动调整\*\*-XX:-UseAdaptiveSizePolicy **；调整老年代和新生代比例，同时增加survivor区的比例，-XX:NewRatio=1 -XX:SurvivorRatio=6；将MetaspaceSize和MaxMetaspaceSize设置成一样的值，并设置得比初始值要大，64G的机器设置为1G，**\-XX:MetaspaceSize=1024m -XX:*MaxMetaspaceSize*=1024m \*\*。
3.  修改代码中关于线程池相关的配置，根据目前仓的数量，决定有多少线程。

## 六、调优过程中常用命令 ##

[JVM常用命令官方文档][JVM]

[JVM8官方参数说明][JVM8]

[Arthas工具使用][Arthas]

[GC日志文件格式][GC]

jmap -heap $pid

jstat -gcmetacapacity -h5 $pid1000

[image-20210726162020979]: /images/20220829/715da2595f8b4efc8cf5ba33f4f13a14.png
[image-20210726180129544]: /images/20220829/0d78cd38285c47ee8995391dcacc4891.png
[image-20210726180235119]: /images/20220829/93e4bfb0d6394c7f901ed4869af670ed.png
[9b84a5d8bdc7165b9966f6b9dc760c31.png]: /images/20220829/4c007dcaddd248428f94674cc04493d3.png
[JVM]: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/index.html
[JVM8]: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html
[Arthas]: https://blog.csdn.net/wuxiaolongah/article/details/114461027
[GC]: https://blog.csdn.net/wuxiaolongah/article/details/118612506