JVM性能调优监控jps、jstack、jmap、jhat、jstat详解

逃离我推掉我的手 2023-07-05 10:09 1阅读 0赞

**目录**

1、jps

2、jinfo

2.1、打印虚拟机标记参数的值

2.2、开启或关闭虚拟机标记参数

2.3、设置虚拟机标记参数

2.4、打印虚拟机参数

2.5、打印系统参数

3、jstat

3.1、查看class加载信息

3.2、查看编译统计

3.3、垃圾回收统计

4、jmap

4.1、堆内存使用信息

4.2、查看内存中对象数量及大小

4.3、保存内存使用信息到dump文件

5、jhat

5.1、分析dump文件

6、jstack

7、实战

7.1、死锁案例

--------------------

# 1、jps #

java提供的一个显示当前所有java进程pid的命令，我们可以通过它来查看我们到底启动了几个java进程。

格式：**jps \[-q\] \[-mlvV\] \[<hostid>\]**

*  \-q ： 只显示pid，不显示class名称、jar文件名、传递给main方法的参数
 *  \-m：输出传递给main方法的参数
 *  \-l  ：输出应用程序main class的完整package名或者应用程序的jar文件完整路径名
 *  \-v ：输出传递给JVM的参数
 *  \-V： 隐藏输出传递给JVM的参数

[root@jeespring ~]# jps
    12034 jar
    8132 Jps
    12524 QuorumPeerMain
    [root@jeespring ~]# jps -mlv
    8160 sun.tools.jps.Jps -mlv -Dapplication.home=/usr/java/jdk1.8.0_221-amd64 -Xms8m
    12034 dubbo-admin-0.0.1-SNAPSHOT.jar
    12524 org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/zookeeper-3.5.6/bin/../conf/zoo.cfg
     -Dzookeeper.log.dir=/usr/zookeeper-3.5.6/bin/../logs 
     -Dzookeeper.log.file=zookeeper-root-server-jeespring.log 
     -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError 
     -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -Dcom.sun.management.jmxremote 
     -Dcom.sun.management.jmxremote.local.only=false
    [root@jeespring ~]#

# 2、jinfo #

jinfo主要用于打印配置信息，包括命令行参数、系统变量。极少数的情况下，我们可以用其来修改命令行参数。

格式：**jinfo -<option> <pid>**

## **2.1、**打印虚拟机标记参数的值 ##

\-flag用于打印虚拟机标记参数的值，name表示虚拟机标记参数的名称。

格式：**jinfo -flag <name> <pid>**

[root@jeespring ~]# jinfo -flag PrintGC 12034
    -XX:-PrintGC

## **2.**2**、**开启或关闭虚拟机标记参数 ##

\-flag \[+|-\]用于开启或关闭虚拟机标记参数。+表示开启，-表示关闭。

格式：**jinfo -flag \[+|-\]<name> <pid>**

[root@jeespring ~]# jinfo -flag PrintGC 12034
    -XX:-PrintGC
    # 启用PrintGC
    [root@jeespring ~]# jinfo -flag +PrintGC 12034
    [root@jeespring ~]# jinfo -flag PrintGC 12034
    -XX:+PrintGC

## **2.**3**、**设置虚拟机标记参数 ##

\-flag <name>=<value>用于设置虚拟机标记参数。

格式：**jinfo -flag <name>=<value>** **<pid>**

[root@jeespring ~]# jinfo -flag HeapDumpPath 12034
    -XX:HeapDumpPath=
    #设置dump文件路径
    [root@jeespring ~]# jinfo -flag HeapDumpPath=/app/dump 12034
    [root@jeespring ~]# jinfo -flag HeapDumpPath 12034
    -XX:HeapDumpPath=/app/dump

## 2.4、打印虚拟机参数 ##

\-flags用于打印虚拟机参数。例如-XX:InitialHeapSize，-XX:MaxHeapSize等虚拟机参数。

格式：**jinfo -flags <pid>**

[root@jeespring ~]# jinfo -flags 12034
    Attaching to process ID 12034, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.221-b11
    Non-default VM flags: -XX:CICompilerCount=2 -XX:InitialHeapSize=31457280 -XX:MaxHeapSize=482344960 
    -XX:MaxNewSize=160759808 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=10485760 -XX:OldSize=20971520 
    -XX:+PrintGC -XX:+UseCompressedClassPointers -XX:+UseCompressedOops

## 2.5、打印系统参数 ##

\-sysprops用于打印系统参数。

格式：**jinfo -sysprops <pid>**

[root@jeespring ~]# jinfo -sysprops 12034
    Attaching to process ID 12034, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.221-b11
    java.runtime.name = Java(TM) SE Runtime Environment
    java.vm.version = 25.221-b11
    sun.boot.library.path = /usr/java/jdk1.8.0_221-amd64/jre/lib/amd64
    java.protocol.handler.pkgs = org.springframework.boot.loader
    java.vendor.url = http://java.oracle.com/
    java.vm.vendor = Oracle Corporation
    path.separator = :
    file.encoding.pkg = sun.io
    java.vm.name = Java HotSpot(TM) 64-Bit Server VM
    ...//省略

# **3、jstat** #

jstat命令可以查看堆内存各部分的使用情况，已经类的加载数量

格式：**jstat -<option> \[-t\] \[-h<lines>\] <vmid> \[<interval> \[<count>\]\]**

*  option：命令选项
 *  vmid：VM的进程号
 *  interval：时间间隔
 *  count：查询次数

## 3.1、查看class加载信息 ##

**jstat -class <pid>**

[root@jeespring ~]# jps
    12034 jar
    12715 Jps
    12524 QuorumPeerMain
    [root@jeespring ~]# jstat -class 12034
    Loaded  Bytes  Unloaded  Bytes     Time   
      7838 14397.5        0     0.0      13.55

*  Loaded：加载class的数量
 *  Bytes：所占用空间大小
 *  Unloaded：未加载数量
 *  Bytes：未加载占用空间
 *  Time：时间

## 3.2、查看编译统计 ##

**jstat -compiler <pid>**

[root@jeespring ~]# jstat -compiler 12034
    Compiled Failed Invalid   Time   FailedType FailedMethod
        4188      2       0    12.31          1 java/lang/ClassLoader getPackage

*  Compiled：编译数量
 *  Failed：失败数量
 *  Invalid：不可用数量
 *  Time：时间
 *  FailedType：失败类型
 *  FailedMethod：失败方法

## 3.3、垃圾回收统计 ##

**jstat -gc <pid>**

[root@jeespring ~]# jstat -gc 12034
     S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
    1472.0 1472.0 1224.4  0.0   11840.0   9785.9   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440
    #每隔1秒打印一次，共打印5次
    [root@jeespring ~]# jstat -gc 12034 1000 5
     S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
    1472.0 1472.0 1224.4  0.0   11840.0  10255.1   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440
    1472.0 1472.0 1224.4  0.0   11840.0  10255.1   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440
    1472.0 1472.0 1224.4  0.0   11840.0  10255.1   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440
    1472.0 1472.0 1224.4  0.0   11840.0  10255.1   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440
    1472.0 1472.0 1224.4  0.0   11840.0  10255.1   29512.0    27765.6   42880.0 41198.6 5504.0 5209.8    114    0.349   2      0.091    0.440

*  S0C：第一个survivor的大小 (KB)
 *  S1C：第二个survivor的大小 (KB)
 *  S0U：第一个survivor目前已使用空间 (KB)
 *  S1U：第二个survivor目前已使用空间 (KB)
 *  EC：Eden区的大小 (KB)
 *  EU：Eden区目前已使用空间 (KB)
 *  OC：Old区的大小 (KB)
 *  OU：Old区目前已使用空间 (KB)
 *  MC：方法区大小 (KB)
 *  MU：方法区目前已使用空间 (KB)
 *  CCSC：压缩类空间大小
 *  CCSU：压缩类目前已使用空间
 *  YGC：年轻代垃圾回收次数
 *  YGCT：年轻代垃圾回收消耗时间
 *  FGC：老年代垃圾回收次数
 *  FGCT：老年代垃圾回收消耗时间
 *  GCT：垃圾回收消耗总时间

# 4、jmap #

内存使用情况汇总，内存溢出定位与分析

## 4.1、堆内存使用信息 ##

**jmap -heap <pid>**

[root@jeespring ~]# jmap -heap 12034
    Attaching to process ID 12034, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.221-b11
    
    using thread-local object allocation.
    Mark Sweep Compact GC
    
    Heap Configuration:#堆内存配置信息
       MinHeapFreeRatio         = 40
       MaxHeapFreeRatio         = 70
       MaxHeapSize              = 482344960 (460.0MB)
       NewSize                  = 10485760 (10.0MB)
       MaxNewSize               = 160759808 (153.3125MB)
       OldSize                  = 20971520 (20.0MB)
       NewRatio                 = 2
       SurvivorRatio            = 8
       MetaspaceSize            = 21807104 (20.796875MB)
       CompressedClassSpaceSize = 1073741824 (1024.0MB)
       MaxMetaspaceSize         = 17592186044415 MB
       G1HeapRegionSize         = 0 (0.0MB)
    
    Heap Usage:#堆内存使用信息
    New Generation (Eden + 1 Survivor Space):
       capacity = 13631488 (13.0MB)
       used     = 11825112 (11.277305603027344MB)
       free     = 1806376 (1.7226943969726562MB)
       86.74850463867188% used
    Eden Space:
       capacity = 12124160 (11.5625MB)
       used     = 10571336 (10.081611633300781MB)
       free     = 1552824 (1.4808883666992188MB)
       87.19231682854729% used
    From Space:
       capacity = 1507328 (1.4375MB)
       used     = 1253776 (1.1956939697265625MB)
       free     = 253552 (0.2418060302734375MB)
       83.1787109375% used
    To Space:
       capacity = 1507328 (1.4375MB)
       used     = 0 (0.0MB)
       free     = 1507328 (1.4375MB)
       0.0% used
    tenured generation:
       capacity = 30220288 (28.8203125MB)
       used     = 28432008 (27.11487579345703MB)
       free     = 1788280 (1.7054367065429688MB)
       94.08251833999729% used
    
    18060 interned Strings occupying 2342184 bytes.

## 4.2、查看内存中对象数量及大小 ##

**jmap -histo <pid> |more   **查看所有对象数量

**jmap -histo:live <pid> |more ** 查看活跃对象数量

[root@jeespring ~]# jmap -histo 12034 |more
    
     num     #instances         #bytes  class name
    ----------------------------------------------
       1:         98501       16905160  [C
       2:          8554        3300904  [I
       3:          8183        2891616  [B
       4:         80162        1923888  java.lang.String
       5:         18737        1648856  java.lang.reflect.Method
       6:          8404         936032  java.lang.Class
       7:         25485         815520  java.util.concurrent.ConcurrentHashMap$Node
       8:         18321         732840  java.util.LinkedHashMap$Entry
    ...//省略

## 4.3、保存内存使用信息到dump文件 ##

格式：**jmap -dump:format=b,file=dumpFileName <pid>**

示例：**jmap -dump:format=b,file=/app/dump.dat 12034**

[root@jeespring ~]# jmap -dump:format=b,file=/app/dump.dat 12034
    Dumping heap to /app/dump.dat ...
    Heap dump file created
    [root@jeespring ~]# ll /app/
    total 90528
    -rw-r--r-- 1 root root 31739793 Feb 13 21:12 dubbo-admin-0.0.1-SNAPSHOT.jar
    -rw------- 1 root root 60959743 Feb 13 22:24 dump.dat

# 5、jhat #

jmap保存的dump文件时二进制的文件，需要借助工具进行查看分析，jhat工具可以堆dump文件进行分析。

## 5.1、分析dump文件 ##

格式：**jhat -port <port><file>**

示例：**jhat -port 9000 dump.dat **

[root@jeespring app]# jhat -port 9000 dump.dat 
    Reading from dump.dat...
    Dump file created Thu Feb 13 22:24:33 CST 2020
    Snapshot read, resolving...
    Resolving 593968 objects...
    Chasing references, expect 118 dots................................................................
    ...............................
    Eliminating duplicate references...................................................................
    ...............................
    Snapshot resolved.
    Started HTTP server on port 9000
    Server is ready.

上面的命令 **jhat -port 9000 dump.dat** 把分析的dump文件在**9000**端口打开，我们可以通过浏览器打开9000端口查看dump文件内容。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70][]

在页面最下面，有个

*  [Execute Object Query Language (OQL) query][Execute Object Query Language _OQL_ query]

我们可以点进去，使用OQL语言进行查询

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70 1][]

OQL的语法可以通过下面OQL Help查看OQL语法。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70 2][]

# 6、jstack #

服务器CPU的负载突然增高了，出现了死锁、死循环等异常情况，程序是正常运行的，没有任何输出，日志也看不出问题，所以就需要查看JVM内部线程的执行情况，这时候就需要借助于jstack命令。

jstack是将正在运行的JVM线程进行快照打印。

格式：**jstack <pid>**

[root@jeespring ~]# jstack 12034
    2020-02-14 00:18:53
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode):
    
    "Attach Listener" #38 daemon prio=9 os_prio=0 tid=0x00007f8ca8001000 nid=0x3d7f waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "DestroyJavaVM" #37 prio=5 os_prio=0 tid=0x00007f8cd4009800 nid=0x3c2d waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    ...//省略

# 7、实战 #

## 7.1、死锁案例 ##

1、死锁代码

public class DeadLock {
        private static Object object1 = new Object();
        private static Object object2 = new Object();
    
        public static void main(String[] args) {
            //启动线程1
            new Thread(()->{
                synchronized (object1){
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    System.out.println(Thread.currentThread().getName() + ":拿到了object1");
                    synchronized (object2){
                        System.out.println(Thread.currentThread().getName() + ":拿到了object2");
                    }
                }
            },"线程1").start();
            //启动线程2
            new Thread(()->{
                synchronized (object2){
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    System.out.println(Thread.currentThread().getName() + ":拿到了object2");
                    synchronized (object1){
                        System.out.println(Thread.currentThread().getName() + ":拿到了object1");
                    }
                }
            },"线程2").start();
        }
    }

[root@jeespring ~]# javac DeadLock.java 
    [root@jeespring ~]# java DeadLock
    线程1:拿到了object1
    线程2:拿到了object2

2、 使用jstack命令分析死锁

[root@jeespring ~]# jps
    12034 jar
    4067 DeadLock
    4170 Jps
    12524 QuorumPeerMain
    [root@jeespring ~]# jstack 4067
    2020-02-13 23:25:42
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode):
    
    "Attach Listener" #11 daemon prio=9 os_prio=0 tid=0x00007f9cb0001000 nid=0x106f waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "DestroyJavaVM" #10 prio=5 os_prio=0 tid=0x00007f9cd8009800 nid=0xfe4 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "线程2" #9 prio=5 os_prio=0 tid=0x00007f9cd80f7000 nid=0xfee waiting for monitor entry [0x00007f9cc8efd000]
       java.lang.Thread.State: BLOCKED (on object monitor)
    	at DeadLock.lambda$main$1(DeadLock.java:28)
    	- waiting to lock <0x00000000e345c110> (a java.lang.Object)
    	- locked <0x00000000e345c120> (a java.lang.Object)
    	at DeadLock$$Lambda$2/303563356.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    
    "线程1" #8 prio=5 os_prio=0 tid=0x00007f9cd80f5800 nid=0xfed waiting for monitor entry [0x00007f9cc8ffe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
    	at DeadLock.lambda$main$0(DeadLock.java:15)
    	- waiting to lock <0x00000000e345c120> (a java.lang.Object)
    	- locked <0x00000000e345c110> (a java.lang.Object)
    	at DeadLock$$Lambda$1/471910020.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    
    "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f9cd80b4000 nid=0xfeb runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9cd80b1000 nid=0xfea waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9cd80af000 nid=0xfe9 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9cd80ad000 nid=0xfe8 runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9cd807a000 nid=0xfe7 in Object.wait() [0x00007f9cdc6b4000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x00000000e3408ed8> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
    	- locked <0x00000000e3408ed8> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
    	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
    
    "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9cd8077800 nid=0xfe6 in Object.wait() [0x00007f9cdc7b5000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x00000000e3406c00> (a java.lang.ref.Reference$Lock)
    	at java.lang.Object.wait(Object.java:502)
    	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    	- locked <0x00000000e3406c00> (a java.lang.ref.Reference$Lock)
    	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
    
    "VM Thread" os_prio=0 tid=0x00007f9cd806e000 nid=0xfe5 runnable 
    
    "VM Periodic Task Thread" os_prio=0 tid=0x00007f9cd80b7000 nid=0xfec waiting on condition 
    
    JNI global references: 310
    
    
    Found one Java-level deadlock:
    =============================
    "线程2":
      waiting to lock monitor 0x00007f9cbc0062c8 (object 0x00000000e345c110, a java.lang.Object),
      which is held by "线程1"
    "线程1":
      waiting to lock monitor 0x00007f9cbc0038d8 (object 0x00000000e345c120, a java.lang.Object),
      which is held by "线程2"
    
    Java stack information for the threads listed above:
    ===================================================
    "线程2":
    	at DeadLock.lambda$main$1(DeadLock.java:28)
    	- waiting to lock <0x00000000e345c110> (a java.lang.Object)
    	- locked <0x00000000e345c120> (a java.lang.Object)
    	at DeadLock$$Lambda$2/303563356.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    "线程1":
    	at DeadLock.lambda$main$0(DeadLock.java:15)
    	- waiting to lock <0x00000000e345c120> (a java.lang.Object)
    	- locked <0x00000000e345c110> (a java.lang.Object)
    	at DeadLock$$Lambda$1/471910020.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    
    Found 1 deadlock.

上面输出信息中，Found one Java-level deadlock:，发现一个java级别的锁，下面是两个线程等待锁的信息。

Thread1获取了<0x00000000e345c110>的锁，等待获取<0x00000000e345c120>锁

Thread2获取了<0x00000000e345c120>的锁，等待获取<0x00000000e345c110>锁

Java stack information for the threads listed above:
    ===================================================
    "线程2":
    	at DeadLock.lambda$main$1(DeadLock.java:28)
    	- waiting to lock <0x00000000e345c110> (a java.lang.Object)
    	- locked <0x00000000e345c120> (a java.lang.Object)
    	at DeadLock$$Lambda$2/303563356.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    "线程1":
    	at DeadLock.lambda$main$0(DeadLock.java:15)
    	- waiting to lock <0x00000000e345c120> (a java.lang.Object)
    	- locked <0x00000000e345c110> (a java.lang.Object)
    	at DeadLock$$Lambda$1/471910020.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:748)
    
    Found 1 deadlock.

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200213224409123.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc=,size_16,color_FFFFFF,t_70
[Execute Object Query Language _OQL_ query]: http://139.199.32.191:9000/oql/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200213224452502.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200213224553693.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nY2hhb3lhbmc=,size_16,color_FFFFFF,t_70