JVM内存溢出-数据库连接对象大量累积

墨蓝 2023-02-23 05:23 30阅读 0赞

## JVM内存溢出-数据库连接对象大量累积 ##

### 题记 ###

上周三晚上截单之前，部分订单交易处理缓慢，少量订单支付由于超时连接，直接导致支付失败。信息来源有二：其一是`jvm`内存监控，间断性预警内存使用率过高，而后降下来；重复循环。其二是运营团队反应的客户下单异常情况。由此，打开电脑，开始排查。

### 分析 ###

1.  由于项目对接新的渠道，近一周内没有新需求上线，基本排除新`bug`引起的交易异常和内存预警；
2.  由于内存占用过高，周而复始，说明`GC`后内存确实会降下来；
3.  交易延时，部分交易不可用可能是由于`GC`时间过长直接导致的；

### 排查步骤 ###

1.联系运维导出线上内存占比过高节点的`dupm`文件；  
\- 提供导出脚本如下

1、步骤：
    ps -ef|grep tomcat  或 ps -ef|grep java
    2、找到xx-xxx的进程pid 
    jmap -dump:format=b,file=/xxx/xxx/m.hprof <pid>
    3、将生成的快照文件/xxx/xxx/heap.hprof 导出

2.查看服务器监控包括`CPU`、`JVM内存使用率`

监控发现`CPU`使用正常

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 1]

###### `JVM`内存使用 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 2]

###### `old区内存监控` ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 3]

###### `eden区内存监控` ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 4]

###### `survivor区内存监控` ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 5]

3.查看订单交易接口监控

###### 交易监控正常时间均值在`100ms`以内 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 6]

###### 支付接口超时时间为`2000ms` ######

导致交易超时的原因，推测是`full gc`时间过长

#### JVM监控大盘 ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 7]

#### 分析dump文件 ####

###### `memory analyzer`推测内存溢出原因 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 8]  
`problem 1`分析结果  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 9]

###### `problem 2`分析结果 ######

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 10][]

###### 查看对象数 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 11]

###### 查看对象引用 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 12]

`com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference` 这个对象堆积了`64992` 个  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 13]

###### 现象分析 ######

`com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference`对象累计增长的原因

static class ConnectionPhantomReference extends PhantomReference<ConnectionImpl> { 
            private NetworkResources io;
    
            ConnectionPhantomReference(ConnectionImpl connectionImpl, ReferenceQueue<ConnectionImpl> q) { 
                super(connectionImpl, q);
    
                try { 
                    this.io = connectionImpl.getIO().getNetworkResources();
                } catch (SQLException e) { 
                    // if we somehow got here and there's really no i/o, we deal with it later
                }
            }
    
            void cleanup() { 
                if (this.io != null) { 
                    try { 
                        this.io.forceClose();
                    } finally { 
                        this.io = null;
                    }
                }
            }
        }

查看数据库连接

public ConnectionImpl(String hostToConnectTo, int portToConnectTo, Properties info, String databaseToConnectTo, String url) throws SQLException { 
    
            this.connectionCreationTimeMillis = System.currentTimeMillis();
    
            if (databaseToConnectTo == null) { 
                databaseToConnectTo = "";
            }
    
            // Stash away for later, used to clone this connection for Statement.cancel and Statement.setQueryTimeout().
            //
    
            this.origHostToConnectTo = hostToConnectTo;
            this.origPortToConnectTo = portToConnectTo;
            this.origDatabaseToConnectTo = databaseToConnectTo;
    
            try { 
                Blob.class.getMethod("truncate", new Class[] {  Long.TYPE });
    
                this.isRunningOnJDK13 = false;
            } catch (NoSuchMethodException nsme) { 
                this.isRunningOnJDK13 = true;
            }
    
            this.sessionCalendar = new GregorianCalendar();
            this.utcCalendar = new GregorianCalendar();
            this.utcCalendar.setTimeZone(TimeZone.getTimeZone("GMT"));
    
            //
            // Normally, this code would be in initializeDriverProperties, but we need to do this as early as possible, so we can start logging to the 'correct'
            // place as early as possible...this.log points to 'NullLogger' for every connection at startup to avoid NPEs and the overhead of checking for NULL at
            // every logging call.
            //
            // We will reset this to the configured logger during properties initialization.
            //
            this.log = LogFactory.getLogger(getLogger(), LOGGER_INSTANCE_NAME, getExceptionInterceptor());
    
            this.openStatements = new HashMap<Statement, Statement>();
    
            if (NonRegisteringDriver.isHostPropertiesList(hostToConnectTo)) { 
                Properties hostSpecificProps = NonRegisteringDriver.expandHostKeyValues(hostToConnectTo);
    
                Enumeration<?> propertyNames = hostSpecificProps.propertyNames();
    
                while (propertyNames.hasMoreElements()) { 
                    String propertyName = propertyNames.nextElement().toString();
                    String propertyValue = hostSpecificProps.getProperty(propertyName);
    
                    info.setProperty(propertyName, propertyValue);
                }
            } else { 
    
                if (hostToConnectTo == null) { 
                    this.host = "localhost";
                    this.hostPortPair = this.host + ":" + portToConnectTo;
                } else { 
                    this.host = hostToConnectTo;
    
                    if (hostToConnectTo.indexOf(":") == -1) { 
                        this.hostPortPair = this.host + ":" + portToConnectTo;
                    } else { 
                        this.hostPortPair = this.host;
                    }
                }
            }
    
            this.port = portToConnectTo;
    
            this.database = databaseToConnectTo;
            this.myURL = url;
            this.user = info.getProperty(NonRegisteringDriver.USER_PROPERTY_KEY);
            this.password = info.getProperty(NonRegisteringDriver.PASSWORD_PROPERTY_KEY);
    
            if ((this.user == null) || this.user.equals("")) { 
                this.user = "";
            }
    
            if (this.password == null) { 
                this.password = "";
            }
    
            this.props = info;
    
            initializeDriverProperties(info);
    
            // We store this per-connection, due to static synchronization issues in Java's built-in TimeZone class...
            this.defaultTimeZone = TimeUtil.getDefaultTimeZone(getCacheDefaultTimezone());
    
            this.isClientTzUTC = !this.defaultTimeZone.useDaylightTime() && this.defaultTimeZone.getRawOffset() == 0;
    
            if (getUseUsageAdvisor()) { 
                this.pointOfOrigin = LogUtils.findCallingClassAndMethod(new Throwable());
            } else { 
                this.pointOfOrigin = "";
            }
    
            try { 
                this.dbmd = getMetaData(false, false);
                initializeSafeStatementInterceptors();
                createNewIO(false);
                unSafeStatementInterceptors();
            } catch (SQLException ex) { 
                cleanup(ex);
    
                // don't clobber SQL exceptions
                throw ex;
            } catch (Exception ex) { 
                cleanup(ex);
    
                StringBuilder mesg = new StringBuilder(128);
    
                if (!getParanoid()) { 
                    mesg.append("Cannot connect to MySQL server on ");
                    mesg.append(this.host);
                    mesg.append(":");
                    mesg.append(this.port);
                    mesg.append(".\n\n");
                    mesg.append("Make sure that there is a MySQL server ");
                    mesg.append("running on the machine/port you are trying ");
                    mesg.append("to connect to and that the machine this software is running on ");
                    mesg.append("is able to connect to this host/port (i.e. not firewalled). ");
                    mesg.append("Also make sure that the server has not been started with the --skip-networking ");
                    mesg.append("flag.\n\n");
                } else { 
                    mesg.append("Unable to connect to database.");
                }
    
                SQLException sqlEx = SQLError.createSQLException(mesg.toString(), SQLError.SQL_STATE_COMMUNICATION_LINK_FAILURE, getExceptionInterceptor());
    
                sqlEx.initCause(ex);
    
                throw sqlEx;
            }
    
            NonRegisteringDriver.trackConnection(this);
        }

###### 分析 ######

调用链路：`ConnectionImpl` \-> `createNewIO`\->`NonRegisteringDriver.trackConnection`\->`NonRegisteringDriver.connectionPhantomRefs`

protected static final ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference> connectionPhantomRefs = new ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference>();
    
    	// 加入 
      protected static void trackConnection(Connection newConn) { 
    
            ConnectionPhantomReference phantomRef = new ConnectionPhantomReference((ConnectionImpl) newConn, refQueue);
            connectionPhantomRefs.put(phantomRef, phantomRef);
        }

###### 分析 `ConnectionPhantomReference` \-> `PhantomReference` ######

*  创建对象，使用`Phantom Reference`
 *  对象不可达
 *  `GC`发生只有`Phantom Reference`引用，检查是否需要执行`Finalize`
 *  对象已处于`Finalized`状态
 *  `GC`发现，认为它是`Phantom`可达对象，则加入`Reference`的`Pending`链表
 *  `ReferenceHandler`将`Pending`链表表头加入`ReferenceQueue`
 *  手动清除`Phantom Reference`，使对象成为完全不可达对象
 *  `GC`发现对象不可达，回收对象所占的空间

###### 代码示例 ######

public class Test { 
        public static boolean isRun = true;
    
        @SuppressWarnings("static-access")
        public static void main(String[] args) throws Exception { 
            String abc = new String("abc");
            System.out.println(abc.getClass() + "@" + abc.hashCode());
            final ReferenceQueue<String> referenceQueue = new ReferenceQueue<String>();
            new Thread() { 
                public void run() { 
                    while (isRun) { 
                        Object obj = referenceQueue.poll();
                        if (obj != null) { 
                            try { 
                                Field rereferent = Reference.class
                                        .getDeclaredField("referent");
                                rereferent.setAccessible(true);
                                Object result = rereferent.get(obj);
                                System.out.println("gc will collect："
                                        + result.getClass() + "@"
                                        + result.hashCode() + "\t"
                                        + (String) result);
                            } catch (Exception e) { 
                                e.printStackTrace();
                            }
                        }
                    }
                }
            }.start();
            PhantomReference<String> abcWeakRef = new PhantomReference<String>(abc,
                    referenceQueue);
            abc = null;
            Thread.currentThread().sleep(3000);
            System.gc();
            Thread.currentThread().sleep(3000);
            isRun = false;
        }
    }

### 问题排查 ###

根据以上描述，提出以下问题：

*  1、com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference 到底是个什么对象呢？
 *  2、这种对象为什么会大量堆积，JVM回收不过来了？

NonRegisteringDriver$ConnectionPhantomReference 到底是个什么对象呢？简单来说，NonRegisteringDriver类有个虚引用集合connectionPhantomRefs用于存储所有的数据库连接，NonRegisteringDriver.trackConnection方法负责把新创建的连接放入connectionPhantomRefs集合。源码如下：

public class NonRegisteringDriver implements java.sql.Driver { 
    
        protected static final ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference> connectionPhantomRefs = new ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference>();
    
        protected static final ReferenceQueue<ConnectionImpl> refQueue = new ReferenceQueue<ConnectionImpl>();
     
        ......
        protected static void trackConnection(Connection newConn) { 
            ConnectionPhantomReference phantomRef = new ConnectionPhantomReference((ConnectionImpl) newConn, refQueue);
            connectionPhantomRefs.put(phantomRef, phantomRef);
        }
        
    }

我们追踪创建数据库连接的过程源码，发现其中会调到com.mysql.jdbc.ConnectionImpl的构造函数，该方法会调用createNewIO方法创建一个新的数据库连接MysqlIO对象，然后调用我们上面提到的NonRegisteringDriver.trackConnection方法，把该对象放入NonRegisteringDriver.connectionPhantomRefs集合。源码如下：

public class ConnectionImpl extends ConnectionPropertiesImpl implements MySQLConnection { 
    
        public ConnectionImpl(String hostToConnectTo, int portToConnectTo, Properties info, String databaseToConnectTo, String url) throws SQLException { 
            ...
            createNewIO(false);
            ...
            NonRegisteringDriver.trackConnection(this);
            ...
        }
    }

connectionPhantomRefs 是一个虚引用集合，何为虚引用？为什么设计为虚引用队列。

*  虚引用队列也称为“幽灵引用”，它是最弱的一种引用关系。
 *  如果一个对象仅持有虚引用，那么它就和没有任何引用一样，在任何时候都可能被垃 圾回收器回收。
 *  为一个对象设置虚 引用关联的唯一目的只是为了能在这个对象被收集器回收时收到一个系统通知。
 *  当垃圾回收器准备回收一个对象时，如果发现它还有虚引用，就会在垃圾回收后，将这个虚引用加入引用队列，在其关联的虚引用出队前，不会彻底销毁该对象。所以可以通过检查引用队列中是否有相应的虚引用来判断对象是否已经被回收了。

connectionPhantomRefs 这种对象为什么会大量堆积，JVM回收不过来了。

### 解决方案 ###

经调研、排查问题产生的根源是数据库连接对象堆积，导致full gc时间过长。

1.  调整`dbcp`配置参数。例如把maxWait设置为合适的值（比数据库的wait\_timeout少30s），不能过大，minIdle和maxIdle值不能设置太大，或者直接采用默认值即可。
2.  采用G1垃圾回收器。
3.  建立巡查系统，在业务低峰期主动触发full gc。

### 补充 ###

`parallel` 发生`full gc`后堆内存使用率降低示例：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 1]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 3]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200705200704469.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020070520050459.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200705195047623.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200705195226781.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200705200917227.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200705201001616.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200721101934994.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/2020072110172675.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200705201910793.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200705202408187.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200705202312273.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200705205311992.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200705205141616.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200705210255668.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20201022105252538.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20201022105256633.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20201022105259950.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/20201022105303486.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoYW5nX3hz,size_16,color_FFFFFF,t_70#pic_center