JVM 调优实战--tomcat8优化

小灰灰 2023-10-07 08:17 50阅读 0赞

**目录**

--------------------

tomcat配置优化

部署安装tomcat8

配置tomcat8

禁用AJP连接

设置tomcat线程池（执行器）

tomcat的3种运行模式

使用Apache Jmeter进行测试

下载安装jmeter

使用步骤

调整tomcat参数进行优化

调整JVM参数进行优化

设置并行垃圾回收器

查看GC日志文件

调整年轻代大小

查看GC日志

设置G1垃圾收集器

小结

--------------------

![2020011617043033.png][]

## tomcat配置优化 ##

### 部署安装tomcat8 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70][]

### 配置tomcat8 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 3][]

#### 禁用AJP连接 ####

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 5][]

![20200116172229770.png][]

![20200116172517544.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 6][]

> 重启tomcat，即可生效。
> 
> 查看禁用后的效果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 7][]

### 设置tomcat线程池（执行器） ###

> 频繁地创建线程会造成性能浪费，所以使用线程池来优化：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 9][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 10][]

### tomcat的3种运行模式 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 11][]

> tomcat8之前的版本用的是BIO，推荐使用NIO，tomcat8中有最新的NIO2，速度更快，建议使用NIO2。

## 使用Apache Jmeter进行测试 ##

> 将web应用部署到服务器之后，使用jmeter对其进行压力测试。

![20200116200417311.png][]

### 下载安装jmeter ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 12][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 13][]

### 使用步骤 ###

> 设置中文：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 14][]

> 创建并保存测试用例：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 15][]

> 添加线程组，使用线程模拟用户的并发：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 17][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 18][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 19][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 20][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 21][]

![20200116201840565.png][]

## 调整tomcat参数进行优化 ##

> tomcat不进行任何参数调整时，吞吐量为73次/秒，接下来进行参数优化：
> 
> **①禁用AJP服务：吞吐量提升至117.6**
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 22][]
> 
> **②设置线程池：单纯提升线程数量不一定能一直提升性能。**
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 23][]
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 24][]
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 25][]
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 26][]
> 
> **③设置nio2的运行模式：tomcat8默认的是nio的模式**
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 27][]

## 调整JVM参数进行优化 ##

> 接下来，通过调整JVM参数进行优化，其他tomcat参数与上面的保持一致：
> 
> 最大线程数设置为500，启用nio2运行模式。

### 设置并行垃圾回收器 ###

![20200117092449363.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 28][]

> **vi catalina.sh**：
> 
> 将jvm参数添加进去--设置年轻代和老年代的垃圾收集器均为ParallelGC并行垃圾收集器。不设置jdk8默认也是使用ParallelGC：
> 
> ![20200117094515705.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 29][]

> 修改完catalina.sh文件后，重启tomcat：
> 
> **sh startup.sh && tail -f ../logs/catalina.out**

![20200117092939907.png][]

> gc日志存放目录： **apache-tomcat-8.5.34/logs/gc.log**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 30][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 31][]

> 测试结果比较接近，因为jdk8默认使用的也是ParallelGC垃圾收集器。

### 查看GC日志文件 ###

> 将**apache-tomcat-8.5.34/logs/gc.log** 文件上传到gceasy.io查看gc中是否存在问题：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 32][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 33][]

> 只有在堆内存空间不足时才会触发FGC，FGC次数过高说明堆内存大小设置不合理。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 34][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 35][]

### 调整年轻代大小 ###

> **vi catalina.sh**
> 
> 对比下之前的配置，将初始堆大小，年轻代大小均进行提升--

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 36][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 37][]

#### 查看GC日志 ####

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 38][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 39][]

> 可以看到GC次数明显减少，说明调整是有效的。

### 设置G1垃圾收集器 ###

> 理论上而言，设置为G1垃圾收集器，性能是会提升的。但是会受制于多方面的影响，也不一定绝对有提升。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 40][]

> **vi cataliina.sh**文件：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 41][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 42][]

### 小结 ###

> 通过上述的测试，可以总结出，对tomcat性能优化就是需要不断地进行参数调整，然后测试结果，可能会调优也可能会调差，这时就需要借助于gc的可视化工具来看gc的情况。再帮助我们作出决策应该调整哪些参数。

[2020011617043033.png]: https://img-blog.csdnimg.cn/2020011617043033.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200116170654772.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200116171248297.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020011617144482.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200116171727713.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200116172113422.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200116172157247.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200116172229770.png]: https://img-blog.csdnimg.cn/20200116172229770.png
[20200116172517544.png]: https://img-blog.csdnimg.cn/20200116172517544.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200116172400409.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200116172634944.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200116172741394.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200116173134994.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/2020011617333768.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200116195408354.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200116200417311.png]: https://img-blog.csdnimg.cn/20200116200417311.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200116200513186.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200116200623935.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200116200759143.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200116200936101.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200116201021323.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/2020011620114022.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200116201227908.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200116201401295.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200116201448601.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20200116201857454.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200116201840565.png]: https://img-blog.csdnimg.cn/20200116201840565.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200116202433452.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/20200116202637174.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/20200116202902984.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20200116203242607.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 26]: https://img-blog.csdnimg.cn/20200116204047902.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 27]: https://img-blog.csdnimg.cn/20200116204400342.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200117092449363.png]: https://img-blog.csdnimg.cn/20200117092449363.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 28]: https://img-blog.csdnimg.cn/20200117092627543.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200117094515705.png]: https://img-blog.csdnimg.cn/20200117094515705.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 29]: https://img-blog.csdnimg.cn/20200117092805586.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[20200117092939907.png]: https://img-blog.csdnimg.cn/20200117092939907.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 30]: https://img-blog.csdnimg.cn/20200117093133355.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 31]: https://img-blog.csdnimg.cn/20200117094150314.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 32]: https://img-blog.csdnimg.cn/20200117094948864.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 33]: https://img-blog.csdnimg.cn/20200117095125736.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 34]: https://img-blog.csdnimg.cn/20200117095239407.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 35]: https://img-blog.csdnimg.cn/20200117095427352.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 36]: https://img-blog.csdnimg.cn/20200117095808309.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 37]: https://img-blog.csdnimg.cn/20200117095525167.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 38]: https://img-blog.csdnimg.cn/20200117100334867.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 39]: https://img-blog.csdnimg.cn/20200117100400424.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 40]: https://img-blog.csdnimg.cn/20200117100542464.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 41]: https://img-blog.csdnimg.cn/2020011710070065.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ_size_16_color_FFFFFF_t_70 42]: https://img-blog.csdnimg.cn/20200117101231414.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly96aGFuZ3h1ZWxpYW5nLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70