jenkins（三）--- jenkins + gitlab + docker + harbor 实现自动触发更新

淩亂°似流年 2023-02-11 12:26 37阅读 0赞

# 一、针对jenkins本地docker主机 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70]

首先需要配置jenkins + gitlab自动触发更新：[https://blog.csdn.net/qq\_35887546/article/details/106304046][https_blog.csdn.net_qq_35887546_article_details_106304046]

<table> 
 <thead> 
  <tr> 
   <th>主机</th> 
   <th>ip</th> 
   <th>作用</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>server1</td> 
   <td>172.25.63.1</td> 
   <td>gitlab主机</td> 
  </tr> 
  <tr> 
   <td>server2</td> 
   <td>172.25.63.2</td> 
   <td>jenkins+docker主机</td> 
  </tr> 
  <tr> 
   <td>server3</td> 
   <td>172.25.63.3</td> 
   <td>harbor仓库主机</td> 
  </tr> 
  <tr> 
   <td>server4</td> 
   <td>172.25.63.4</td> 
   <td>docker主机</td> 
  </tr> 
 </tbody> 
</table>

首先需要在jenkins安装docker的插件：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 1]

准备harbor仓库：

[https://blog.csdn.net/qq\_35887546/article/details/105411743][https_blog.csdn.net_qq_35887546_article_details_105411743]

我们将harbor仓库部署在server3 （172.25.63.3）上。

配置项目test：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 2]添加认证如下：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 3]其中需要在server2安装docker，可以参考之前的博客，还需要更改docker.sock的权限（jenkins使用的是普通用户jenkins，因此需要添加权限）：

[root@server2 ~]# chmod 777 /run/docker.sock

同时确保server2可以连接上server3部署的harbor仓库（比如证书，解析等都需要做）。

可以先在server2拉取nginx节省以后的时间。

之后进行保存。

接下来需要进行测试，在git仓库主机server1创建Dockerfile:

vim Dockerfile
    
    FROM nginx
    EXPOSE  80
    COPY index.html /usr/share/nginx/html
    
    
    cat index.html
    	8888888888
    
    git add Dockerfile
    git commit -m "add dockerfile"
    git push -u origin master

push后查看gitlab可以看出已经变更：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 4]这个变更就会触发jenkins的更新：  
可以看到构建成功：

![在这里插入图片描述][20200523191035464.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 5]现在我们将跳过push取消掉，之后保存手动触发，可以看到构建成功：  
![在这里插入图片描述][20200523191253906.png]

我的编号为20是因为中间更改了很多错误。

之后在harbor仓库里面就可以看到是上传的镜像：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 6]

这个latest标签的镜像始终和最新一次构建的版本相同。

也可以添加构建后的动作。

接下来我们新建一个项目docker：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 7]  
配置项目docker：

设置在test构建稳定后执行触发：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 8]其动作为使用刚刚创建的镜像构建一个容器：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 9]保存。

手动触发test项目：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 10]

可以看出docker项目构建成功。

我们在docker主机查看：

![在这里插入图片描述][20200523192011394.png]容器成功运行。

这一系列就部署成功，**我们的流程是 用户通过git将代码提交到gitlab，gitlab通过其配置的webhook方式触发jenkins，jenkins调用docker build 插件通过dockerfile创建镜像然后推送到harbor仓库上，之后触发jenkins 里的项目docker执行docker run命令，创建容器。**

# 完整流程 #

我们接下来再次进行一遍完整的流程，实验之前首先需要更改docker项目的构建动作，因为当运行有同名容器时容器会启动失败：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 11]保存后开始实验：

现在server1更改`index.html`文件并提交:

[root@server1 ~]# cd demo/
    [root@server1 demo]# ls
    Dockerfile  index.html  README.md
    [root@server1 demo]# vim index.html 
    [root@server1 demo]# cat index.html 
    8888888888
    8888888888
    8888888888
    8888888888
    8888888888
    8888888888
    [root@server1 demo]# git add index.html
    [root@server1 demo]# git commit -m "update index.html"
    [master 1cfa872] update index.html
     1 file changed, 5 insertions(+)
    [root@server1 demo]# git push -u origin master 
    Counting objects: 5, done.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 249 bytes | 0 bytes/s, done.
    Total 3 (delta 1), reused 0 (delta 0)
    To git@172.25.63.1:root/demo.git
       4c0b5bd..1cfa872  master -> master
    Branch master set up to track remote branch master from origin.

push后会更新gitlab：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 12]之后会触发jenkins项目test和docker的构建：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 13]可以看出都构建成功。

再docker主机访问容器地址：

[root@server2 ~]# curl 172.17.0.2
    8888888888
    8888888888
    8888888888
    8888888888
    8888888888
    8888888888

查看harbor仓库：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 14]可以看出镜像已经更新。

实验结束。

# 二、针对远程docker主机 #

我们在上个实验中设置的Docker Host URI为本地主机 `unix:///...`,这里的主机指的是jenkins本地的docker引擎，接下来我们实现远程主机的docker调用：

首先准备虚拟机 server4（172.25.63.4），并安装docker，并配置镜像加速器。

安装后使用`systemctl start docker`开启docker后默认不会打开对外的接口，我们需要打开这个端口，打开时我们需要进行tls加密：

生成key和ca证书：

[root@server4 ~]# openssl genrsa -aes256 -out ca-key.pem 4096
    Generating RSA private key, 4096 bit long modulus
    .........................++
    .............++
    e is 65537 (0x10001)
    Enter pass phrase for ca-key.pem:			#这里输入密码
    Verifying - Enter pass phrase for ca-key.pem:
    [root@server4 ~]# openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem
    Enter pass phrase for ca-key.pem:			#输入刚才的密码
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [XX]:**
    State or Province Name (full name) []:***
    Locality Name (eg, city) [Default City]:**
    Organization Name (eg, company) [Default Company Ltd]:**
    Organizational Unit Name (eg, section) []:**
    Common Name (eg, your name or your server's hostname) []:server4		#这里写上主机名。
    Email Address []:test@11.com

生成server-key和csr文件（server4为dcker主机名）：

[root@server4 ~]# openssl genrsa -out server-key.pem 4096
    [root@server4 ~]# openssl req -subj "/CN=server4" -sha256 -new -key server-key.pem -out server.csr

可以使用ip地址方式进行tls连接：

[root@server4 ~]# openssl req -subj "/CN=server4" -sha256 -new -key server-key.pem -out server.csr
    [root@server4 ~]# echo subjectAltName = DNS:server4,IP:172.25.63.4,IP:127.0.0.1 >> extfile.cnf
    [root@server4 ~]# echo extendedKeyUsage = serverAuth >> extfile.cnf
    [root@server4 ~]# 
    [root@server4 ~]# cat extfile.cnf
    subjectAltName = DNS:server4,IP:172.25.63.4,IP:127.0.0.1
    extendedKeyUsage = serverAuth
    [root@server4 ~]# 
    [root@server4 ~]# openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem   -CAcreateserial -out server-cert.pem -extfile extfile.cnf
    Signature ok
    subject=/CN=server4
    Getting CA Private Key
    Enter pass phrase for ca-key.pem:			#输入刚才的密码

接下来安装docker证书：

[root@server4 ~]# cp ca.pem server-cert.pem server-key.pem /etc/docker/
    [root@server4 ~]# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/
    [root@server4 ~]# vim /etc/systemd/system/docker.service 
    ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem -H tcp://0.0.0.0:2376

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 15]重启docker：

[root@server4 ~]# systemctl daemon-reload 
    [root@server4 ~]# systemctl restart docker
    [root@server4 ~]# netstat -antlp | grep 2376
    tcp6       0      0 :::2376                 :::*                    LISTEN      4032/dockerd

可以看到2376端口已经打开说明远程主机的docker已经可以使用。

接下来进行jenkins的配置。

对项目test进行配置：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 16]  
其中server credentials中添加的信息需要客户端的认证：

[root@server4 ~]# openssl genrsa -out key.pem 4096
    [root@server4 ~]# openssl req -subj '/CN=client' -new -key key.pem -out client.csr
    [root@server4 ~]# echo extendedKeyUsage = clientAuth >> extfile.cnf
    [root@server4 ~]# openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem   -CAcreateserial -out cert.pem -extfile extfile.cnf
    Signature ok
    subject=/CN=client
    Getting CA Private Key
    Enter pass phrase for ca-key.pem:			#输入刚才的密码

此时就生成了客户端的证书（cert.pem）和key（key.pem）。

在网页端添加：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 17]添加描述：  
![在这里插入图片描述][20200527104444371.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 18]之后保存。使server4能够连接harbor仓库，需要做解析和复制证书。

[root@server4 ~]# cd /etc/docker/certs.d/reg.westos.org/
    [root@server4 reg.westos.org]# ls
    ca.crt
    [root@server4 docker]# vim /etc/hosts
    [root@server4 docker]# cat /etc/hosts
    
    172.25.63.3     server3 reg.westos.org

接下来手动触发test项目的更新：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 19]上面的docker项目没有成功是因为我们在docker重启后没有将`/run/docker.sock`的权限设置为777.  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 20]控制台输出成功，查看harbor仓库：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 21]  
再进行以下整体的流程：

再gitlab主机编写测试文件：

[root@server1 ~]# cd demo/
    [root@server1 demo]# ls
    Dockerfile  index.html  README.md
    [root@server1 demo]# vim index.html 
    [root@server1 demo]# cat index.html 
    99999999
    99999999
    99999999
    [root@server1 demo]# vim test.html
    [root@server1 demo]# cat test.html 
    www.test.com
    [root@server1 demo]# vim Dockerfile 
    [root@server1 demo]# cat Dockerfile 
    FROM nginx
    EXPOSE 80
    COPY index.html /usr/share/nginx/html
    COPY test.html /usr/share/nginx/html

推送：

[root@server1 demo]# git add .
    [root@server1 demo]# git commit -m "update v2"
    
    [root@server1 demo]# git push -u origin master

push后查看gitlab：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 22]可以看到gitlab更新成功，之后会触发jenkins的构建：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 23]查看harbor仓库：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 24]可以看出仓库也更新了。因此说明整体流程触发没有问题。

# 三、使用ssh插件远程控制主机 #

我们之前在docker项目中都是在docker主机的本地启动容器，现在我们尝试使用ssh的方式远程控制主机启动容器。

这里将server4设置为远程ssh主机，也可以再开启一个主机（需要安装配置docker）。

首先安装ssh插件：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 25]  
进行jenkins系统配置：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 26]配置ssh远程主机：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 27]添加认证信息：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 28]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 29]保存后配置：  
接下来进行docker项目的配置：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 30]保存后就可以进行手动构建了：  
查看控制台输出：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 31]成功。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 32]我们来访问一下：

[root@server4 ~]# curl 172.25.63.4
    99999999
    99999999
    99999999
    [root@server4 ~]# curl 172.25.63.4/test.html
    www.test.com

可以看出docker项目构建完成。

接下来我们进行一遍完整的流程。

[root@server1 demo]# vim test.html 
    [root@server1 demo]# cat test.html 
    www.redhat.com
    [root@server1 demo]# git add .
    [root@server1 demo]# git commit -m "update v3"
    [root@server1 demo]# git push -u origin master

push后查看gitlab已更新：  
![在这里插入图片描述][20200527114638285.png]jenkins两个项目均构建成功：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 33]harbor仓库的镜像也更新了：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 34]进行测试访问：

[root@server4 ~]# curl 172.25.63.4/test.html
    www.redhat.com

表示这一整套触发成功。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200527134146668.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[https_blog.csdn.net_qq_35887546_article_details_106304046]: https://blog.csdn.net/qq_35887546/article/details/106304046
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200523184753555.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[https_blog.csdn.net_qq_35887546_article_details_105411743]: https://blog.csdn.net/qq_35887546/article/details/105411743
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200523185505734.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200523185606892.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200523190544421.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[20200523191035464.png]: https://img-blog.csdnimg.cn/20200523191035464.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200523190824559.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[20200523191253906.png]: https://img-blog.csdnimg.cn/20200523191253906.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200523191421898.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200523191640629.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200523191717506.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200523191755110.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200523191928181.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[20200523192011394.png]: https://img-blog.csdnimg.cn/20200523192011394.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200523192508378.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200523192647779.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200523192728475.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200523192844587.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200527103451207.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200527103722173.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200527104425141.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[20200527104444371.png]: https://img-blog.csdnimg.cn/20200527104444371.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200527104601645.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200527105220182.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200527105109413.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20200527105143309.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200527105558832.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/2020052710582642.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/20200527105859554.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20200527111525260.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 26]: https://img-blog.csdnimg.cn/20200527111553194.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 27]: https://img-blog.csdnimg.cn/20200527111659955.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 28]: https://img-blog.csdnimg.cn/20200527111815667.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 29]: https://img-blog.csdnimg.cn/20200527112158644.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 30]: https://img-blog.csdnimg.cn/20200527112325463.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 31]: https://img-blog.csdnimg.cn/20200527114339907.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 32]: https://img-blog.csdnimg.cn/20200527114357465.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[20200527114638285.png]: https://img-blog.csdnimg.cn/20200527114638285.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 33]: https://img-blog.csdnimg.cn/20200527114710595.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2_size_16_color_FFFFFF_t_70 34]: https://img-blog.csdnimg.cn/20200527114723824.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1ODg3NTQ2,size_16,color_FFFFFF,t_70