使用 Zabbix 监控私有信息-邮件报警

野性酷女 2022-10-11 01:16 192阅读 0赞

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70][]

*   从   [http://repo.zabbix.com/zabbix/5.0/rhel/7/x86\_64/][http_repo.zabbix.com_zabbix_5.0_rhel_7_x86_64]    下载需要的安装包

![20210630112545981.png][]

![20210630112608757.png][]

![20210630112632567.png][]

*  实验环境
 *  Zabbix-Server     192.168.2.3    CentOS 7.6    主服务器  
    Zabbix-Proxy      192.168.2.4    CentOS 7.6    代理服务器  
    Zabbix-Agent      192.168.2.5    CentOS 7.6    Web 服务器  
    Zabbix-Agent      192.168.2.6    CentOS 7.6    Mysql 服务器  
     
 *  首先在 192.168.2.5 上 配置 Zabbix 监控 Apache 服务
 *  yum -y install httpd  
    启用 Apache 服务器状态，开启 Apache 的 server-status
 *    
    vim /etc/httpd/conf/httpd.conf       
     
 *  在文件最后插入以下内容

<location /server-status>  
        SetHandler server-status  
        Order allow,deny  
        Allow from 127.0.0.1 192.168.2.0/24  
</location>

**注：允许访问 status 页面的主机，在生产环境中，可以直接固定为 zabbix-server 端的 IP 地址或者 zabbix 代理地址。**

![20210629200225885.png][]

![20210629200254502.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 1][]

*  zabbix 关联 apache 模板
 *  配置 -----   主机  ----- 模板

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 3][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 8][]

*  如果长时间获取不到数据，重启 proxy 服务
 *  systemctl restart zabbix-proxy

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 9][]

--------------------

*  配置 Zabbix 监控 MySQL
 *   mysql 数据库配置监控用户  
    Zabbix 默认提供了 MySQL 的监控模板，我们直接使用即可。
 *  配置 Zabbix 监控 mysql 服务的过程：  
    1、配置 mysql 性能监控页面。创建一个 mysql 用户，让 agent 使用此用户来获得 mysql 数据  
    2、在 zabbix 平台上，给被监控主机上关联 mysql 监控模板   
    3、查看最新监控数据数据库用户授权
 *  在 192.168.2.6 配置数据库用户授权

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 10][]

*  **注：usage 权限：该权限只能用于数据库登录，不能执行任何操作。**
 *  拷贝监控配置文件模板
 *   cp /usr/share/doc/zabbix-agent-5.0.13/userparameter\_mysql.conf /etc/zabbix/zabbix\_agentd.d/

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 11][]

*  **注：这里记录了，zabbix-agent 要对 mysql 监控的数据。 模板文件中提示我们需要创建.my.cnf 配置文件， 我们直接在数据库的/etc/my.cnf.d/client.cnf 文件中，添加 zabbix 监控 mysql 使用的帐号就可以了。**
 *  建立数据库和 zabbix 的链接信息  
    vim /etc/my.cnf.d/client.cnf
    
    \[client\]  
    user=zabbix                       \#添加之前授权的用户和密码  
    password=123456

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 12][]

*  注：MySQL 用户设置密码后，只要在 client 字段下添加用户名和密码就可以免密码登录。每次登录MySQL 的交互界面，只需键入 mysql 命令，即可默认将 host, user, password 的值根据 client.cnf中的内容传给 mysql 命令，进行 mysql 的登录操作。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 13][]

*  在 zabbix server 单独给 192.168.2.6 主机增加一个 mysql 监控模版

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 14][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 15][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 17][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 18][]

*  如果长时间收不到信息
 *  依次重启 mysqld  -----   zabbix-agent    --------   zabbix-proxy

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 19][]

*  对 mysql 进行压力测试 ：  
    mysqlslap 是 Mysql 自带的压力测试工具，可以模拟出大量客户端同时操作数据库的情况，通过结果信息来了解数据库的性能状况。  
    mysqlslap -uroot -p --concurrency=100 --number-of-queries=1000 --number-int-cols=5 --number-char-cols=20 --auto-generate-sql
 *  参数说明:  
    \--concurrency=100 \#指定同时有 100 个客户端连接; concurrence \[kənˈkʌrəns\] 同时，并  
    发  
    \--number-of-queries=1000 \#指定总的测试查询次数（并发客户端数 \* 每个客户端的查询次  
    数）  
    \--auto-generate-sql \#作用是自动生成测试 SQL。自动测试时，创建的表结构非常简单，只有两列，实际的产品环境肯定会更复杂，可以使用参数指定列的数量和类型。  
    –number-int-cols=5 \#指定生成 5 个 int 类型的列  
    –number-char-cols=20 \#指定生成 20 个 char 类型的列

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 20][]

*  结果中各项含义：  
    Average number of … \#运行所有语句的平均秒数  
    Minimum number of … \#运行所有语句的最小秒数  
    Maximum number of … \#运行所有语句的最大秒数  
    Number of clients … \#客户端数量  
    Average number of queries per client \#每个客户端运行查询的平均数

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 21][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 22][]

--------------------

*  # **使用 163 邮件服务器实现自动邮件报警功能** #
 *  邮件服务可以使用系统自带的邮件服务或第三方邮件服务来发送警告邮件。  
    使用 163 邮件服务器发送邮件，这样更加稳定，而且不会被当成垃圾邮件。
 *  首先安装 mailx 组件并配置上 163 邮箱的帐号  
    yum install mailx -y  
    然后编辑 mailx 的配置文件  
    vim /etc/mail.rc  
    在最末尾添加如下信息：  
    set from=hengsheng521@163.com  
    set smtp=imap.163.com  
    set smtp-auth-user=hengsheng521@163.com  
    set smtp-auth-password=xxxxx  
    set smtp-auth=login

![20210629212437139.png][]

*  set from=hengsheng521@163.com \#表示邮件发送方，用于 zabbix 服务端转发邮件到指定邮箱
 *  set smtp-auth-password=xxxxxx \#注意：此为 163 邮箱设置的客户端授权密码，不是邮箱密码
 *  163 邮箱授权码示意图

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 23][]

*  保存退出后测试邮件是否能够正常发送出去  
    echo "zabbix test mail" |mailx -s "zabbix" hengsheng521@163.com

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 24][]

*  创建报警媒介类型

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 25][]

*  配置报警媒介，使用脚本发送邮件。名称： 163 邮件报警

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 26][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 27][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 28][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 29][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 30][]

*  给 zabbix 用户添加报警邮箱
 *  设置报警用户（zabbix 触发报警是发送给 zabbix 用户所绑定的邮箱。所以要在用户中配置相关报警媒介）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 31][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 32][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 33][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 34][]

*  创建报警动作  
    动作由触发器进行触发，例如：192.168.2.5 c7\_2\_5 主机宕机，监控项检测不到数值后触发器就会触发动作。  
    注意：这里的事件源是触发器。是由触发器触发的动作。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 35][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 36][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 37][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 38][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 39][]

*  修改操作，邮件内容为：  
    主题：故障\{TRIGGER.STATUS\},服务器:\{HOSTNAME1\}发生: \{TRIGGER.NAME\}故障!
 *  消息：
 *  告警主机:\{HOSTNAME1\}  
    告警时间:\{EVENT.DATE\} \{EVENT.TIME\}  
    告警等级:\{TRIGGER.SEVERITY\}  
    告警信息: \{TRIGGER.NAME\}  
    告警项目:\{TRIGGER.KEY1\}  
    问题详情:\{ITEM.NAME\}:\{ITEM.VALUE\}  
    当前状态:\{TRIGGER.STATUS\}:\{ITEM.VALUE1\}  
    事件 ID:\{EVENT.ID\}  
    注：TRIGGER.SEVERITY 触发严重性 ； severity \[sɪ'verəti\] 严重程度 ；

**继续添加：恢复操作**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 40][]

默认接收人：已恢复！\{TRIGGER.STATUS\}, 服务器:\{HOSTNAME1\}: \{TRIGGER.NAME\}  
恢复信息：  
告警主机:\{HOSTNAME1\}  
告警时间:\{EVENT.DATE\} \{EVENT.TIME\}  
告警等级:\{TRIGGER.SEVERITY\}  
告警信息: \{TRIGGER.NAME\}  
告警项目:\{TRIGGER.KEY1\}  
问题详情:\{ITEM.NAME\}:\{ITEM.VALUE\}  
当前状态:\{TRIGGER.STATUS\}:\{ITEM.VALUE1\}  
事件 ID:\{EVENT.ID\}

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 41][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 42][]

*  点击添加

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 43][]

*  测试邮件发送
 *  重启 zabbix-server
 *  systemctl restart zabbix-server
 *    
    测试，关闭 192.168.2.5 上的 zabbix 客户端服务 zabbix\_agentd  
    systemctl stop zabbix-agent

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 44][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 45][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 46][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 47][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 48][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 49][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 50][]

*  测试发送邮件完成
 *  使用 stress 工具压力测试 cpu 并触发告警
 *  yum install -y epel-release  
    yum install stress -y
 *  tress 参数解释  
    \-? 显示帮助信息  
    \-v 显示版本号  
    \-q 不显示运行信息  
    \-n 显示已完成的指令情况  
    \-t --timeout N 指定运行 N 秒后停止  
    \--backoff N 等待 N 微妙后开始运行  
    \-c 产生 n 个进程 每个进程都反复不停的计算随机数的平方根，测试 cpu  
    \-i 产生 n 个进程 每个进程反复调用 sync()，sync()用于将内存上的内容写到硬盘上，测试磁  
    盘  
    \-m --vm n 产生 n 个进程,每个进程不断调用内存分配 malloc（）和内存释放 free（）函数 ，  
    测试内存  
    \--vm-bytes B 指定 malloc 时内存的字节数 （默认 256MB）  
    \--vm-hang N 表示 malloc 分配的内存多少时间后在 free()释放掉   
    \-d --hadd n 产生 n 个执行 write 和 unlink 函数的进程
 *  \-hadd-bytes B 指定写的字节数

\--hadd-noclean 不 unlink  
**注：时间单位可以为秒 s，分 m，小时 h，天 d，年 y，文件大小单位可以为 K，M，G**

*  产生 6 个 cpu 进程， 5分钟秒后停止运行  
    stress -c 6 -v -t 5m   
    再打开一个终端，使用 top 命令查看，1 核 cpu 已经占满 100%

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 51][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 52][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 53][]

*  可以看到 5 分钟内 cpu 负载达到预设值会发出告警

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 54][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 55][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 56][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 57][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 58][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 59][]

*  添加 nginx 监控

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 60][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 61][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 62][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 63][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 64][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 65][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 66][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 67][]

![2021070319575172.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 68][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 69][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 70][]

--------------------

*  **添加对 KeepAlived 和 Haproxy 的监控**
 *  **在 agent 端 编辑脚本**
 *  **mkdir /etc/zabbix/scripts**
 *  **vim /etc/zabbix/scripts/check\_keepalived.sh   
    \#!/bin/bash**
    
    **if \[\[ \`ip address show dev ens33 |grep -E "192.168.2.168/32|192.168.2.169/32"|wc -l\` == "0" \]\]; then  
            echo "0";  
    else  
            echo "1";  
    fi**
 *  **chmod +x /etc/zabbix/scripts/check\_keepalived.sh**

![20210710211132108.png][]

*  **vim /etc/zabbix/zabbix\_agentd.conf**
 *  **在末尾添加**
 *  **UnsafeUserParameters=1  
    UserParameter=check\_keepalived,/etc/zabbix/scripts/check\_keepalived.sh**

![2021071012281915.png][]

*  **保存重启 zabbix-agent 服务**
 *  **在 server 端检测**
 *  **zabbix\_get -s 192.168.2.6 -k check\_keepalived**

![20210710211211689.png][]

*  在 web 界面添加 监控项

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 71][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 72][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 73][]

*   点击添加

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 74][]

*   添加触发器

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 75][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 76][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 77][]

*   测试 停止 2.6 的 keepalived 服务

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 78][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 79][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 80][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 81][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70]: /images/20221005/c9a0280ae2be4b939ba06ddf005647da.png
[http_repo.zabbix.com_zabbix_5.0_rhel_7_x86_64]: http://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/
[20210630112545981.png]: /images/20221005/5c230c7c4a5249a3a91bef7d8ecdf97b.png
[20210630112608757.png]: /images/20221005/165c6902ec1f4c208676084330869ec0.png
[20210630112632567.png]: /images/20221005/afae21c66dbb4508810b2a7affe3ece6.png
[20210629200225885.png]: /images/20221005/bad7817df0ed4c1ba1ae9140120bc2fb.png
[20210629200254502.png]: /images/20221005/6037ce39aecf412daa0658d18f873f3d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 1]: /images/20221005/7fb46eb7e7f74eb3bcc92cae291544f6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 2]: /images/20221005/f7d2c5d4774d4f5c954ac91f131f2028.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 3]: /images/20221005/2e609a100e48404a98740cd1dc05fbb5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 4]: /images/20221005/cf7fdf2ccb104e30a66df2b0a698159d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 5]: /images/20221005/0e3fcb3072b749d9a07f0337f5eb01a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 6]: /images/20221005/a3fa8e89072744c181f94a37d99ce147.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 7]: /images/20221005/af867bb5bcf0449f9166e31187c4f9f1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 8]: /images/20221005/e4b19c3802714cf69f56f98fa4eeade5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 9]: /images/20221005/9372ced65f8845bda2eff74dd795ac84.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 10]: /images/20221005/b69b71acbae0492abf25b0ae43dba2d9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 11]: /images/20221005/b4fe3a19c3ea41f09d0d3c85f91a103a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 12]: /images/20221005/adbede19f97847bbba1ef841bdfd8905.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 13]: /images/20221005/92d0ef1bfbc44d4b85974dd4d8cbe6d6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 14]: /images/20221005/6b44ee55e95e4d4fa4b99ba15da8a845.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 15]: /images/20221005/bf26f7cc4cbf422b8db2e69367066f1b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 16]: /images/20221005/6fc72e433e494e199f8e7def15f58181.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 17]: /images/20221005/aa57d4f6cfd2490f9d2228fb299dbccd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 18]: /images/20221005/0f82afd73e1142f6aa3f0e45f24dfc49.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 19]: /images/20221005/645c6d8cab2a4574adbb70f5e39eb442.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 20]: /images/20221005/a4467c1fcbe64effb7541e9d54e561cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 21]: /images/20221005/bc0f3948965e4bf19644fad639e2f4a1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 22]: /images/20221005/a018f7157f1f4010a0f6afbaadd3ad22.png
[20210629212437139.png]: /images/20221005/bc7c0bb58b494b9ba29bcbc145622eaf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 23]: /images/20221005/3ceb2209ff4e42b8b01e0716651462be.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 24]: /images/20221005/bb844e6d3c2e4a18bf74e67faa43320a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 25]: /images/20221005/d9e6d54a5ce240c0bd5380a9eb3f3eaf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 26]: /images/20221005/69b9d54191804fc39e0640218ea9b5d3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 27]: /images/20221005/8b31f918f9bd4b93b95a793d59a9ba64.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 28]: /images/20221005/52c44c5a5be74463995c00e0f752134a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 29]: /images/20221005/69c7ee45d92e4043b526256c4682e057.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 30]: /images/20221005/2f755a8b137e4d91831649f00a877330.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 31]: /images/20221005/d2927f3c36d04ff98b502ccee91359cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 32]: /images/20221005/864dfd02d0904b169b734c4b87d2e61b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 33]: /images/20221005/fb48084b1b2848d7a1880821bda03f54.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 34]: /images/20221005/00c4031d05144e03a75a818501da2e4f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 35]: /images/20221005/1bb5df4cbf514e9fba145a5193ce056c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 36]: /images/20221005/8747e937c8824aa09e8cd5792559dbb5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 37]: /images/20221005/129b2db2a95947faacffd4121ffd9350.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 38]: /images/20221005/7b9b4c9628614d74b92bd39bc213c184.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 39]: /images/20221005/400419b74f0647c686e17a28a79b8261.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 40]: /images/20221005/2ea8f49b139e4473849a9a86b35f3ac2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 41]: /images/20221005/3f1749e72b1e4690abaf4dc70fea4f39.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 42]: /images/20221005/05f38c15afcf413ab64ed1916db1b4e4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 43]: /images/20221005/089aa25edaa94aa28b33fc1a4059556a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 44]: /images/20221005/328dc24f8c004aa59b58b7c8b13a5dc5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 45]: /images/20221005/cd5201551cc5429190ac25e9d2423291.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 46]: /images/20221005/2fc970c18782442e8de2658f0d8a5f30.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 47]: /images/20221005/1e3378eb55af4c04af4a5189e2e237dc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 48]: /images/20221005/3443042cc1c44445aa40aca5c487b2da.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 49]: /images/20221005/5cf2f1d1e0d543c19028eff4df5b7ab6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 50]: /images/20221005/4ffec6e8ffd140eebffa60efba01f07e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 51]: /images/20221005/9b79f7ad7d654903a8934c84966c3082.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 52]: /images/20221005/f6e6394cc53b41719a61bbbf7ff27be1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 53]: /images/20221005/b8b2c29a594145f0a4faf80c8114ea3f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 54]: /images/20221005/f673287cdc1b4365be8db467db3dbab1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 55]: /images/20221005/672ed8d9bed14d50b14d34e5d1b2e297.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 56]: /images/20221005/1441f45382c94bac9d38c5d9eb501423.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 57]: /images/20221005/1f953f97de784e59bd174cde02486f04.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 58]: /images/20221005/ba00ba2c7ddc4f83b1e64beca18d8d00.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 59]: /images/20221005/705ac0118c654e90821485ae9840913d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 60]: /images/20221005/d9f0d5aa59504338b1a2414167fb3410.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 61]: /images/20221005/cb63591948694f8885d0bfca15453354.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 62]: /images/20221005/7793f3ac4fe249ec92d31aa8937ad212.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 63]: /images/20221005/a5afc394f9d1477a9abd821c67f9772c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 64]: /images/20221005/0d513cdfe6ae4ab4b9f62cc4ac305daf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 65]: /images/20221005/c368334a979242f599fb50307b9e1444.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 66]: /images/20221005/3b12e778d13643d2b273bfd6c1e15ab2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 67]: /images/20221005/94ffc784f4b643109e096eeed2041106.png
[2021070319575172.png]: /images/20221005/a8ca48aee8f848988d3851dd3230ce41.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 68]: /images/20221005/7fe31af2438f47c3bf8b2e6d11cdf719.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 69]: /images/20221005/e86dc9c6c99d4e32a3f9e8bc5e0cff08.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 70]: /images/20221005/2ef81d0fd1c54c61875ca46c448fe1d7.png
[20210710211132108.png]: /images/20221005/1b4f4c96469c4406ad050f8fc0ec0665.png
[2021071012281915.png]: /images/20221005/f103d73633b049cc96e2e6d93f40d6fe.png
[20210710211211689.png]: /images/20221005/bec81a58f4a3412a914fa0431645e7ad.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 71]: /images/20221005/70d64a3145584d26bc428b7bf32fd292.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 72]: /images/20221005/a5834dd527ec4ed39780ab86f1b4659b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 73]: /images/20221005/551239b0f2524782a7855355896150f7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 74]: /images/20221005/3611826a1f5f4f15ad88ea2646c0c482.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 75]: /images/20221005/29c38a88144e479cb804491d69271551.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 76]: /images/20221005/da53f93c135c4b048efeee9bbd6832bd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 77]: /images/20221005/88e3ffbd84164e23b8be68a33e3c20f3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 78]: /images/20221005/1723f03140b04debbc75130fb3f2f014.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 79]: /images/20221005/72d0a6b289a64f0c9874616dceeef944.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 80]: /images/20221005/46a69a6c2e9742f4abe19227cc862293.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 81]: /images/20221005/b73a798a18d34eaeb36f3228d665c6fd.png