Prometheus-Operation钉钉报警

谁践踏了优雅 2024-04-18 20:14 83阅读 0赞

> 本篇文章基于Prometheus-Operation，收集Redis、MySQL、RabbitMQ的监控信息并报警

##### Redis钉钉报警 #####

###### 数据收集 ######

首先启动一个Redis的信息收集的应用程序

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 </pre></td> 
 <td><pre>docker run -d --name redis_exporter -e REDIS_ADDR="redis://172.16.40.170:6379" -e REDIS_PASSWORD="123456”-p 9121:9121 oliver006/redis_exporter </pre></td> 
 </tr> 
 </tbody> 
</table>

此时我们打开浏览器，访问ip:9121这个地址就可以获取到redis的监控信息了。

###### 数据保存 ######

这一步就是把上方应用程序收集的监控数据保存到prometheus中  
将信息收集程序通过Endpoints、Service、ServiceMonitor与k8s中的prometheus绑定

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 </pre></td> 
 <td><pre>apiVersion: v1 kind: Endpoints metadata: name: redis-metrics namespace: monitoring labels: k8s-app: redis-metrics subsets: - addresses: - ip: 172.16.40.170 ports: - name: redis-exporter port: 9121 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: redis-metrics namespace: monitoring labels: k8s-app: redis-metrics spec: type: ClusterIP clusterIP: None ports: - name: redis-exporter port: 9121 protocol: TCP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: redis-metrics namespace: monitoring labels: app: redis-metrics k8s-app: redis-metrics prometheus: kube-prometheus release: kube-prometheus spec: selector: matchLabels: k8s-app: redis-metrics namespaceSelector: matchNames: - monitoring endpoints: - port: redis-exporter interval: 15s </pre></td> 
 </tr> 
 </tbody> 
</table>

###### 数据展示 ######

此时在grafann中添加一个模板763即可成功展示刚刚收集到的数据

![640?wx\_fmt=png][640_wx_fmt_png]

###### 报警 ######

首先你需要在一个钉钉群建一个报警集群人，集群人类型选择自定义，然后在这个机器人身上你会发现一个token，记住这个token哦

紧接着需要对prometheus做一下改造，来到安装包的 contrib/kube-prometheus/manifests/目录  
在这个目录下新建一个alertmanager.yaml文件

通过如下命令替换原有的配置

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 2 </pre></td> 
 <td><pre>kubectl delete secret alertmanager-main -n monitoring kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring </pre></td> 
 </tr> 
 </tbody> 
</table>

因为钉钉与普罗米修斯的数据格式是不一样的，所以在这里还不能直接把普罗米修斯的数据发给钉钉。  
上方有一个url地址，这个地址呢，需要我们自己写个小程序，把普罗米修斯发来的数据经过格式转换后发给钉钉，而钉钉的格式要求可见这个文档

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 </pre></td> 
 <td><pre>https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq </pre></td> 
 </tr> 
 </tbody> 
</table>

##### MySQL钉钉报警 #####

使用如下方式收集MySQL监控信息

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 </pre></td> 
 <td><pre>docker run -d --name mysql_exporter -e DATA_SOURCE_NAME="root:123456@(172.16.40.159:3306)/" -p 9104:9104 prom/mysqld-exporter </pre></td> 
 </tr> 
 </tbody> 
</table>

数据存储参见上方Redis的配置  
数据展示使用Grafan模板736

![640?wx\_fmt=png][640_wx_fmt_png 1]

##### RabbitMQ钉钉报警 #####

使用如下方式收集MySQL监控信息

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 </pre></td> 
 <td><pre>docker run -d --name rabbitmq --env RABBIT_URL="http://172.16.40.169:15672" --env RABBIT_USER="monitoring" --env RABBIT_PASSWORD="monitoring" -p 9419:9419 kbudde/rabbitmq-exporter </pre></td> 
 </tr> 
 </tbody> 
</table>

注意RabbitMQ的用户必须具有monitoring权限，否则会报如下错误

<table> 
 <tbody> 
 <tr> 
 <td><pre>1 2 </pre></td> 
 <td><pre>time="2019-08-07T01:17:08Z" level=error msg="Error while retrieving data from rabbitHost" error="&lt;nil&gt;" host="http://172.16.40.169:15672" statusCode=401 time="2019-08-07T01:17:08Z" level=warning msg="retrieving node failed" error="Error while retrieving data from rabbitHost" </pre></td> 
 </tr> 
 </tbody> 
</table>

数据存储参见上方Redis的配置  
数据展示使用Grafan模板4279

![640?wx\_fmt=png][640_wx_fmt_png 2]

![640?wx\_fmt=gif][640_wx_fmt_gif]

不得不看

1.[SpringCloud系列博客汇总][SpringCloud]

2.[为啥一线大厂面试必问Redis，有啥好问的？][Redis]

3.[Java多线程面试必备基础知识汇总][Java]

4.[Java集合源码分析汇总][Java 1]

5.[Linux常用命令汇总][Linux]

6.[JVM系列文章汇总][JVM]

7.[MySQL系列文章汇总][MySQL]

8.[RabbitMQ系列文章汇总][RabbitMQ]

![640?wx\_fmt=jpeg][640_wx_fmt_jpeg]

[640_wx_fmt_png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/51dd0161d5014e198bf3c11296512bcb.png
[640_wx_fmt_png 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/1eac72a9e68c4a8db129f2ad5f12cb07.png
[640_wx_fmt_png 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/213e613f637049c58fb0aa0a926352b4.png
[640_wx_fmt_gif]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/7f76dad1d4014142a3e5a671a2256ef0.gif
[SpringCloud]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484030&idx=1&sn=7ad29239aca876e29155f5b1439281bb&chksm=fe396840c94ee156adfcd44c63a3b5c1cf93805cf592ad7d45bc5222c349ba6f552b553a5c09&scene=21#wechat_redirect
[Redis]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484172&idx=1&sn=cad78126f0367dbd67e0336b0b369b5d&chksm=fe396932c94ee0248ce1527844d9664a01e29a43b11b1910aa415facdc569c82a63cb07e44cf&scene=21#wechat_redirect
[Java]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484030&idx=2&sn=ddf120fcbfc45238e64662fd183156de&chksm=fe396840c94ee156a80b744033ac34d67e331545193d2f9d4f2943995aa4412a53fec023d599&scene=21#wechat_redirect
[Java 1]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484098&idx=1&sn=ba605ae1b35d6f2a8caef5901f0b86f6&chksm=fe3968fcc94ee1ea2493ce7faaab40316fa3954effa1613165b47955f5c097ef96c400b1485b&scene=21#wechat_redirect
[Linux]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484117&idx=1&sn=b4c6eb9b4027390767e92efad018a08e&chksm=fe3968ebc94ee1fdf40b5192f574b0e7cdc176663411c7f9f44c814a71a14139c510b0c35bf2&scene=21#wechat_redirect
[JVM]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484238&idx=1&sn=a30cd56a84f93da11bf916591612ef8d&chksm=fe396970c94ee066ef23c65375f88dd6ff727fd1c03c60ba61d1a9072a879cbb54084f779107&scene=21#wechat_redirect
[MySQL]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484315&idx=1&sn=8aaa78331f58909420c5e1d283d070e1&chksm=fe3969a5c94ee0b38f5152157b187f136a8dcb283039a715e8161d2e5fa98de92b66d0522cb4&scene=21#wechat_redirect
[RabbitMQ]: http://mp.weixin.qq.com/s?__biz=MzU5MDgzOTYzMw==&mid=2247484392&idx=1&sn=700f4eb1b4b3ca3ac29f1445a3a3f79c&chksm=fe3969d6c94ee0c019e758880cc53d57496f6f104d76231d1e3eea8973558db5cc30611f8154&scene=21#wechat_redirect
[640_wx_fmt_jpeg]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/645cfab7aec84f8bb97da618d11812f5.png