基于grafana+prometheus构建Flink监控 --转载

左手的ㄟ右手 2024-04-18 08:51 50阅读 0赞

## 基于grafana+prometheus构建Flink监控 ##

原创： Flink菜鸟 Flink菜鸟 *前天*

先上一个架构图

![format_png][]

Flink App ：通过report 将数据发出去

Pushgateway :  Prometheus 生态中一个重要工具

Prometheus :  一套开源的系统监控报警框架 （Prometheus 入门与实践）

Grafana： 一个跨平台的开源的度量分析和可视化工具，可以通过将采集的数据查询然后可视化的展示，并及时通知（可视化工具Grafana：简介及安装）

Node\_exporter : 跟Pushgateway一样是Prometheus 的组件，采集到主机的运行指标如CPU, 内存，磁盘等信息

以下安装，大部分参考博客：https://www.cnblogs.com/xiao987334176/p/9930517.html\#autoid-0-0-0

**1、docker  pull 镜像**

*   
 *   
 *   
 *

docker pull prom/node-exporterdocker pull prom/pushgatewaydocker pull prom/prometheusdocker pull grafana/grafana

查看下载的镜像

*   
 *   
 *   
 *   
 *   
 *

$ docker imagesREPOSITORY           TAG                 IMAGE ID            CREATED             SIZEprom/prometheus      latest              d5b9d7ed160a        2 weeks ago         138MBgrafana/grafana      latest              a6e14b4109af        2 weeks ago         253MBprom/pushgateway     latest              20e6dcae675f        4 weeks ago         19.2MBprom/node-exporter   latest              e5a616e4b9cf        2 months ago        22.9MB

**2、编辑prometheus.yml 、创建 Grafana 数据存储目录**

*   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *

$ mkdir /opt/grafana-storage  # grafana 数据存储目录
    $ cat /opt/prometheus/prometheus.yml # prometheus 配置global:  scrape_interval:     60s  evaluation_interval: 60s scrape_configs:  - job_name: prometheus    static_configs:      - targets: ['localhost:9090']        labels:          instance: prometheus   - job_name: linux    static_configs:      - targets: ['venn:9100']        labels:          instance: localhost  - job_name: 'pushgateway'    static_configs:      - targets: ['venn:9091']        labels:          instance: 'pushgateway'

**3、启动各个组件**

*   
 *   
 *   
 *

docker run -d -p 3000:3000   --name=grafana   -v /opt/grafana-storage:/var/lib/grafana   grafana/grafanadocker run -d -p 9100:9100  -v "/proc:/host/proc:ro"  -v "/sys:/host/sys:ro"  -v "/:/rootfs:ro"  --net="host"  prom/node-exporterdocker run -d -p 9090:9090  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  prom/prometheusdocker run -d -p 9091:9091 prom/pushgateway

查看docker进程

*   
 *   
 *   
 *   
 *   
 *

$ docker psCONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                    NAMES4a689cf48e10        prom/pushgateway     "/bin/pushgateway"       5 days ago          Up 5 days           0.0.0.0:9091->9091/tcp   infallible_goldstinefcc40433bf75        grafana/grafana      "/run.sh"                5 days ago          Up 5 days           0.0.0.0:3000->3000/tcp   grafana8ba942d0cf35        prom/prometheus      "/bin/prometheus --c…"   5 days ago          Up 5 days           0.0.0.0:9090->9090/tcp   quizzical_coldenb84b0f4be2b2        prom/node-exporter   "/bin/node_exporter"     5 days ago          Up 5 days                                    fervent_poitras

查看端口

*   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *   
 *

$ netstat -apn | grep -E '9091|3000|9090|9100'(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)tcp        0      0 172.17.0.1:39028        172.17.0.4:9091         ESTABLISHED -                   tcp6       0      0 :::9100                 :::*                    LISTEN      -                   tcp6       0      0 :::3000                 :::*                    LISTEN      -                   tcp6       0      0 :::9090                 :::*                    LISTEN      -                   tcp6       0      0 :::9091                 :::*                    LISTEN      -                   tcp6       0      0 192.168.229.129:45864   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45856   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45824   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45874   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45854   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45836   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45814   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.128:9100    192.168.229.1:13405     ESTABLISHED -                   tcp6       0      0 192.168.229.129:45826   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45844   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.128:9091    172.17.0.2:53930        ESTABLISHED -                   tcp6       0      0 192.168.229.129:45846   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.128:9100    172.17.0.2:54776        ESTABLISHED -                   tcp6       0      0 192.168.229.129:45816   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45876   192.168.229.128:9091    ESTABLISHED 40846/java          tcp6       0      0 192.168.229.129:45834   192.168.229.128:9091    TIME_WAIT   -                   tcp6       0      0 192.168.229.129:45866   192.168.229.128:9091    TIME_WAIT   -

**4、查看组件页面**

node_exporter:  ip:9100/metrics

![format_png 1][]

**查看 prometheus：ip:9090/targets**

![format_png 2][]

如果state 不是 UP 的，等一会就起来了* *

**查看Grafana： **

![format_png 3][]

默认用户名密码 ：amin/admin

此处不再赘述，**配置数据源、创建系统负载监控**参考博客：https://www.cnblogs.com/xiao987334176/p/9930517.html\#autoid-0-0-0

**5、配置Flink report ：**

在Flink 配置文件 flink-conf.yml 中添加如下内容：

*   
 *   
 *   
 *   
 *   
 *   
 *

##metricsmetrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReportermetrics.reporter.promgateway.host: vennmetrics.reporter.promgateway.port: 9091metrics.reporter.promgateway.jobName: myJobmetrics.reporter.promgateway.randomJobNameSuffix: truemetrics.reporter.promgateway.deleteOnShutdown: false

启动一个任务（上一篇博客的案例：迟到数据处理）：

flink run -m yarn-cluster -ynm LateDataProcess -yn 1 -c com.venn.stream.api.sideoutput.lateDataProcess.LateDataProcess jar/flinkDemo-1.0.jar

查看任务webUI：

![format_png 4][]

PS：任务已经跑了一段时间了

**6、Grafana 中配置Flink监控**

由于上面一句配置好Flink report、 pushgateway、prometheus，并且在Grafana中已经添加了prometheus 数据源，所以Grafana中会自动获取到 flink job的metrics 。

Grafana 首页，点击New dashboard，创建一个新的dashboard

![format_png 5][]

选中之后，即会出现对应的监控指标

![format_png 6][]

至此，Flink 的metrics 的指标展示在Grafana 中了

flink 指标对应的指标名比较长，可以在Legend 中配置显示内容，在\{ \{key\}\} 将key换成对应需要展示的字段即可，如： \{ \{job\_name\}\},\{ \{operator\_name\}\}

对应显示如下：

![format_png 7][]

保存，搞定

[阅读原文][Link 1]

![format_png 8][]

微信扫一扫  
关注该公众号

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MjBzb3RocjRnQXljNGxEWjMxSnFGOEROdWthRWFZa29FdXVLYjJuM3JNa2NtSXpKemZVY3dEQS82NDA_d3hfZm10PXBuZyZ0cD13ZWJwJnd4ZnJvbT01Jnd4X2xhenk9MSZ3eF9jbz0x?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MlJ3T2liN2RoMWQ5ZDZUM2hpYmpaaWFkd3IxRGZzREh1QzJpYmtoaE1uZEZjQXozOVdMU0x1MjBjZkEvNjQwP3d4X2ZtdD1wbmcmdHA9d2VicCZ3eGZyb209NSZ3eF9sYXp5PTEmd3hfY289MQ?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MkZvbUVHQzd5WTVpY2lhUjR4TTRsUm41cmppYk1XeURHY2NieXE0aWNiRXhuRGd3V2pScHh1Y2VYcncvNjQwP3d4X2ZtdD1wbmcmdHA9d2VicCZ3eGZyb209NSZ3eF9sYXp5PTEmd3hfY289MQ?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MnpYYlhjNWVJaWJPZXdxd3kzT1htY1h5OHZ3OENLNVNRYVRWcnRTV01YZzhXRHFFWlRaYmFhU0EvNjQwP3d4X2ZtdD1wbmcmdHA9d2VicCZ3eGZyb209NSZ3eF9sYXp5PTEmd3hfY289MQ?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0Mm95d3pyM2NlaFU0d1BSSXAyY003cWlhOWhBWHkwRUoyRW5SY1VwTE4zaWI2Z1RCa2lhVUU0a3NyZy82NDA_d3hfZm10PXBuZyZ0cD13ZWJwJnd4ZnJvbT01Jnd4X2xhenk9MSZ3eF9jbz0x?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MjFZbEJZaWFUT05rWlo3SmljdlcwOGZreldlMDBYemY0NmVxWTZsYWlhMEQ3b2JnMXg1aWFJd0x3dHcvNjQwP3d4X2ZtdD1wbmcmdHA9d2VicCZ3eGZyb209NSZ3eF9sYXp5PTEmd3hfY289MQ?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MktnOUVmMFUxNU9tTXRaQlhnYlY2S29jYlJYWGdjWFh4UWVST3Fma09oOGdROUlwbzd3MENXUS82NDA_d3hfZm10PXBuZyZ0cD13ZWJwJnd4ZnJvbT01Jnd4X2xhenk9MSZ3eF9jbz0x?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vQ2R2U1M4NTBQb2RURXJ2RW5oZnIzUThHMHNRczE0MjUxbWliNXMyQ3l5TXBvN20yZ3J1eHQ2Rnp1a0tsM0VzeTM5VVdzUjNYdmhhck41c2Z2aWJocXVBLzY0MD93eF9mbXQ9cG5nJnRwPXdlYnAmd3hmcm9tPTUmd3hfbGF6eT0xJnd4X2NvPTE?x-oss-process=image/format,png
[Link 1]: https://mp.weixin.qq.com/s/b_XTpuohHUFmzhIf3SIOuA#%23
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tcC53ZWl4aW4ucXEuY29tL21wL3FyY29kZT9zY2VuZT0xMDAwMDAwNCZzaXplPTEwMiZfX2Jpej1NekkzTWpBeE5EWXdPQT09Jm1pZD0yMjQ3NDgzNzU5JmlkeD0xJnNuPTE3ZDZlMzVhMDk2YjExNTQxNTE1YmNkNzJmMmE0OTJlJnNlbmRfdGltZT0?x-oss-process=image/format,png