Prometheus 监控 Java 应用

╰+哭是因爲堅強的太久メ 2022-05-15 12:15 786阅读 0赞

##### Prometheus 监控 Java 应用 #####

Prometheus 监控 Java 应用有两种方式：一种是使用官方提供的jar包，然后嵌入到应用中。这种方式一般都是新项目。我认为也是最合适的一种。不过这种情况一般是理想而已。而除了这种方式，第二种是prometheus的jmx\_exporter。  
我们就是用的第二种。使用jmx\_exporter的方式来监控我们的java应用程序。我们的java应用基本上是使用tomcat作为服务器的。这种情况下有两种方式，一种是基于springboot的jar包启动方式，一种是直接下载tomcat软件之后，将应用打成war包部署的方式。

jmx\_exporter的使用非常简单，但是如果不了解就会非常懵逼。jmx\_exporter实际也是基于java的jmx通过暴露Mbean来做为代理，使用http的方式来给Prometheus进行指标采集。

###### 1.jar 包启动应用 ######

如果是jar包启动的方式，那么github上面就已经有示例了。可以参照：java -javaagent:./jmx\_prometheus\_javaagent-0.3.0.jar=9151:config.yaml -jar yourJar.jar，这种方式启动。这种属于在应用启动的时候就给它加上代理。  
这种方式是没有加认证的，如果需要加认证，嗯，有点麻烦，实验过一次，后来发现，还是算了。可能在虚拟机上直接运行程序还好，但是打成docker镜像就真的就有点多余了。  
这种方式是监控的内嵌tomcat的启动的应用，在访问[http://ip:9151/metrics,/metrics][http_ip_9151_metrics_metrics]可有可无，这个时候可以看到很多tomcat指标，当然如果你的config.yaml没有改动，那么可能并不会看到，因为官网的config.yaml中rules下的pattern:Catalina\*，这里是不适用与内嵌tomcat的。内嵌的tomcat需要修改为Tomcat。

---
    startDelaySeconds: 0
    #hostPort: 192.168.226.128:8999
    #jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:8999/jmxrmi
    ssl: false
    wercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    - pattern: 'Tomcat<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
      name: tomcat_$3_total
      labels:
        port: "$2"
        protocol: "$1"
      help: Tomcat global $3
      type: COUNTER
    - pattern: 'Tomcat<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
      name: tomcat_servlet_$3_total
      labels:
        module: "$1"
        servlet: "$2"
      help: Tomcat servlet $3 total
      type: COUNTER
    - pattern: 'Tomcat<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
      name: tomcat_threadpool_$3
      labels:
        port: "$2"
        protocol: "$1"
      help: Tomcat threadpool $3
      type: GAUGE
    - pattern: 'Tomcat<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
      name: tomcat_session_$3_total
      labels:
        context: "$2"
        host: "$1"
      help: Tomcat session $3 total
      type: COUNTER
    - pattern: ".*"

当然，你也看到了我的pattern这里有个：”.\*”为啥这样？因为这样可以让所有的jmx metrics全部暴露出来，这方式有点暴力但是很好，很有用。

###### 2.Tomcat war包应用（目前应用此方式） ######

进入bin目录（$TOMCAT\_HOME/bin），将jmx\_exporter.jar包文件和config.yaml文件复制到这里。然后修改里面的一个catalina.sh的脚本，找到JAVA\_OPTS,加上代理：

#修改bin/catalina.sh 文件
    添加：
    JAVA_OPTS=" -javaagent:/usr/local//bin/jmx_prometheus_javaagent-0.3.1.jar=9151:/usr/local/tomcat/bin/config.yaml"
    
    #创建./bin/config.yaml 文件
    ---   
    lowercaseOutputLabelNames: true
    lowercaseOutputName: true
    rules:
    - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
      name: tomcat_$3_total
      labels:
        port: "$2"
        protocol: "$1"
      help: Tomcat global $3
      type: COUNTER
    - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
      name: tomcat_servlet_$3_total
      labels:
        module: "$1"
        servlet: "$2"
      help: Tomcat servlet $3 total
      type: COUNTER
    - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
      name: tomcat_threadpool_$3
      labels:
        port: "$2"
        protocol: "$1"
      help: Tomcat threadpool $3
      type: GAUGE
    - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
      name: tomcat_session_$3_total
      labels:
        context: "$2"
        host: "$1"
      help: Tomcat session $3 total
      type: COUNTER

这种启动tomcat之后，你就可以通过访问9151端口来访问metrics了。这里要记得，修改上面的config.yaml中pattern的部分，改为和github上一样就可以了，也就是pattern为Catalina\*\*。  
config.yaml下载地址：[https://github.com/prometheus/jmx\_exporter/blob/master/example\_configs/tomcat.yml][https_github.com_prometheus_jmx_exporter_blob_master_example_configs_tomcat.yml]  
jar包下载地址：[https://github.com/prometheus/jmx\_exporter][https_github.com_prometheus_jmx_exporter]  
将jar包和config.yaml放在$tomcat\_path/bin/ 启动tomcat就好了

Grafana图形化：Tomcat JVM.json

{
      "__inputs": [ { "name": "DS_PROMETHEUS", "label": "Prometheus", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ],
      "__requires": [ { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "5.0.4" }, { "type": "panel", "id": "graph", "name": "Graph", "version": "5.0.0" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "5.0.0" }, { "type": "panel", "id": "singlestat", "name": "Singlestat", "version": "5.0.0" } ],
      "annotations": { "list": [ { "builtIn": 1, "datasource": "-- Grafana --", "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" } ] },
      "description": "Dashboard for JVM metrics with Prometheus / JMX Exporter",
      "editable": true,
      "gnetId": 3066,
      "graphTooltip": 0,
      "id": null,
      "iteration": 1535530883437,
      "links": [ { "asDropdown": true, "icon": "external link", "includeVars": true, "keepTime": true, "tags": [ "llzgnode" ], "targetBlank": true, "title": "OS", "type": "dashboards" }, { "asDropdown": true, "icon": "external link", "includeVars": true, "keepTime": true, "tags": [ "mysql" ], "targetBlank": true, "title": "mysql", "type": "dashboards" }, { "asDropdown": true, "icon": "external link", "includeVars": true, "keepTime": true, "tags": [ "tomcat" ], "targetBlank": true, "title": "jvm", "type": "dashboards" } ],
      "panels": [ { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "#d683ce", "#d683ce", "#d683ce" ], "datasource": "${DS_PROMETHEUS}", "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "gridPos": { "h": 3, "w": 4, "x": 0, "y": 0 }, "id": 13, "interval": null, "links": [], "mappingType": 1, "mappingTypes": [ { "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 } ], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [ { "from": "null", "text": "N/A", "to": "null" } ], "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "tableColumn": "", "targets": [ { "expr": "tomcat_requestcount_total{port=\"8080\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "refId": "A" } ], "thresholds": "", "title": "requestcount", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [ { "op": "=", "text": "N/A", "value": "null" } ], "valueName": "total" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "#299c46", "rgba(237, 129, 40, 0.89)", "#d44a3a" ], "datasource": "${DS_PROMETHEUS}", "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "gridPos": { "h": 3, "w": 4, "x": 4, "y": 0 }, "id": 14, "interval": null, "links": [], "mappingType": 1, "mappingTypes": [ { "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 } ], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [ { "from": "null", "text": "N/A", "to": "null" } ], "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "tableColumn": "", "targets": [ { "expr": "tomcat_requestcount_total{port=\"8080\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "refId": "A" } ], "thresholds": "", "title": "Requestcount Current", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [ { "op": "=", "text": "N/A", "value": "null" } ], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "#299c46", "rgba(237, 129, 40, 0.89)", "#d44a3a" ], "datasource": "${DS_PROMETHEUS}", "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "gridPos": { "h": 3, "w": 4, "x": 8, "y": 0 }, "id": 15, "interval": null, "links": [], "mappingType": 1, "mappingTypes": [ { "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 } ], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [ { "from": "null", "text": "N/A", "to": "null" } ], "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "tableColumn": "", "targets": [ { "expr": "tomcat_requestcount_total{port=\"8080\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "refId": "A" } ], "thresholds": "", "title": "Requestcount Average", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [ { "op": "=", "text": "N/A", "value": "null" } ], "valueName": "avg" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "#299c46", "rgba(237, 129, 40, 0.89)", "#d44a3a" ], "datasource": "${DS_PROMETHEUS}", "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "gridPos": { "h": 3, "w": 4, "x": 12, "y": 0 }, "id": 16, "interval": null, "links": [], "mappingType": 1, "mappingTypes": [ { "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 } ], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [ { "from": "null", "text": "N/A", "to": "null" } ], "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "tableColumn": "", "targets": [ { "expr": "tomcat_errorcount_total{port=\"8080\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "refId": "A" } ], "thresholds": "", "title": "Errorcount", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [ { "op": "=", "text": "N/A", "value": "null" } ], "valueName": "total" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "#299c46", "rgba(237, 129, 40, 0.89)", "#d44a3a" ], "datasource": "${DS_PROMETHEUS}", "format": "bytes", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "gridPos": { "h": 3, "w": 4, "x": 16, "y": 0 }, "id": 18, "interval": null, "links": [], "mappingType": 1, "mappingTypes": [ { "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 } ], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [ { "from": "null", "text": "N/A", "to": "null" } ], "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "tableColumn": "", "targets": [ { "expr": "jvm_memory_bytes_max{area=\"heap\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "refId": "A" } ], "thresholds": "", "title": "Jvm Max Mem heap", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [ { "op": "=", "text": "N/A", "value": "null" } ], "valueName": "avg" }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 9, "w": 12, "x": 0, "y": 3 }, "id": 7, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": false, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_bytes_used{area=\"heap\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "已经使用的堆内存", "refId": "A" }, { "expr": "jvm_memory_bytes_used{area=\"nonheap\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "已经使用的非堆内存", "refId": "B" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Mem Used", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "decbytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 9, "w": 12, "x": 12, "y": 3 }, "id": 8, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": false, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_bytes_committed{area=\"heap\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "已经提交的堆内存", "refId": "A" }, { "expr": "jvm_memory_bytes_committed{area=\"nonheap\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "已经提交的非堆内存", "refId": "B" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Mem Committed", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "decbytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 12 }, "id": 4, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_pool_bytes_committed{pool=\"Code Cache\",instance=~\"$instance\"} ", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "Code Cache", "refId": "A" }, { "expr": "jvm_memory_pool_bytes_committed{pool=\"Metaspace\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "Metaspace", "refId": "B" }, { "expr": "jvm_memory_pool_bytes_committed{pool=\"Compressed Class Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 1, "legendFormat": "Compressed Class Space", "refId": "C" }, { "expr": "jvm_memory_pool_bytes_committed{pool=\"PS Eden Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 1, "legendFormat": "PS Eden Spac", "refId": "D" }, { "expr": "jvm_memory_pool_bytes_committed{pool=\"PS Survivor Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 1, "legendFormat": "PS Survivor Space", "refId": "E" }, { "expr": "jvm_memory_pool_bytes_committed{pool=\"PS Old Gen\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 1, "legendFormat": "PS Old Gen", "refId": "F" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Pool committed", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 22 }, "id": 9, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_pool_bytes_used{pool=\"Code Cache\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "Code Cache", "refId": "A" }, { "expr": "jvm_memory_pool_bytes_used{pool=\"Metaspace\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "Metaspace", "refId": "B" }, { "expr": "jvm_memory_pool_bytes_used{pool=\"Compressed Class Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "Compressed Class Space", "refId": "C" }, { "expr": "jvm_memory_pool_bytes_used{pool=\"PS Eden Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "PS Eden Space", "refId": "D" }, { "expr": "jvm_memory_pool_bytes_used{pool=\"PS Survivor Space\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "PS Survivor Space", "refId": "E" }, { "expr": "jvm_memory_pool_bytes_used{pool=\"PS Old Gen\",instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "PS Old Gen", "refId": "F" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Pool User", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "decbytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 32 }, "id": 17, "legend": { "alignAsTable": true, "avg": true, "current": false, "max": true, "min": true, "rightSide": true, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_gc_collection_seconds_count{gc=\"PS Scavenge\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "PS Scavenge", "refId": "A" }, { "expr": "jvm_gc_collection_seconds_sum{gc=\"PS Scavenge\",instance=~\"$instance\"} ", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "PS Scavenge SUM", "refId": "B" }, { "expr": "jvm_gc_collection_seconds_count{gc=\"PS MarkSweep\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "PS MarkSweep", "refId": "C" }, { "expr": "jvm_gc_collection_seconds_sum{gc=\"PS MarkSweep\",instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 1, "legendFormat": "PS MarkSweep SUM", "refId": "D" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Count Gc", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 42 }, "id": 6, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_threads_current{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "当前的线程连接数", "refId": "A" }, { "expr": "jvm_threads_started_total{instance=~\"$instance\"}", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "线程数总和", "refId": "B" }, { "expr": "jvm_threads_daemon{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "daemon", "refId": "C" }, { "expr": "jvm_threads_peak{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "线程峰值", "refId": "D" }, { "expr": "jvm_threads_deadlocked{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "deadlocked", "refId": "E" }, { "expr": "jvm_threads_deadlocked_monitor{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "deadlocked_monitor", "refId": "F" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "threads", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 52 }, "id": 11, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_classes_loaded{instance=~\"$instance\"}", "format": "time_series", "instant": false, "intervalFactor": 2, "legendFormat": "已经加载的类", "refId": "A" }, { "expr": "jvm_classes_loaded_total{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "加载类的总和", "refId": "B" }, { "expr": "jvm_classes_unloaded_total{instance=~\"$instance\"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "已卸载的类", "refId": "C" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Classes", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROMETHEUS}", "fill": 1, "gridPos": { "h": 7, "w": 24, "x": 0, "y": 62 }, "id": 19, "legend": { "alignAsTable": true, "avg": true, "current": false, "max": true, "min": true, "rightSide": false, "show": true, "total": true, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "alias": "output", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "sum(irate(tomcat_bytesreceived_total{port=\"8080\",protocol=\"http-nio\",instance=~\"$instance\"}[10m]))", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "input", "refId": "A" }, { "expr": "sum(irate(tomcat_bytessent_total{port=\"8080\",protocol=\"http-nio\",instance=~\"$instance\"}[10m]))", "format": "time_series", "hide": false, "intervalFactor": 2, "legendFormat": "output", "refId": "B" } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Network", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "Bps", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "decbytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ] } ],
      "refresh": "30s",
      "schemaVersion": 16,
      "style": "dark",
      "tags": [ "JVM", "tomcat" ],
      "templating": { "list": [ { "allValue": "", "current": {}, "datasource": "${DS_PROMETHEUS}", "hide": 0, "includeAll": true, "label": "Instance", "multi": false, "name": "instance", "options": [], "query": "label_values(jvm_memory_bytes_max,instance)", "refresh": 1, "regex": "", "sort": 0, "tagValuesQuery": "", "tags": [], "tagsQuery": "", "type": "query", "useTags": false } ] },
      "time": { "from": "now-6h", "to": "now" },
      "timepicker": { "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] },
      "timezone": "",
      "title": "Tomcat JVM",
      "uid": "1fpsMEdmz",
      "version": 3 }

[http_ip_9151_metrics_metrics]: http://ip:9151/metrics,/metrics
[https_github.com_prometheus_jmx_exporter_blob_master_example_configs_tomcat.yml]: https://github.com/prometheus/jmx_exporter/blob/master/example_configs/tomcat.yml
[https_github.com_prometheus_jmx_exporter]: https://github.com/prometheus/jmx_exporter