prometheus 监控之 elasticsearch5.x

╰半橙微兮° 2022-04-05 13:42 287阅读 0赞

#### 安装监控插件 ####

项目地址：[https://github.com/justwatchcom/elasticsearch\_exporter][https_github.com_justwatchcom_elasticsearch_exporter]  
插件下载地址：[https://github.com/justwatchcom/elasticsearch\_exporter/releases/download/v1.0.4rc1/elasticsearch\_exporter-1.0.4rc1.linux-amd64.tar.gz][https_github.com_justwatchcom_elasticsearch_exporter_releases_download_v1.0.4rc1_elasticsearch_exporter-1.0.4rc1.linux-amd64.tar.gz]  
简单介绍下安装：

[xxx@elk-node1 ~]$ tar zxvf elasticsearch_exporter-1.0.4rc1.linux-amd64.tar.gz
    [xxx@elk-node1 ~]$ cd elasticsearch_exporter-1.0.4rc1.linux-amd64/ 
    [xxx@elk-node1 ~]$ cp elasticsearch_exporter  /usr/local/bin
    [xxx@elk-node1 ~]$ /usr/local/bin/elasticsearch_exporter --web.listen-address "0.0.0.0:9109"  --es.uri http://elastic:changeme@localhost:9200
    [xxx@elk-node1 ~]$ curl localhost:9109/metrics

配置prometheus

- job_name: 'elasticsearch'
      file_sd_configs:
      - refresh_interval: 1m
        files:
        - ./conf.d/es*.json

配置consul自动发现请参考：[我的另一篇博客][Link 1]

#### 指标解析 ####

参考：  
[https://grafana.com/dashboards/2322][https_grafana.com_dashboards_2322]  
[https://yq.aliyun.com/articles/548354][https_yq.aliyun.com_articles_548354]  
[https://blog.csdn.net/u013613428/article/details/78179430][https_blog.csdn.net_u013613428_article_details_78179430]

<table> 
 <thead> 
  <tr> 
   <th>指标</th> 
   <th>解析</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>##搜索和索引性能</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_search_query_total</td> 
   <td>查询总数 吞吐量</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_search_query_time_seconds</td> 
   <td>查询总时间 性能</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_search_fetch_total</td> 
   <td>提取总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_search_fetch_time_seconds</td> 
   <td>花费在提取上的总时间</td> 
  </tr> 
  <tr> 
   <td>##索引请求</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_indexing_index_total</td> 
   <td>索引的文件总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_indexing_index_time_seconds_total</td> 
   <td>索引文档总时间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_indexing_delete_total</td> 
   <td>索引的文件删除总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_indexing_delete_time_seconds_total</td> 
   <td>索引的文件删除总时间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_refresh_total</td> 
   <td>索引刷新总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_refresh_time_seconds_total</td> 
   <td>刷新指数的总时间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_flush_total</td> 
   <td>索引刷新总数到磁盘</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_flush_time_seconds</td> 
   <td>将索引刷新到磁盘上的总时间 累计flush时间</td> 
  </tr> 
  <tr> 
   <td>##JVM内存和垃圾回收</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_gc_collection_seconds_sum</td> 
   <td>GC run time in seconds垃圾回收时间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_gc_collection_seconds_count</td> 
   <td>Count of JVM GC runs垃圾搜集数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_committed_bytes</td> 
   <td>JVM memory currently committed by area最大使用内存限制</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_max_bytes</td> 
   <td>配置的最大jvm值</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_pool_max_bytes</td> 
   <td>JVM内存最大池数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_pool_peak_max_bytes</td> 
   <td>最大的JVM内存峰值</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_pool_peak_used_bytes</td> 
   <td>池使用的JVM内存峰值</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_pool_used_bytes</td> 
   <td>目前使用的JVM内存池</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_jvm_memory_used_bytes</td> 
   <td>JVM memory currently used by area 内存使用量</td> 
  </tr> 
  <tr> 
   <td>##集群健康和节点可用性</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_status</td> 
   <td>集群状态，green（ 所有的主分片和副本分片都正常运行）、yellow（所有的主分片都正常运行，但不是所有的副本分片都正常运行）red（有主分片没能正常运行）值为1的即为对应状态</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_number_of_data_nodes</td> 
   <td>node节点的数量</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_number_of_in_flight_fetch</td> 
   <td>正在进行的碎片信息请求的数量</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_number_of_nodes</td> 
   <td>集群内所有的节点</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_number_of_pending_tasks</td> 
   <td>尚未执行的集群级别更改</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_initializing_shards</td> 
   <td>正在初始化的分片数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_unassigned_shards</td> 
   <td>未分配分片数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_active_primary_shards</td> 
   <td>活跃的主分片总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_active_shards</td> 
   <td>活跃的分片总数（包括复制分片）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_relocating_shards</td> 
   <td>当前节点正在迁移到其他节点的分片数量，通常为0，集群中有节点新加入或者退出时该值会增加</td> 
  </tr> 
  <tr> 
   <td>##资源饱和度</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_thread_pool_completed_count</td> 
   <td>线程池操作完成（bulk、index、search、force_merge）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_thread_pool_active_count</td> 
   <td>线程池线程活动（bulk、index、search、force_merge）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_thread_pool_largest_count</td> 
   <td>线程池最大线程数（bulk、index、search、force_merge）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_thread_pool_queue_count</td> 
   <td>线程池中的排队线程数（bulk、index、search、force_merge）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_thread_pool_rejected_count</td> 
   <td>线程池的被拒绝线程数（bulk、index、search、force_merge）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_fielddata_memory_size_bytes</td> 
   <td>fielddata缓存的大小（字节）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_fielddata_evictions</td> 
   <td>来自fielddata缓存的驱逐次数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_filter_cache_evictions</td> 
   <td>来自过滤器缓存的驱逐次数（仅版本2.x）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_filter_cache_memory_size_bytes</td> 
   <td>过滤器高速缓存的大小（字节）（仅版本2.x）</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_cluster_health_number_of_pending_tasks</td> 
   <td>待处理任务数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_time_seconds</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_missing_total</td> 
   <td>丢失的文件的GET请求总数</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_missing_time_seconds</td> 
   <td>花费在文档丢失的GET请求上的总时间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_exists_time_seconds</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_exists_total</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_indices_get_total</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>#主机级别的系统和网络指标</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>elasticsearch_process_cpu_percent</td> 
   <td>Percent CPU used by process CPU使用率</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_filesystem_data_free_bytes</td> 
   <td>Free space on block device in bytes 磁盘可用空间</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_process_open_files_count</td> 
   <td>Open file descriptors ES进程打开的文件描述符</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_transport_rx_packets_total</td> 
   <td>Count of packets receivedES节点之间网络入流量</td> 
  </tr> 
  <tr> 
   <td>elasticsearch_transport_tx_packets_total</td> 
   <td>Count of packets sentES节点之间网络出流量</td> 
  </tr> 
 </tbody> 
</table>

#### 预警rules ####

groups:
    - name: elasticsearchStatsAlert
      rules:
      - alert: Elastic_Cluster_Health_RED
        expr: elasticsearch_cluster_health_status{ color="red"}==1 
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance { { $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster { { $labels.cluster }}"
          description: "Instance { { $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster { { $labels.cluster }}."
      - alert: Elastic_Cluster_Health_Yellow 
        expr: elasticsearch_cluster_health_status{ color="yellow"}==1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " Instance { { $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster { { $labels.cluster }}" 
          description: "Instance { { $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster { { $labels.cluster }}."
      - alert: Elasticsearch_JVM_Heap_Too_High
        expr: elasticsearch_jvm_memory_used_bytes{ area="heap"} / elasticsearch_jvm_memory_max_bytes{ area="heap"} > 0.8
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "ElasticSearch node { { $labels.instance }} heap usage is high "
          description: "The heap in { { $labels.instance }} is over 80% for 15m."
      - alert: Elasticsearch_health_up
        expr: elasticsearch_cluster_health_up !=1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " ElasticSearch node: { { $labels.instance }} last scrape of the ElasticSearch cluster health failed"                               
          description: "ElasticSearch node: { { $labels.instance }} last scrape of the ElasticSearch cluster health failed"
      - alert: Elasticsearch_Too_Few_Nodes_Running
        expr: elasticsearch_cluster_health_number_of_nodes < 12
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "There are only { { $value}} < 12 ElasticSearch nodes running "                               
          description: "lasticSearch running on less than 12 nodes(total 14)"
      - alert: Elasticsearch_Count_of_JVM_GC_Runs
        expr: rate(elasticsearch_jvm_gc_collection_seconds_count{ }[5m])>5
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "ElasticSearch node { { $labels.instance }}: Count of JVM GC runs > 5 per sec and has a value of { { $value }} "
          description: "ElasticSearch node { { $labels.instance }}: Count of JVM GC runs > 5 per sec and has a value of { { $value }}"
      - alert: Elasticsearch_GC_Run_Time
        expr: rate(elasticsearch_jvm_gc_collection_seconds_sum[5m])>0.3
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " ElasticSearch node { { $labels.instance }}: GC run time in seconds > 0.3 sec and has a value of { { $value }}"
          description: "ElasticSearch node { { $labels.instance }}: GC run time in seconds > 0.3 sec and has a value of { { $value }}"
      - alert: Elasticsearch_json_parse_failures
        expr: elasticsearch_cluster_health_json_parse_failures>0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " ElasticSearch node { { $labels.instance }}: json parse failures > 0 and has a value of { { $value }}"
          description: "ElasticSearch node { { $labels.instance }}: json parse failures > 0 and has a value of { { $value }}"
      - alert: Elasticsearch_breakers_tripped
        expr: rate(elasticsearch_breakers_tripped{ }[5m])>0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " ElasticSearch node { { $labels.instance }}: breakers tripped > 0 and has a value of { { $value }}"
          description: "ElasticSearch node { { $labels.instance }}: breakers tripped > 0 and has a value of { { $value }}"
      - alert: Elasticsearch_health_timed_out
        expr: elasticsearch_cluster_health_timed_out>0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: " ElasticSearch node { { $labels.instance }}: Number of cluster health checks timed out > 0 and has a value of { { $value }}"
          description: "ElasticSearch node { { $labels.instance }}: Number of cluster health checks timed out > 0 and has a value of { { $value }}"

prometheus 配置：

rule_files:
      - /usr/local/prometheus/rules/es*.rules

#### Grafana 画图 ####

模版：链接:[https://pan.baidu.com/s/1mAtVhko18gD4LxdSkuCGEg][https_pan.baidu.com_s_1mAtVhko18gD4LxdSkuCGEg] 密码:3mtd  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]

如对你有所帮助 记得点赞呦 ^ \_ ^！

[https_github.com_justwatchcom_elasticsearch_exporter]: https://github.com/justwatchcom/elasticsearch_exporter
[https_github.com_justwatchcom_elasticsearch_exporter_releases_download_v1.0.4rc1_elasticsearch_exporter-1.0.4rc1.linux-amd64.tar.gz]: https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.0.4rc1/elasticsearch_exporter-1.0.4rc1.linux-amd64.tar.gz
[Link 1]: https://blog.csdn.net/qq_25934401/article/details/82118008
[https_grafana.com_dashboards_2322]: https://grafana.com/dashboards/2322
[https_yq.aliyun.com_articles_548354]: https://yq.aliyun.com/articles/548354
[https_blog.csdn.net_u013613428_article_details_78179430]: https://blog.csdn.net/u013613428/article/details/78179430
[https_pan.baidu.com_s_1mAtVhko18gD4LxdSkuCGEg]: https://pan.baidu.com/s/1mAtVhko18gD4LxdSkuCGEg
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]: /images/20220405/6bd96c4eaea54fd6a5f91e4479035596.png