Prometheus 从入门到入土 -----入土

爱被打了一巴掌 2024-04-01 11:06 96阅读 0赞

**目录**

一、Prometheus的部署

1、实验环境

2、环境初始话

3、上传prometheus安装包到opt下，进行安装（192.168.159.68）

4、配置node\_exporter(192.168.159.11、192.168.159.10)

5、配置prometheus节点的配置文件（192.168.159.68）

6、重启prometheus，查看页面

7、采集数据流向

二、配置文件解析

prometeus配置文件prometeus.yml的组成

三、表达式浏览器（prometheusUI控制台）

prometheus UI控制台上可以进行数据过滤

1、CPU使用总量

四、Prometheus discovery 服务发现

1、静态发现（上文自定义的指定抓取目标信息为静态方式）

2、动态发现

2.1、基于文件服务发现

2.1.1、动态发现基于文件服务发现实验部分

3、基于DNS发现

4、基于consul发现

4.1、consul服务部署

五、grafana部署

1、grafana简介

2、部署grafana

六、部署alertmanager，告警对接邮箱

--------------------

## 一、Prometheus的部署 ##

### 1、实验环境 ###

<table style="width:500px;"> 
 <tbody> 
  <tr> 
   <td>主机</td> 
   <td>IP</td> 
   <td>安装包</td> 
  </tr> 
  <tr> 
   <td>prometheus</td> 
   <td>192.168.159.68</td> 
   <td>prometheus</td> 
  </tr> 
  <tr> 
   <td>server1</td> 
   <td>192.168.159.11</td> 
   <td>node_exporter</td> 
  </tr> 
  <tr> 
   <td>server2</td> 
   <td>192.168.159.10</td> 
   <td>node_exporter</td> 
  </tr> 
 </tbody> 
</table>

### 2、环境初始话 ###

> systemctl stop firewalld  
> systemctl disable firewalld  
> setenforce 0
> 
> ntpdate ntp1.aliyun.com \#\#\#\#时间同步，必须要做的

### 3、上传prometheus安装包到opt下，进行安装（192.168.159.68） ###

> \[root@zwb\_docker opt\]\# rz -E  
> rz waiting to receive.  
> \[root@zwb\_docker opt\]\# tar zxvf prometheus-2.27.1.linux-amd64.tar.gz -C /usr/local/

安装完成，启动prometheus

> \[root@zwb\_docker prometheus-2.27.1.linux-amd64\]\# ./prometheus

prometheus启动后会占用前端界面，复制会话，重新打开另一个界面进行操作。

查看端口是否开启

> \[root@zwb\_prometheus ~\]\# ss -antp | grep 9090  
> LISTEN 0 128 :::9090 :::\* users:(("prometheus",pid=11006,fd=8))  
> ESTAB 0 0 ::ffff:192.168.159.68:9090 ::ffff:192.168.159.1:62136 users:(("prometheus",pid=11006,fd=21))  
> ESTAB 0 0 ::1:9090 ::1:51520 users:(("prometheus",pid=11006,fd=13))  
> ESTAB 0 0 ::ffff:192.168.159.68:9090 ::ffff:192.168.159.1:62135 users:(("prometheus",pid=11006,fd=18))

查看WEB页面

> [http://192.168.159.68:9090/][http_192.168.159.68_9090]

![8a7f32a5e473434596d97607f47410a2.png][]

### 4、配置node\_exporter(192.168.159.11、192.168.159.10) ###

两个节点的配置方法一样

> \[root@zwb opt\]\# rz -E \#\#\# 上传  
> rz waiting to receive.  
> \[root@zwb opt\]\# tar zxvf node\_exporter-1.1.2.linux-amd64.tar.gz \#\#\# 解压  
> \-C /usr/local/  
> \[root@zwb opt\]\# cd /usr/local/node\_exporter-1.1.2.linux-amd64/
> 
> \[root@zwb node\_exporter-1.1.2.linux-amd64\]\# mv node\_exporter /ur/bin/
> 
> \#\#\#\#\#\#\#\#\#\#\#\#\#移动执行脚本到全局环境变量下

开启node\_exporter

> \[root@zwb node\_exporter-1.1.2.linux-amd64\]\# node\_exporter

### 5、配置prometheus节点的配置文件（192.168.159.68） ###

![686f552f9fe141e2b859e983a40f523f.png][]

### 6、重启prometheus，查看页面 ###

![83950341d20d4f9f9a9782fb8efd8ba1.png][]

如果整体关机后，开机需要重新启动prometheus和exporter

### 7、采集数据流向 ###

prometheus节点通过收集。被监测点的node\_exporter周期性抓取数据并转化为prometheus兼容格式。node\_exporter被监测点的信息需要在prometheus节点配置。

## 二、配置文件解析 ##

### prometeus配置文件prometeus.yml的组成 ###

> **\#\# 用于定义全局配置，主要定义的是周期。一个是scrape周期，一个自动识别、更新配置文件的周期**
> 
> \# my global config  
> global:  
> scrape\_interval: 15s
> 
> \# Set the scrape interval to every 15 seconds. Default is every 1 minute.
> 
> \# 用来指定Prometheus从监控端抓取数据的时间间隔  
> evaluation\_interval: 15s
> 
> \# Evaluate rules every 15 seconds. The default is every 1 minute.
> 
> \# 用于指定检测告警规则的时间间隔，每15s重新检测告警规则，并对变更进行更新  
> \# scrape\_timeout is set to the global default (10s).
> 
> \# 定义拉取实例指标的超时时间
> 
> **\#\# 用于设置Prometheus与Alertmanager的通信，在Prometheus的整体架构中，Prometheus会根据配置的告警规则触发警报并发送到独立的Alertmanager组件，Alertmanager将对告警进行管理并发送给相关的用户**
> 
> \# Alertmanager configuration  
> alerting:  
> alertmanagers:  
> \- static\_configs: **\#\#\# 配置alertmanager的地址信息**  
> \- targets:  
> \# - alertmanager:9093
> 
> \# Load rules once and periodically evaluate them according to the global 'evaluation\_interval'.  
> rule\_files: **\#\#\# 用于指定告警规则的文件路径，告警逻辑（写了布尔值表达式文件的文件名）**  
> \# - "first\_rules.yml"  
> \# - "second\_rules.yml"
> 
> \# A scrape configuration containing exactly one endpoint to scrape:  
> \# Here it's Prometheus itself.  
> scrape\_configs: **\#\#\# 指定Prometheus抓取的目标信息（采集数据的配置）**  
> \# The job name is added as a label \`job=<job\_name>\` to any timeseries scraped from this config.  
> \- job\_name: 'prometheus'
> 
> \# metrics\_path defaults to '/metrics'  
> \# scheme defaults to 'http'.
> 
> static\_configs:  
> \- targets: \['localhost:9090'\]
> 
> \- job\_name: 'nodes' **\#\#\# 自定义指定Prometheus抓取的目标信息**  
> static\_configs:  
> \- targets:  
> \- 192.168.159.11:9100  
> \- 192.168.159.10:9100

### 三、表达式浏览器（prometheusUI控制台） ###

> [http://192.168.159.68:9090/][http_192.168.159.68_9090]

![ab63580852f947ad84824221761a99e5.png][]

### prometheus UI控制台上可以进行数据过滤 ###

#### 1、CPU使用总量 ####

node\_cpu\_seconds\_total

**进阶1：计算过去5分钟内的CPU使用速率**

**PromQL：irate(node\_cpu\_seconds\_total\{mode="idle"\}\[5m\])**

irate：速率计算函数

node\_cpu\_seconds\_total：node节点CPU使用总量

mode="idle" 空闲指标

5m:过去的5分钟内，所有CPU空闲数的样本值，每个数值做速率运算

## 四、Prometheus discovery 服务发现 ##

prometheus指标抓取的生命周期

发现——》配置——》relabel——》指标数据抓取——》metrics relabel

**发现：**当peometheus启动，启动后发现被监控端

**配置：**定义怎么采集。怎么监控（周期），监控哪些节点

**relabel:**例如exporter采集数据后，需要转换成Pro能兼容识别的格式（时序数据格式），而这种转换就是聚合数据》

**指标数据的抓取：**pro pull数据

**metrics relabel:**重组

### 1、静态发现（上文自定义的指定抓取目标信息为静态方式） ###

> \- job\_name: 'nodes' **\#\#\# 自定义指定Prometheus抓取的目标信息**  
> static\_configs:  
> \- targets:  
> \- 192.168.159.11:9100  
> \- 192.168.159.10:9100

### 2、动态发现 ###

#### 2.1、基于文件服务发现 ####

基于文件的服务发现仅仅略优于静态配置的服务发现方式，它不依赖于任何平台或第三方服务，  
因而也是最为简单和通用的实现方式。

#### 2.1.1、动态发现基于文件服务发现实验部分 ####

> \#\# 进入普罗米修斯的工作目录
> 
> \[root@zwb\_prometheus prometheus\]\# mkdir file\_sd \#\# 创建file\_sd目录  
> \[root@zwb\_prometheus prometheus\]\# ls  
> console\_libraries consoles data file\_sd LICENSE nohup.out NOTICE prometheus prometheus.yml promtool
> 
> \[root@zwb\_prometheus prometheus\]\# cd file\_sd/  
> \[root@zwb\_prometheus file\_sd\]\# ls  
> \[root@zwb\_prometheus file\_sd\]\# pwd  
> /usr/local/prometheus/file\_sd  
> \[root@zwb\_prometheus file\_sd\]\# mkdir targets \#\#\# 创建 targets目录  
> \[root@zwb\_prometheus file\_sd\]\# ls  
> targets
> 
> \#\#\# 编辑3个配置文件
> 
> \#\# 编辑 prometheus.yml文件

> \[root@zwb\_prometheus file\_sd\]\# vim /usr/local/prometheus/file\_sd/prometheus.yml

[root@zwb_prometheus file_sd]# cat prometheus.yml 
    # my global config
    # Author: MageEdu <mage@magedu.com>
    # Repo: http://gitlab.magedu.com/MageEdu/prometheus-configs/
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
    
    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093
    
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
    
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'                    ### 指定job为prometheus
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        file_sd_configs:                          ### 将原先的静态改为文件发现
        - files:                                               
          - targets/prometheus_*.yaml             ### 指定加载的文件位置
          refresh_interval: 2m                    ### 2分钟刷新一次
    
      # All nodes 
      - job_name: 'nodes'                         ### 指定job为nodes
        file_sd_configs:   
        - files:                                               
          - targets/nodes_*.yaml                  ### 指定加载的文件位置
          refresh_interval: 2m                    ### 2分钟刷新一次

> \[root@zwb\_prometheus file\_sd\]\# vim /usr/local/prometheus/file\_sd/targets/prometheus\_servers.yaml

- targets:                     
      - 192.168.159.68:9090                ### 指定prometheus的位置
      labels:
        app: prometheus                    
        job: prometheus                    ### 定义job为prometheus

> \[root@zwb\_prometheus file\_sd\]\# vim /usr/local/prometheus/file\_sd/targets/nodes\_linux.yaml

- targets:                               ### 指定nodes的位置
      - 192.168.159.10:9100             
      - 192.168.159.11:9100
      labels:
        app: node-exporter
        job: nodes                           ###指定job为nodes

> \#\#\# 新建目录和文件的存放位置关系

> root@zwb\_prometheus file\_sd\]\# pwd  
> /usr/local/prometheus/file\_sd  
> \[root@zwb\_prometheus file\_sd\]\# tree ./  
> ./  
> ├── prometheus.yml  
> └── targets  
> ├── nodes\_linux.yaml  
> └── prometheus\_servers.yaml
> 
> 1 directory, 3 files

> 启动prometheus
> 
> \[root@zwb\_prometheus prometheus\]\# pwd \#\#\#当前位置  
> /usr/local/prometheus  
> \[root@zwb\_prometheus prometheus\]\# ./prometheus --config.file=./file\_sd/prometheus.yml
> 
> \#\#\# 通过指定启动文件的位置来启动，不让其启动时加载原先得配置文件

**实验结构和配置文件自行对比；**

3、![1160bc3ff21e46908930e60a2692dfa7.png][]

### 3、基于DNS发现 ###

基于DNS的服务发现针对一组DNS域名进行定期查询，以发现待监控的目标查询时使用的DNS服务器由/etc/resolv.conf文件指定：该发现机制依赖于A、AAAA和SRv资源记录，且仅支持该类方法，尚不支持RFC6763中的高级DNS发现方式。

### 4、基于consul发现 ###

consul是一款基于golang开发的开源工具，主要面向分布式，服务化的系统提供服务注册、服务一发现和配置管理的功能提供服务注册/发现、健康检查、Key/Value存储、多数据中心和分布式一致性保证等功能

原理:通过定义json文件将可以进行数据采集的服务注册到consul中，用于自动发现同时使用prometheus做为client端获取consul上注册的服务，从而进行获取数据

#### 4.1、consul服务部署 ####

第一步：上传安装包（安装于prometheus主机上）

> \[root@zwb\_prometheus opt\]\# rz -E  
> rz waiting to receive.  
> \[root@zwb\_prometheus opt\]\# ls  
> consul\_1.9.0\_linux\_amd64.zip

第二步：解压安装包

> \[root@zwb\_prometheus opt\]\# unzip consul\_1.9.0\_linux\_amd64.zip

[root@zwb_prometheus opt]# unzip consul_1.9.0_linux_amd64.zip    ## 解压
    Archive:  consul_1.9.0_linux_amd64.zip
      inflating: consul                  
    [root@zwb_prometheus opt]# ls          
    consul  consul_1.9.0_linux_amd64.zip  containerd  lnmp.sh  mysql-5.7.20  nginx-1.15.9  php-7.1.10  rh
    [root@zwb_prometheus opt]# mv consul /usr/bin/         ## 加入到环境变量

第三步：配置consul的配置文件

> \[root@zwb\_prometheus opt\]\# mkdir -pv /etc/consul/data  
> mkdir: 已创建目录 "/etc/consul"  
> mkdir: 已创建目录 "/etc/consul/data"  
> \[root@zwb\_prometheus opt\]\# cd /etc/consul/

第四步：启动consul

> \[root@zwb\_prometheus consul\]\# consul agent \\  
> > -dev \\  
> > -ui \\  
> > -data-dir=/consul/data/ \\  
> > -config-dir=/etc/consul/ \\  
> > -client=0.0.0.0 &  
> 注释：
> 
> agent -dev:运行开发模式  
> agent -server：运行server模式  
> \-ui：ui界面  
> \-data-dir：数据位置  
> /etc/consul:可以以文件形式定义各个services的配置，也可以基于api接口直接配置  
> \-client：监听地址

![b18d52fda09c459599f494f9cbe73b81.png][]

第五步：创建/etc/consul目录下的prometheus-servers.json配置文件

[root@zwb_prometheus consul]# vim /etc/consul/prometheus-servers.json
    
    {
      "services": [
        {
          "id": "prometheus-server-node01",
          "name": "prom-server-node01",
          "address": "192.168.159.68",
          "port": 9090,
          "tags": ["prometheus"],
          "checks": [{
            "http": "http://192.168.159.68:9090/metrics",
            "interval": "5s"

第六步：重载consul配置文件

> \[root@zwb\_prometheus consul\]\# consul reload

![751a75d392e143f4a996f0bba280c762.png][]第七步：创建consul自动发现的prometheus.yml文件

[root@zwb_prometheus console_sd]# vim prometheus.yml 
    # my global config
    # Author: MageEdu <mage@magedu.com>
    # Repo: http://gitlab.magedu.com/MageEdu/prometheus-configs/
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
    
    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093
    
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
    
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        consul_sd_configs:
        - server: "192.168.159.68:8500"
          tags:
          - "prometheus"
          refresh_interval: 2m 
    
      # All nodes
      - job_name: 'nodes'
        consul_sd_configs:
        - server: "192.168.159.68:8500"
          tags:
          - "nodes"
          refresh_interval: 2m

第八步：重启prometheus和consul

> \[root@zwb\_prometheus console\_sd\]\# killall prometheus
> 
> \[root@zwb\_prometheus prometheus\]\# ./prometheus --config.file=./consul\_sd/prometheus.yml \#\#\# 指定上面配置的prometheus.yml启动时加载

第九步：注册其他node节点在Pro节点下consul的配置文件下配置

[root@zwb_prometheus consul_sd]# cd /etc/consul/
    [root@zwb_prometheus consul]# vim nodes.json
    {
      "services": [
        {
          "id": "node_exporter-node01",
          "name": "node01",
          "address": "192.168.159.11",
          "port": 9100,
          "tags": ["nodes"],
          "checks": [{
            "http": "http://192.168.226.11:9100/metrics",
            "interval": "5s"
          }]
        },
        {
          "id": "node_exporter-node02",
          "name": "node02",
          "address": "192.168.159.10",
          "port": 9100,
          "tags": ["nodes"],
          "checks": [{
            "http": "http://192.168.159.10:9100/metrics",
            "interval": "5s"
          }]
        },
        {
          "id": "node_exporter-node03",
          "name": "node03",
          "address": "192.168.159.68",
          "port": 9100,
          "tags": ["nodes"],
          "checks": [{
            "http": "http://192.168.159.68:9100/metrics",
            "interval": "5s"
          }]
        }
      ]
    }

![bec8f4e8e6934a68a054c14b26a011a1.png][]

## 五、grafana部署 ##

### 1、grafana简介 ###

grafana是一款基于go语言开发的通用可视化工具，支持从不同的数据源加载并展示数据，可作为其数据源的部分储存系统如下所示：

TSDB：Prometheus、IfluxDB、OpenTSDB和Graphit  
日志和文档存储：Loki和ElasitchSearch  
分布式请求跟踪：Zipkin、Jaeger和Tenpo  
SQL DB：Mysql、PostgreSQL和Microsoft SQL server  
grafana基础默认监听于TCP协议的3000端口，支持集成其他认证服务，且能够通过/metrics输出内建指标；

数据源（Data Source）：提供用于展示的数据的储存系统  
仪表盘（Dashboard）：组织和管理数据的可视化面板（Panel）  
团队和用户：提供了面向企业组织层级的管理能力；

### 2、部署grafana ###

第一步：上传安装包

> \[root@server1 opt\]\# rz -E  
> rz waiting to receive.

第二步：第一种方法：直接在虚拟机环境安装。第二种方法：跑容器中

本实验中直接安装虚拟机环境中

> \[root@server1 opt\]\# yum -y install grafana-7.3.6-1.x86\_64.rpm  
> \#\#\#\# yum 指定本地安装包，他在安装的同时会自动升级

第三步：开启grafana 服务

> \[root@server1 opt\]\# systemctl start grafana-server.service
> 
> \[root@server1 opt\]\# vim /etc/grafana/grafana.ini
> 
> \#\#\# 默认用户admin，密码admin。在配置文件中可修改密码

![a75405178cb5417dabefab6cbcc93559.png][]

**打开web:**

![dc7e28b38eba42cea8c4eb39395cfe19.png][]

第四步：创建监控的展现仪表盘

![64bc58ca77d2436d8258743817e0fbd9.png][]![a1284c0d285b416fa278b4092bda9337.png][]

![6e98cb0476ec4096bfee6d3bbe3267fa.png][]

点击 ![b2931fb2c4d54de79594ae2eb4b8918b.png][]

![42e00d43353545e99fa1a0343529c471.png][]![557469cf5156463fb044e0f71465237e.png][]![edd9b4a885914bd28ba3d0afc9579430.png][]

等待几分钟。会自行跳转

![3e7a95cc9a414f3885eea027fe0999ab.png][]

结果：

![c9be2fdad21148a5bab90042ab23121f.png][]

## 六、部署alertmanager，告警对接邮箱 ##

第一步：安装在prometheus客户端处

> \[root@zwb\_prometheus opt\]\# rz -E  
> rz waiting to receive.  
> \[root@zwb\_prometheus opt\]\# tar zxvf alertmanager-0.22.2.linux-amd64.tar.gz -C /usr/local/

> \[root@zwb\_prometheus local\]\# mv alertmanager-0.22.2.linux-amd64/ alertmanager
> 
> \#\#\# 修改目录名
> 
> \#\#\# 查看配置文件
> 
> cat /usr/local/alertmanager/alertmanager.yml  
> route: \#路由信息  
> group\_by: \['alertname'\] \#分组  
> group\_wait: 30s \#分组缓冲/等待时间  
> group\_interval: 5m \#重新分组时间  
> repeat\_interval: 1h \#重新告警间隔  
> receiver: 'web.hook' \#接收方/媒介  
> receivers:  
> \- name: 'web.hook'  
> webhook\_configs:  
> \- url: 'http://127.0.0.1:5001/' \#标注5001端口  
> inhibit\_rules: \#抑制规则的策略  
> \- source\_match: \#匹配项  
> severity: 'critical' \#严重的级别  
> target\_match:  
> severity: 'warning' \#target匹配warning级别  
> equal: \['alertname', 'dev', 'instance'\] \#符合alertname、dev、instance

第二步：邮箱设置

![1db2616369774ca4be1095d3c98b8813.png][]

**往下翻**

![fd60a583e4664f73b25a48d11cc206e4.png][]

![4e01f1ec6a2a4adbb9a34b28db831343.png][]

![520dfd996bf94e5089a2ac7c5aa034db.png][] 记录授权码

第三步：备份配置文件、配置alertmanager.yml文件

> \[root@zwb\_prometheus alertmanager\]\# mv alertmanager.yml alertmanager.yml.bck

![7010afd7f37e47b9ab56ccd8fb948fac.png][]

第四步：启动alertmanager

> \[root@zwb\_prometheus alertmanager\]\# ./alertmanager

第五步：定义告警发送条件

> \[root@zwb\_prometheus prometheus\]\# pwd  
> /usr/local/prometheus
> 
> \#\#\#\# 定义一个目录用于配置告警促发条件  
> \[root@zwb\_prometheus prometheus\]\# mkdir alert-config  
> \[root@zwb\_prometheus prometheus\]\# cd alert-config/  
> \[root@zwb\_prometheus alert-config\]\# mkdir alert\_rules targets
> 
> \[root@zwb\_prometheus alert-config\]\# ls  
> alert\_rules targets  
> \[root@zwb\_prometheus alert-config\]\# vim instance\_down.yml  
> \[root@zwb\_prometheus alert-config\]\# cd alert\_rules/
> 
> \#\#\#\# 定义告警条件
> 
> \[root@zwb\_prometheus alert\_rules\]\# vim instance\_down.yml
> 
> groups:  
> \- name: AllInstances  
> rules:  
> \- alert: InstanceDown \#节点服务挂掉  
> \# Condition for alerting  
> expr: up == 0 \#up状态为0时  
> for: 1m  
> \# Annotation - additional informational labels to store mo  
> annotations:  
> title: 'Instance down'  
> description: Instance has been down for more than 1 minu  
> \# Labels - additional labels to be attached to the alert  
> labels:  
> severity: 'critical' \#告警级别
> 
> \[root@zwb\_prometheus alert\_rules\]\# cd ../targets/
> 
> \[root@zwb\_prometheus targets\]\# vim alertmanagers.yaml
> 
> \- targets:  
> \- 192.168.159.68:9093  
> labels:  
> app: alertmanager
> 
> \[root@zwb\_prometheus targets\]\# vim nodes-linux.yaml
> 
> \- targets:  
> \- 192.168.159.68:9100  
> \- 192.168.159.10:9100  
> \- 192.168.159.11:9100  
> labels:  
> app: node-exporter  
> job: node
> 
> \[root@zwb\_prometheus targets\]\# vim prometheus-servers.yaml
> 
> \- targets:  
> \- 192.168.159.68:9090  
> labels:  
> app: prometheus  
> job: prometheus
> 
> \[root@zwb\_prometheus alert-config\]\# pwd  
> /usr/local/prometheus/alert-config
> 
> \#\#\# 定义启动prometheus
> 
> \[root@zwb\_prometheus alert-config\]\# vim prometheus.yml
> 
> \# my global config  
> \# Author: MageEdu <mage@magedu.com>  
> \# Repo: http://gitlab.magedu.com/MageEdu/prometheus-configs/  
> global:  
> scrape\_interval: 15s \# Set the scrape interval to every 15 seconds. Default is every 1 minute.  
> evaluation\_interval: 15s \# Evaluate rules every 15 seconds. The default is every 1 minute.  
> \# scrape\_timeout is set to the global default (10s).
> 
> \# Alertmanager configuration  
> alerting:  
> alertmanagers:  
> \- file\_sd\_configs:  
> \- files:  
> \- "targets/alertmanagers\*.yaml"
> 
> \# Load rules once and periodically evaluate them according to the global 'evaluation\_interval'.  
> rule\_files:  
> \- "rules/\*.yaml"  
> \- "alert\_rules/\*.yaml"
> 
> \# A scrape configuration containing exactly one endpoint to scrape:  
> \# Here it's Prometheus itself.  
> scrape\_configs:  
> \# The job name is added as a label \`job=<job\_name>\` to any timeseries scraped from this config.  
> \- job\_name: 'prometheus'  
> \# metrics\_path defaults to '/metrics'  
> \# scheme defaults to 'http'.
> 
> file\_sd\_configs:  
> \- files:  
> \- targets/prometheus-\*.yaml  
> refresh\_interval: 2m
> 
> \# All nodes  
> \- job\_name: 'nodes'  
> file\_sd\_configs:  
> \- files:  
> \- targets/nodes-\*.yaml  
> refresh\_interval: 2m
> 
> \- job\_name: 'alertmanagers'  
> file\_sd\_configs:  
> \- files:  
> \- targets/alertmanagers\*.yaml  
> refresh\_interval: 2m
> 
> \[root@zwb\_prometheus prometheus\]\# ./prometheus --config.file=./alert-config/prometheus.yml \#\#\# 再pro工作目录下启动

第六步：模拟单节点宕机

> killall node\_exporter
> 
> \[root@server2 ~\]\# ss -antp | grep node\_exporter  
> \[root@server2 ~\]\# \#\#端口已经关闭

第七步：验证

查看PrometheusUI

![2c6853f7f0014b9dbd4712389cf0a9cd.png][]**查看邮箱**

![7fdc515644ca4b518fe6f073ed74ec9e.png][]

![e7e5a33cc12444c9a51f3d2d4921ca18.png][]

[http_192.168.159.68_9090]: http://192.168.159.68:9090/
[8a7f32a5e473434596d97607f47410a2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/a9b3c8fe0ac44ae7bf4996891433d2ce.png
[686f552f9fe141e2b859e983a40f523f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/c437be1884074bbf80c5faca9d8b9d90.png
[83950341d20d4f9f9a9782fb8efd8ba1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/942cdd35e2c34aa3945b6d13c8a1087e.png
[ab63580852f947ad84824221761a99e5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/3c7cb4dd6f5b4fc388b8ec766c7d8285.png
[1160bc3ff21e46908930e60a2692dfa7.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/6d87ef3a06db47838b5e40c1f51b606d.png
[b18d52fda09c459599f494f9cbe73b81.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/6cef9c1b9fa846f1b2accf527ca47468.png
[751a75d392e143f4a996f0bba280c762.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/7644d930e0d441d79659916b6b358229.png
[bec8f4e8e6934a68a054c14b26a011a1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/dd701b17a1f8464ea2b8f8b2bf640137.png
[a75405178cb5417dabefab6cbcc93559.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/23fe4625b249452583555b38ec991305.png
[dc7e28b38eba42cea8c4eb39395cfe19.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/b043d3dae47c42a3ac151ffc24f9acd3.png
[64bc58ca77d2436d8258743817e0fbd9.png]: https://img-blog.csdnimg.cn/64bc58ca77d2436d8258743817e0fbd9.png
[a1284c0d285b416fa278b4092bda9337.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/685916c12b4e41248bb0df97b99dbdb5.png
[6e98cb0476ec4096bfee6d3bbe3267fa.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/164faf37e1ab476eb0b1b35f0cb47939.png
[b2931fb2c4d54de79594ae2eb4b8918b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/7e0bd52136da4f009c49214edbaf087b.png
[42e00d43353545e99fa1a0343529c471.png]: https://img-blog.csdnimg.cn/42e00d43353545e99fa1a0343529c471.png
[557469cf5156463fb044e0f71465237e.png]: https://img-blog.csdnimg.cn/557469cf5156463fb044e0f71465237e.png
[edd9b4a885914bd28ba3d0afc9579430.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/6ad79052739e4dbfb01608e02696d637.png
[3e7a95cc9a414f3885eea027fe0999ab.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/4f1a6769eee74088bc9b9e8aafc5373f.png
[c9be2fdad21148a5bab90042ab23121f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/152429672f4e4ed4892ec4b3002666cd.png
[1db2616369774ca4be1095d3c98b8813.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/2d883c203d5249aba72c91b5bbb27efa.png
[fd60a583e4664f73b25a48d11cc206e4.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/e9cd3a69b1ce4cdfb443c81ab2a4342a.png
[4e01f1ec6a2a4adbb9a34b28db831343.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/f340e08f00f14593bdd7eb42d9de374f.png
[520dfd996bf94e5089a2ac7c5aa034db.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/e0fb46323f73483799b31b3f4b9707a6.png
[7010afd7f37e47b9ab56ccd8fb948fac.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/583aac3a9bf04e50a8634f485843db5f.png
[2c6853f7f0014b9dbd4712389cf0a9cd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/925e933a493740468602cbc4cda51005.png
[7fdc515644ca4b518fe6f073ed74ec9e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/8c460cfb28aa470bae780f722df3e521.png
[e7e5a33cc12444c9a51f3d2d4921ca18.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/01/9eeb6578c657483586c5107f3909cd53.png