Prometheus 监控告警配置

目录

  1. Prometheus 架构概述
  2. Prometheus 安装与配置
  3. 数据抓取配置
  4. Alertmanager 配置
  5. 告警规则编写
  6. Grafana 可视化
  7. 常用 Exporter
  8. 服务发现
  9. 高可用配置

1. Prometheus 架构概述

1.1 核心组件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Prometheus Server          # 数据采集、存储、查询
├── TSDB                  # 时序数据库
├── Retrieval             # 数据抓取
└── HTTP Server           # API 服务

Exporters                 # 数据导出器
├── Node Exporter         # 主机指标
├── MySQL Exporter        # 数据库指标
├── Redis Exporter        # 缓存指标
└── ...

Alertmanager              # 告警管理
├── 去重
├── 分组
└── 路由

Pushgateway               # 推送网关（用于短期任务）

Grafana                   # 可视化展示

1.2 数据模型

1
2
3
4
5
指标名称{标签名=标签值, ...}

示例：
http_requests_total{method="POST", handler="/api/users", status="200"}
node_cpu_seconds_total{cpu="0", mode="idle"}

2. Prometheus 安装与配置

2.1 二进制安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 下载 Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.40.0/prometheus-2.40.0.linux-amd64.tar.gz
tar xzf prometheus-2.40.0.linux-amd64.tar.gz
cd prometheus-2.40.0.linux-amd64

# 创建用户
useradd -r -s /sbin/nologin prometheus
mkdir -p /etc/prometheus /var/lib/prometheus
chown -R prometheus:prometheus /var/lib/prometheus

# 复制文件
cp prometheus promtool /usr/local/bin/
cp -r consoles/ console_libraries/ /etc/prometheus/
cp prometheus.yml /etc/prometheus/

2.2 Systemd 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --storage.tsdb.retention.time=15d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target
1
2
3
4
5
# 启动 Prometheus
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
systemctl status prometheus

2.3 基础配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s      # 抓取间隔
  evaluation_interval: 15s  # 规则评估间隔
  external_labels:
    monitor: 'prometheus-monitor'
    environment: 'production'

# 告警管理器
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

# 规则文件
rule_files:
  - /etc/prometheus/rules/*.yml

# 抓取配置
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

3. 数据抓取配置

3.1 静态配置

1
2
3
4
5
6
7
8
9
10
scrape_configs:
  - job_name: 'web-servers'
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']
        labels:
          env: 'production'
          team: 'infra'
      - targets: ['192.168.1.20:9100']
        labels:
          env: 'staging'

3.2 动态配置

1
2
3
4
5
6
7
8
9
10
scrape_configs:
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:9100'
        target_label: __address__
        action: replace

3.3 抓取参数

1
2
3
4
5
6
7
8
9
10
11
12
scrape_configs:
  - job_name: 'slow-service'
    scrape_interval: 30s      # 覆盖全局配置
    scrape_timeout: 10s       # 抓取超时
    metrics_path: /metrics    # 指标路径
    scheme: https             # HTTP/HTTPS
    static_configs:
      - targets: ['service:8080']
    tls_config:
      ca_file: /etc/ssl/certs/ca.crt
      cert_file: /etc/ssl/certs/client.crt
      key_file: /etc/ssl/certs/client.key

3.4 重标签配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
scrape_configs:
  - job_name: 'webapp'
    static_configs:
      - targets: ['webapp:8080']
    relabel_configs:
      # 保留特定标签
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: webapp
        action: keep
      
      # 替换标签
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      
      # 添加标签
      - target_label: monitored_by
        replacement: prometheus
      
      # 删除标签
      - regex: __meta_kubernetes_pod_label_.+
        action: labeldrop

4. Alertmanager 配置

4.1 安装 Alertmanager

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# 下载 Alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
tar xzf alertmanager-0.25.0.linux-amd64.tar.gz
cd alertmanager-0.25.0.linux-amd64

# 安装
cp alertmanager amtool /usr/local/bin/
mkdir -p /etc/alertmanager /var/lib/alertmanager

# Systemd 配置
cat > /etc/systemd/system/alertmanager.service << EOF
[Unit]
Description=Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=0.0.0.0:9093

[Install]
WantedBy=multi-user.target
EOF

systemctl start alertmanager
systemctl enable alertmanager

4.2 基础配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# /etc/alertmanager/alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'password'
  smtp_require_tls: true

  # 默认通知模板
  resolve_timeout: 5m

# 模板文件
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# 路由配置
route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  
  routes:
    - match:
        severity: critical
      receiver: 'critical-receiver'
      continue: true
    
    - match:
        team: database
      receiver: 'db-team'
    
    - match_re:
        service: web.*
      receiver: 'web-team'

# 接收器
receivers:
  - name: 'default-receiver'
    email_configs:
      - to: 'ops-team@example.com'
        send_resolved: true

  - name: 'critical-receiver'
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://localhost:5001/webhook'
        send_resolved: true

  - name: 'db-team'
    email_configs:
      - to: 'db-team@example.com'
        send_resolved: true

  - name: 'web-team'
    email_configs:
      - to: 'web-team@example.com'
        send_resolved: true

# 抑制规则
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

4.3 通知渠道

邮件通知

1
2
3
4
5
6
7
8
9
10
11
receivers:
  - name: 'email-receiver'
    email_configs:
      - to: 'team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'password'
        send_resolved: true
        headers:
          Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'

钉钉通知

1
2
3
4
5
receivers:
  - name: 'dingtalk'
    webhook_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
        send_resolved: true

企业微信通知

1
2
3
4
5
6
7
8
receivers:
  - name: 'wechat'
    wechat_configs:
      - corp_id: 'YOUR_CORP_ID'
        api_secret: 'YOUR_API_SECRET'
        agent_id: 'YOUR_AGENT_ID'
        to_user: '@all'
        send_resolved: true

Slack 通知

1
2
3
4
5
6
7
8
receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#alerts'
        send_resolved: true
        title: '{{ .Status | toUpper }}: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

5. 告警规则编写

5.1 规则文件结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# /etc/prometheus/rules/alerts.yml
groups:
  - name: example-alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "实例 {{ $labels.instance }} 已宕机"
          description: "{{ $labels.instance }} 已经宕机超过 5 分钟"

      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "实例 {{ $labels.instance }} CPU 使用率过高"
          description: "CPU 使用率：{{ $value }}%"

5.2 常用告警规则

主机监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
groups:
  - name: node-alerts
    rules:
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "主机 {{ $labels.instance }} 已宕机"

      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "主机 {{ $labels.instance }} CPU 使用率过高"
          description: "CPU 使用率：{{ $value }}%"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "主机 {{ $labels.instance }} 内存使用率过高"
          description: "内存使用率：{{ $value }}%"

      - alert: HighDiskUsage
        expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "主机 {{ $labels.instance }} 磁盘使用率过高"
          description: "磁盘使用率：{{ $value }}%"

      - alert: HighLoadAverage
        expr: node_load1 > (count by(instance) (count by(instance, cpu) (node_cpu_seconds_total)))
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "主机 {{ $labels.instance }} 负载过高"
          description: "1 分钟负载：{{ $value }}"

服务监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
groups:
  - name: service-alerts
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "服务 {{ $labels.job }} 不可用"

      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "服务 {{ $labels.service }} 错误率过高"
          description: "错误率：{{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "服务 {{ $labels.service }} 延迟过高"
          description: "P95 延迟：{{ $value }}s"

数据库监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
groups:
  - name: database-alerts
    rules:
      - alert: MySQLDown
        expr: mysql_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MySQL {{ $labels.instance }} 不可用"

      - alert: MySQLHighConnections
        expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MySQL {{ $labels.instance }} 连接数过高"
          description: "连接数使用率：{{ $value | humanizePercentage }}"

      - alert: RedisDown
        expr: redis_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Redis {{ $labels.instance }} 不可用"

      - alert: RedisHighMemory
        expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Redis {{ $labels.instance }} 内存使用率过高"
          description: "内存使用率：{{ $value | humanizePercentage }}"

5.3 验证告警规则

1
2
3
4
5
6
7
8
# 检查配置文件
promtool check config /etc/prometheus/prometheus.yml

# 检查规则文件
promtool check rules /etc/prometheus/rules/alerts.yml

# 测试告警规则
promtool test rules test.yml

6. Grafana 可视化

6.1 安装 Grafana

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Ubuntu/Debian
apt-get install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" >> /etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install -y grafana

# CentOS/RHEL
cat > /etc/yum.repos.d/grafana.repo << EOF
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
yum install -y grafana

# 启动
systemctl start grafana-server
systemctl enable grafana-server

6.2 配置数据源

1
2
3
4
5
6
7
8
9
10
# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false

6.3 常用仪表盘

  • Node Exporter Full: ID 1860
  • Prometheus Stats: ID 2
  • MySQL Overview: ID 7362
  • Redis Dashboard: ID 11835
  • Docker and System Metrics: ID 893

7. 常用 Exporter

7.1 Node Exporter（主机监控）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 安装
wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
tar xzf node_exporter-1.5.0.linux-amd64.tar.gz
cp node_exporter-1.5.0.linux-amd64/node_exporter /usr/local/bin/

# Systemd 配置
cat > /etc/systemd/system/node_exporter.service << EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

systemctl start node_exporter
systemctl enable node_exporter

7.2 其他常用 Exporter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# MySQL Exporter
https://github.com/prometheus/mysqld_exporter

# Redis Exporter
https://github.com/oliver006/redis_exporter

# Nginx Exporter
https://github.com/nginxinc/nginx-prometheus-exporter

# Blackbox Exporter（黑盒监控）
https://github.com/prometheus/blackbox_exporter

# Kafka Exporter
https://github.com/danielqsj/kafka_exporter

# Elasticsearch Exporter
https://github.com/prometheus-community/elasticsearch_exporter

8. 服务发现

8.1 Kubernetes 服务发现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

8.2 Consul 服务发现

1
2
3
4
5
6
7
8
scrape_configs:
  - job_name: 'consul'
    consul_sd_configs:
      - server: 'consul:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: job

8.3 Docker 服务发现

1
2
3
4
5
6
7
8
scrape_configs:
  - job_name: 'docker'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: [__meta_docker_container_label_prometheus_scrape]
        action: keep
        regex: true

9. 高可用配置

9.1 多副本 Prometheus

1
2
3
# 运行多个 Prometheus 实例
# 使用相同的配置，独立存储数据
# 通过负载均衡器分发查询请求

9.2 Thanos 架构

1
2
3
4
5
6
7
Thanos Sidecar → Prometheus
     ↓
Thanos Store → 对象存储（S3/GCS）
     ↓
Thanos Query → 统一查询入口
     ↓
Grafana → 可视化

9.3 VictoriaMetrics

1
2
3
4
5
6
7
8
# 作为 Prometheus 的长期存储替代
# 更高的性能和压缩率
# 兼容 PromQL

docker run -it --rm \
  -p 8428:8428 \
  -v /data:/storage \
  victoriametrics/victoria-metrics

文档版本: 1.0
最后更新: 2026-02-27
适用版本: Prometheus 2.40+, Alertmanager 0.25+, Grafana 9.x