Grafana Alerting 告警配置实战

一、Grafana Alerting 概述

1.1 版本差异

特性 Grafana 8.x 之前 Grafana 8.x 及之后
告警引擎 基于 Panel 的旧告警 统一告警 (Unified Alerting)
规则存储 Dashboard JSON 内 独立告警规则
表达式 简单阈值 支持 PromQL/多数据源
通知策略 简单通知渠道 复杂路由和分组
推荐度 ❌ 已弃用 ✅ 推荐使用

1.2 核心概念

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
告警规则 (Alert Rule)
    ├── 查询 (Query) - 从数据源获取数据
    ├── 表达式 (Expression) - 处理查询结果
    ├── 条件 (Condition) - 触发告警的阈值
    ├── 评估间隔 (Evaluation Interval) - 多久评估一次
    └── 待处理时间 (Pending Period) - 持续多久才触发

通知策略 (Notification Policy)
    ├── 路由 (Routes) - 根据标签路由到不同渠道
    ├── 分组 (Grouping) - 将相关告警分组发送
    └── 抑制 (Inhibition) - 阻止冗余告警

通知渠道 (Contact Point)
    ├── Email
    ├── Slack
    ├── Webhook
    ├── PagerDuty
    ├── 钉钉
    └── 企业微信

二、告警规则配置

2.1 创建告警规则

步骤 1：进入告警页面

1
Grafana UI → Alerting → Alert rules → New alert rule

步骤 2：配置查询

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 示例 1：CPU 使用率
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 示例 2：内存使用率
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# 示例 3：磁盘使用率
(1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100

# 示例 4：服务宕机
up{job="nginx"} == 0

# 示例 5：请求错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

步骤 3：配置表达式

1
2
3
Query: A (上述查询)
Expression: $A > 80
Condition: WHEN last() OF query(A, 5m, now) IS ABOVE 80

步骤 4：配置评估规则

1
2
3
4
5
6
7
8
9
10
Folder: 基础设施监控
Evaluation group: 系统告警组
Evaluation interval: 1m  # 每 1 分钟评估一次
Pending period: 5m       # 持续 5 分钟超过阈值才触发
Annotations:
  summary: "实例 {{ $labels.instance }} CPU 使用率过高"
  description: "CPU 使用率 {{ $value }}% 超过阈值 80%"
Labels:
  severity: warning
  team: infrastructure

2.2 告警规则 YAML 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
# alert_rules.yaml
groups:
  - name: 系统资源告警
    folder: 基础设施监控
    interval: 1m
    rules:
      # CPU 使用率告警
      - uid: cpu_usage_high
        title: CPU 使用率过高
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
              intervalMs: 15000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 80
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    params: []
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "实例 {{ $labels.instance }} CPU 使用率过高"
          description: "CPU 使用率 {{ $value }}% 超过阈值 80%，持续 5 分钟"
        labels:
          severity: warning
          team: infrastructure

      # 内存使用率告警
      - uid: memory_usage_high
        title: 内存使用率过高
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
              intervalMs: 15000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 85
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    params: []
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "实例 {{ $labels.instance }} 内存使用率过高"
          description: "内存使用率 {{ $value }}% 超过阈值 85%"
        labels:
          severity: warning
          team: infrastructure

      # 磁盘使用率告警
      - uid: disk_usage_high
        title: 磁盘使用率过高
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100
              intervalMs: 15000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 90
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    params: []
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        execErrState: Error
        for: 10m
        annotations:
          summary: "实例 {{ $labels.instance }} 磁盘使用率过高"
          description: "磁盘使用率 {{ $value }}% 超过阈值 90%"
        labels:
          severity: critical
          team: infrastructure

      # 服务宕机告警
      - uid: service_down
        title: 服务宕机
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: up
              intervalMs: 15000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 0
                    type: eq
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    params: []
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: Alerting
        execErrState: Error
        for: 1m
        annotations:
          summary: "服务 {{ $labels.job }} 宕机"
          description: "实例 {{ $labels.instance }} 服务 {{ $labels.job }} 已宕机"
        labels:
          severity: critical
          team: oncall

2.3 使用 Terraform 管理告警规则

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# main.tf
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "~> 2.0"
    }
  }
}

provider "grafana" {
  url  = "http://grafana:3000"
  auth = "admin:admin"
}

# 告警规则
resource "grafana_rule_group" "system_alerts" {
  name             = "系统告警组"
  folder_uid       = grafana_folder.infrastructure.uid
  interval_seconds = 60

  rule {
    name      = "CPU 使用率过高"
    condition = "C"
    for       = "5m"

    data {
      ref_id         = "A"
      datasource_uid = "prometheus"
      query_type     = ""
      relative_time_range {
        from = 300
        to   = 0
      }
      model = jsonencode({
        expr          = "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
        intervalMs    = 15000
        maxDataPoints = 43200
        refId         = "A"
      })
    }

    data {
      ref_id         = "C"
      datasource_uid = "__expr__"
      query_type     = ""
      relative_time_range {
        from = 300
        to   = 0
      }
      model = jsonencode({
        type        = "threshold"
        expression  = "A"
        refId       = "C"
        conditions = [{
          evaluator = {
            params = [80]
            type   = "gt"
          }
          operator = {
            type = "and"
          }
          query = {
            params = ["A"]
          }
          reducer = {
            type = "last"
          }
          type = "query"
        }]
      })
    }

    no_data_state  = "NoData"
    exec_err_state = "Error"

    annotations = {
      summary     = "实例 {{ $labels.instance }} CPU 使用率过高"
      description = "CPU 使用率 {{ $value }}% 超过阈值 80%"
    }

    labels = {
      severity = "warning"
      team     = "infrastructure"
    }
  }
}

# 文件夹
resource "grafana_folder" "infrastructure" {
  title = "基础设施监控"
}

三、通知渠道配置

3.1 配置通知渠道

邮件通知

1
2
3
4
5
6
7
8
# 在 Grafana UI 中配置
Alerting → Contact points → Add contact point

Name: 邮件通知
Type: Email
Settings:
  Addresses: ops@example.com,admin@example.com
  Single Email: true  # 合并为一封邮件发送

Slack 通知

1
2
3
4
5
6
7
8
Name: Slack 通知
Type: Slack
Settings:
  URL: https://hooks.slack.com/services/XXX/YYY/ZZZ
  Username: Grafana Alertbot
  Icon Emoji: 🚨
  Mention Users: "@oncall"
  Mention Groups: "@channel"

Webhook 通知

1
2
3
4
5
6
7
8
9
10
Name: 自定义 Webhook
Type: Webhook
Settings:
  URL: https://api.example.com/alerts
  HTTP Method: POST
  Authorization:
    Type: Bearer
    Credentials: your-token-here
  Custom Headers:
    X-Source: Grafana

钉钉通知

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Name: 钉钉通知
Type: Webhook
Settings:
  URL: https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN
  HTTP Method: POST
  Message: |
    {
      "msgtype": "markdown",
      "markdown": {
        "title": "Grafana 告警",
        "text": "## ${labels.alertname}\n\n**严重级别**: ${labels.severity}\n**实例**: ${labels.instance}\n**描述**: ${annotations.description}\n**时间**: ${startsAt}"
      },
      "at": {
        "isAll": true
      }
    }

企业微信通知

1
2
3
4
5
6
7
8
9
10
11
12
Name: 企业微信通知
Type: Webhook
Settings:
  URL: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY
  HTTP Method: POST
  Message: |
    {
      "msgtype": "markdown",
      "markdown": {
        "content": "## ${labels.alertname}\n- **严重级别**: ${labels.severity}\n- **实例**: ${labels.instance}\n- **描述**: ${annotations.description}\n- **时间**: ${startsAt}"
      }
    }

3.2 通知策略配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# 通知策略路由
Alerting → Notification policies

Default Contact Point: 邮件通知

Routes:
  # 严重告警 → Slack + 电话
  - Matcher: severity = critical
    Contact Point: Slack 通知
    Group By: [alertname, instance]
    Group Wait: 30s
    Group Interval: 5m
    Repeat Interval: 4h
    Continue: true  # 继续匹配下一条

  # 警告告警 → 邮件
  - Matcher: severity = warning
    Contact Point: 邮件通知
    Group By: [alertname, team]
    Group Wait: 1m
    Group Interval: 10m
    Repeat Interval: 12h

  # 基础设施团队告警
  - Matcher: team = infrastructure
    Contact Point: Slack 通知
    Group By: [alertname]
    Group Wait: 30s
    Group Interval: 5m
    Repeat Interval: 4h

  # 值班团队告警
  - Matcher: team = oncall
    Contact Point: PagerDuty
    Group By: [alertname, instance]
    Group Wait: 0s  # 立即发送
    Group Interval: 1m
    Repeat Interval: 1h

3.3 告警抑制配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 抑制规则
Alerting → Notification policies → Inhibition rules

# 规则 1：当实例宕机时，抑制该实例的其他告警
Source Match:
  alertname = 服务宕机
Target Match:
  alertname != 服务宕机
Equal:
  - instance

# 规则 2：当集群不可用时，抑制节点告警
Source Match:
  severity = critical
  alertname = 集群不可用
Target Match:
  severity = warning
Equal:
  - cluster

# 规则 3：维护期间抑制所有告警
Source Match:
  alertname = 维护中
Target Match:
  {}  # 匹配所有
Equal:
  - instance

四、告警模板配置

4.1 自定义告警模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# 告警标题模板
{{ define "alert.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }}
{{ end }}

# 告警内容模板
{{ define "alert.content" }}
{{ if gt (len .Alerts.Firing) 0 }}
### 🔴 告警触发

{{ range .Alerts.Firing }}
**告警名称**: {{ .Labels.alertname }}
**严重级别**: {{ .Labels.severity }}
**实例**: {{ .Labels.instance }}
**开始时间**: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
**描述**: {{ .Annotations.description }}
**详情**: {{ .Annotations.runbook_url }}

{{ end }}
{{ end }}

{{ if gt (len .Alerts.Resolved) 0 }}
### 🟢 告警恢复

{{ range .Alerts.Resolved }}
**告警名称**: {{ .Labels.alertname }}
**实例**: {{ .Labels.instance }}
**恢复时间**: {{ .EndsAt.Format "2006-01-02 15:04:05" }}
**持续时间**: {{ .StartsAt.Sub .EndsAt | abs | duration }}

{{ end }}
{{ end }}
{{ end }}

# Slack 专用模板
{{ define "slack.message" }}
{{ if eq .Status "firing" }}
🚨 *告警触发*
{{ else }}
✅ *告警恢复*
{{ end }}

*告警*: {{ .GroupLabels.SortedPairs.Values | join " " }}
*状态*: {{ .Status }}
*数量*: {{ .Alerts.Firing | len }} firing, {{ .Alerts.Resolved | len }} resolved

{{ range .Alerts }}
────────────────────
*{{ .Labels.alertname }}*
严重级别：{{ .Labels.severity }}
实例：{{ .Labels.instance }}
{{ if .Annotations.description }}描述：{{ .Annotations.description }}{{ end }}
{{ if .Annotations.runbook_url }}[查看 Runbook]({{ .Annotations.runbook_url }}){{ end }}
{{ end }}
{{ end }}

4.2 在通知策略中使用模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# 通知策略配置
Alerting → Notification policies → Templates

Name: 告警通知模板
Template: |
  {{ define "myorg.notification" }}
  [{{ .Status | toUpper }}] {{ .GroupLabels.SortedPairs.Values | join " " }}
  
  {{ if gt (len .Alerts.Firing) 0 }}
  🔴 Firing Alerts:
  {{ range .Alerts.Firing }}
  - {{ .Labels.alertname }} ({{ .Labels.severity }})
    Instance: {{ .Labels.instance }}
    {{ .Annotations.description }}
  {{ end }}
  {{ end }}
  
  {{ if gt (len .Alerts.Resolved) 0 }}
  🟢 Resolved Alerts:
  {{ range .Alerts.Resolved }}
  - {{ .Labels.alertname }}
    Instance: {{ .Labels.instance }}
  {{ end }}
  {{ end }}
  {{ end }}

五、告警管理最佳实践

5.1 告警分级标准

级别 定义 响应时间 通知渠道
Critical 服务不可用，影响用户 5 分钟内 电话 + Slack + 邮件
Warning 服务降级，可能影响用户 30 分钟内 Slack + 邮件
Info 需要注意但不紧急 24 小时内 邮件

5.2 告警规则设计原则

1
2
3
4
5
6
7
8
9
10
11
# ✅ 好的告警规则
- 基于症状而非原因（用户能感知的问题）
- 有明确的阈值和持续时间
- 包含可操作的描述和 Runbook 链接
- 避免告警疲劳（合理设置评估间隔）

# ❌ 不好的告警规则
- 基于内部指标（CPU 高但服务正常）
- 瞬时波动就触发（没有 Pending Period）
- 描述模糊（"Something is wrong"）
- 过于频繁（每 10 秒评估一次）

5.3 告警去重和聚合

1
2
3
4
5
6
7
8
9
# 使用 Group By 进行告警聚合
Group By:
  - alertname      # 按告警名称分组
  - cluster        # 按集群分组
  - namespace      # 按命名空间分组

# 避免按以下标签分组（会导致告警爆炸）
# - instance       # 每个实例一条告警
# - pod            # 每个 Pod 一条告警

5.4 告警维护模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 使用 Mute Timing 设置维护窗口
Alerting → Notification policies → Mute timings

Name: 凌晨维护窗口
Time Intervals:
  - times:
      - start_time: "02:00"
        end_time: "06:00"
    weekdays:
      - monday
      - tuesday
      - wednesday
      - thursday
      - friday

# 在通知策略中应用
Routes:
  - Matcher: alertname = 计划内维护
    Mute Time Interval: 凌晨维护窗口

六、监控和测试

6.1 告警健康检查

1
2
3
4
5
6
7
8
9
# 检查告警规则评估状态
grafana_alerting_rule_evaluation_duration_seconds

# 检查告警规则评估错误
grafana_alerting_rule_evaluation_failures_total

# 检查告警通知发送状态
grafana_alerting_notification_sent_total
grafana_alerting_notification_failed_total

6.2 告警测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# 使用 curl 测试 Webhook
curl -X POST https://api.example.com/alerts \
  -H "Content-Type: application/json" \
  -d '{
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "Test Alert",
          "severity": "warning",
          "instance": "test-server"
        },
        "annotations": {
          "summary": "测试告警",
          "description": "这是测试告警内容"
        },
        "startsAt": "2026-03-25T08:00:00Z",
        "endsAt": "2026-03-25T08:10:00Z"
      }
    ]
  }'

# 使用 Grafana API 测试告警规则
curl -X POST http://localhost:3000/api/v1/rulegroups/test \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -d '{
    "rules": [
      {
        "name": "Test Rule",
        "query": "up",
        "condition": "up == 0",
        "for": "1m"
      }
    ]
  }'

6.3 告警审计

1
2
3
4
-- Grafana 数据库中的告警历史
SELECT * FROM alert_rule_version ORDER BY created DESC LIMIT 10;
SELECT * FROM alert_instance ORDER BY updated DESC LIMIT 10;
SELECT * FROM notification_log ORDER BY created DESC LIMIT 10;

七、常见问题排查

7.1 告警不触发

1
2
3
4
5
6
7
8
9
10
11
12
# 检查数据源连接
Grafana → Configuration → Data sources → Test connection

# 检查查询是否返回数据
在 Alert rule 页面点击 Test rule

# 检查评估间隔
确认 Evaluation interval 设置合理

# 查看日志
docker logs grafana | grep -i alert
journalctl -u grafana-server | grep -i alert

7.2 告警通知不发送

1
2
3
4
5
6
7
8
9
10
11
# 检查通知渠道配置
Alerting → Contact points → Test

# 检查通知策略路由
确认 Matcher 规则正确匹配

# 检查抑制规则
确认没有抑制规则阻止通知

# 查看通知日志
Grafana → Alerting → Alert instances → 查看通知历史

7.3 告警风暴

1
2
3
4
5
6
7
8
9
10
11
12
# 增加 Group Wait 和 Group Interval
Group Wait: 30s → 2m
Group Interval: 5m → 10m

# 增加 Repeat Interval
Repeat Interval: 4h → 12h

# 添加抑制规则
当核心服务宕机时，抑制相关依赖服务告警

# 调整阈值和持续时间
for: 1m → 5m

八、总结

配置项 推荐值 说明
Evaluation Interval 1m 评估间隔
Pending Period 5m 持续时间阈值
Group Wait 30s-2m 告警等待时间
Group Interval 5m-10m 告警分组间隔
Repeat Interval 4h-12h 重复通知间隔

关键要点：

  1. 合理分级：Critical/Warning/Info 分级明确
  2. 避免风暴：使用聚合、抑制、去重
  3. 可操作：每条告警都有 Runbook
  4. 可测试：定期测试通知渠道
  5. 可维护：使用 Terraform 管理配置

参考文档：