Alertmanager 告警路由与通知配置实战

一、概述

Alertmanager 是 Prometheus 生态系统中负责处理告警的核心组件，主要职责包括：

  • 去重与分组：将相同告警合并，避免告警风暴
  • 路由分发：根据标签将告警发送到不同接收器
  • 静默抑制：支持告警静默和依赖抑制
  • 多渠道通知：支持邮件、钉钉、企业微信、Slack、Webhook 等

本文基于 Alertmanager v0.27.0，详细介绍告警路由配置与多渠道通知实战。

二、Alertmanager 架构

1
2
3
4
5
6
7
8
9
10
┌─────────────┐    ┌──────────────────┐    ┌─────────────┐
│ Prometheus  │───▶│  Alertmanager    │───▶│  通知渠道   │
│   (告警)    │    │  (路由/分组)     │    │ (邮件/IM)   │
└─────────────┘    └──────────────────┘    └─────────────┘
                           │
                           ▼
                    ┌─────────────┐
                    │  告警面板   │
                    │ (Grafana)   │
                    └─────────────┘

三、安装与部署

3.1 下载与安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# 下载 Alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xvf alertmanager-0.27.0.linux-amd64.tar.gz
cd alertmanager-0.27.0.linux-amd64

# 创建配置目录
sudo mkdir -p /etc/alertmanager
sudo cp alertmanager.yml /etc/alertmanager/
sudo cp amtool /usr/local/bin/

# 创建 systemd 服务
sudo tee /etc/systemd/system/alertmanager.service > /dev/null << 'EOF'
[Unit]
Description=Alertmanager
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
ExecStart=/opt/alertmanager/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager \
    --web.listen-address=:9093 \
    --cluster.listen-address=:9094
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 创建用户并启动
sudo useradd -r -s /bin/false alertmanager
sudo mkdir -p /var/lib/alertmanager
sudo chown -R alertmanager:alertmanager /var/lib/alertmanager /etc/alertmanager
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

3.2 验证安装

1
2
3
4
5
6
7
8
# 检查服务状态
systemctl status alertmanager

# 访问 Web UI
curl http://localhost:9093/api/v2/status

# 查看监听端口
ss -tlnp | grep 9093

四、告警路由配置详解

4.1 基础路由配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.163.com:465'
  smtp_from: 'weijiantu@163.com'
  smtp_auth_username: 'weijiantu@163.com'
  smtp_auth_password: 'KGQ6h49vKhcV2ug2'
  smtp_require_tls: false

# 模板文件
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# 根路由
route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  
  # 子路由 - 按严重程度分发
  routes:
    - match:
        severity: critical
      receiver: 'critical-alerts'
      group_wait: 10s
      repeat_interval: 1h
      
    - match:
        severity: warning
      receiver: 'warning-alerts'
      group_wait: 1m
      repeat_interval: 4h
      
    - match:
        team: database
      receiver: 'db-team'
      
    - match:
        team: frontend
      receiver: 'frontend-team'

# 接收器配置
receivers:
  - name: 'default-receiver'
    email_configs:
      - to: 'ops-team@example.com'
        send_resolved: true
        
  - name: 'critical-alerts'
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://localhost:5001/webhook'
        send_resolved: true
        
  - name: 'warning-alerts'
    email_configs:
      - to: 'dev-team@example.com'
        send_resolved: true
        
  - name: 'db-team'
    email_configs:
      - to: 'dba@example.com'
        send_resolved: true
        
  - name: 'frontend-team'
    email_configs:
      - to: 'frontend@example.com'
        send_resolved: true

# 抑制规则
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

4.2 路由匹配规则

Alertmanager 支持多种匹配方式：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
routes:
  # 精确匹配
  - match:
      severity: critical
    receiver: 'critical-receiver'
    
  # 正则匹配
  - match_re:
      service: '^(api|web|auth).*$'
    receiver: 'core-services'
    
  # 多条件匹配（AND 关系）
  - match:
      severity: critical
      team: backend
    receiver: 'backend-critical'
    
  # 嵌套路由
  - match:
      environment: production
    routes:
      - match:
          severity: critical
        receiver: 'prod-critical'
      - match:
          severity: warning
        receiver: 'prod-warning'

4.3 路由优先级

路由按从上到下顺序匹配，第一个匹配的路由生效：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
route:
  routes:
    # 优先级 1 - 最具体
    - match:
        alertname: DatabaseDown
        severity: critical
      receiver: 'db-critical'
      
    # 优先级 2 - 较具体
    - match:
        severity: critical
      receiver: 'all-critical'
      
    # 优先级 3 - 默认
    - match_re:
        .+: .+
      receiver: 'default'

五、通知渠道配置

5.1 邮件通知

1
2
3
4
5
6
7
8
9
10
11
12
13
receivers:
  - name: 'email-receiver'
    email_configs:
      - to: 'team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'password'
        auth_identity: 'alertmanager@example.com'
        headers:
          Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
        html: '{{ template "email.default.html" . }}'
        send_resolved: true

5.2 钉钉通知

1
2
3
4
5
6
7
8
9
receivers:
  - name: 'dingtalk'
    dingtalk_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
        message_type: 'markdown'
        at:
          mobiles: ['13800138000']
          is_all: false
        send_resolved: true

5.3 企业微信通知

1
2
3
4
5
6
7
8
9
receivers:
  - name: 'wechat'
    wechat_configs:
      - corp_id: 'YOUR_CORP_ID'
        api_secret: 'YOUR_API_SECRET'
        agent_id: 1000001
        to_user: '@all'
        message_type: 'markdown'
        send_resolved: true

5.4 Slack 通知

1
2
3
4
5
6
7
8
9
10
receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts'
        username: 'Alertmanager'
        icon_emoji: ':warning:'
        title: '{{ .Status | toUpper }}: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        send_resolved: true

5.5 Webhook 通知

1
2
3
4
5
6
7
8
9
receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'http://localhost:5001/alertmanager'
        send_resolved: true
        http_config:
          basic_auth:
            username: 'alertmanager'
            password: 'secret'

六、告警模板配置

6.1 自定义邮件模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# /etc/alertmanager/templates/email.tmpl
{{ define "email.default.html" }}
<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <style>
    body { font-family: Arial, sans-serif; }
    .critical { color: #dc3545; }
    .warning { color: #ffc107; }
    .info { color: #17a2b8; }
    table { border-collapse: collapse; width: 100%; }
    th, td { border: 1px solid #ddd; padding: 8px; }
    th { background-color: #f2f2f2; }
  </style>
</head>
<body>
  <h2>告警通知 - {{ .Status | toUpper }}</h2>
  
  <p><strong>告警组:</strong> {{ .GroupLabels.alertname }}</p>
  <p><strong>触发时间:</strong> {{ .StartsAt.Format "2006-01-02 15:04:05" }}</p>
  
  <table>
    <tr>
      <th>告警名称</th>
      <th>严重程度</th>
      <th>实例</th>
      <th>描述</th>
    </tr>
    {{ range .Alerts }}
    <tr>
      <td>{{ .Labels.alertname }}</td>
      <td class="{{ .Labels.severity }}">{{ .Labels.severity }}</td>
      <td>{{ .Labels.instance }}</td>
      <td>{{ .Annotations.description }}</td>
    </tr>
    {{ end }}
  </table>
  
  <p>
    <a href="{{ .ExternalURL }}">查看 Alertmanager</a>
  </p>
</body>
</html>
{{ end }}

6.2 钉钉 Markdown 模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# /etc/alertmanager/templates/dingtalk.tmpl
{{ define "dingtalk.title" }}
## {{ .Status | toUpper }}: {{ .GroupLabels.alertname }}
{{ end }}

{{ define "dingtalk.content" }}
{{ range .Alerts }}
**告警**: {{ .Labels.alertname }}
**级别**: {{ .Labels.severity }}
**实例**: {{ .Labels.instance }}
**时间**: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
**描述**: {{ .Annotations.description }}

---
{{ end }}
{{ end }}

七、静默与抑制

7.1 告警静默

使用 amtool 创建静默规则：

1
2
3
4
5
6
7
8
9
10
11
# 静默特定告警
amtool silence add alertname=NodeDown instance=node-1 --author="admin" --duration=2h

# 静默所有 warning 级别告警
amtool silence add severity=warning --duration=1h

# 查看静默规则
amtool silence query

# 删除静默规则
amtool silence expire <SILENCE_ID>

7.2 告警抑制

抑制规则防止告警风暴：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
inhibit_rules:
  # 当集群宕机时，抑制该集群下的节点告警
  - source_match:
      alertname: ClusterDown
    target_match:
      alertname: NodeDown
    equal: ['cluster']
    
  # 当数据库宕机时，抑制相关应用告警
  - source_match:
      alertname: DatabaseDown
    target_match_re:
      alertname: '.*Database.*'
    equal: ['database']
    
  # 当 severity=critical 时，抑制同组的 warning
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

八、高可用集群部署

8.1 集群配置

1
2
3
4
5
6
7
8
9
# 所有节点配置相同的 cluster.listen-address
ExecStart=/opt/alertmanager/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager \
    --web.listen-address=:9093 \
    --cluster.listen-address=:9094 \
    --cluster.peer=node1:9094 \
    --cluster.peer=node2:9094 \
    --cluster.peer=node3:9094

8.2 负载均衡配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Nginx 负载均衡配置
upstream alertmanager {
    server node1:9093;
    server node2:9093;
    server node3:9093;
}

server {
    listen 80;
    server_name alertmanager.example.com;
    
    location / {
        proxy_pass http://alertmanager;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

九、监控与运维

9.1 Alertmanager 自身监控

1
2
3
4
5
# Prometheus 配置
scrape_configs:
  - job_name: 'alertmanager'
    static_configs:
      - targets: ['localhost:9093']

9.2 关键监控指标

1
2
3
4
5
6
7
8
9
10
11
# 告警处理延迟
alertmanager_cluster_notification_latency_seconds

# 告警发送失败数
alertmanager_notifications_failed_total

# 队列中的告警数
alertmanager_queue_length

# 集群节点状态
alertmanager_cluster_members

9.3 日志排查

1
2
3
4
5
6
7
8
# 查看 Alertmanager 日志
journalctl -u alertmanager -f

# 查看特定级别日志
journalctl -u alertmanager -f | grep -i error

# 日志位置（systemd）
/var/log/syslog | grep alertmanager

十、常见问题排查

10.1 告警未发送

1
2
3
4
5
6
7
8
# 检查配置语法
amtool check-config /etc/alertmanager/alertmanager.yml

# 查看告警状态
curl http://localhost:9093/api/v2/alerts

# 检查通知日志
journalctl -u alertmanager | grep -i notification

10.2 路由未匹配

1
2
3
# 使用 amtool 测试路由
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
    alertname=TestAlert severity=critical team=backend

10.3 邮件发送失败

1
2
3
4
5
6
# 测试 SMTP 连接
openssl s_client -connect smtp.163.com:465

# 检查认证配置
# 确认 smtp_auth_password 正确
# 确认 smtp_require_tls 设置正确

十一、最佳实践

  1. 合理分组：按服务/团队分组，避免告警风暴
  2. 分级通知：critical 电话/短信，warning 邮件/IM
  3. 设置静默：维护窗口前提前静默相关告警
  4. 配置抑制：避免级联告警噪音
  5. 定期演练：测试通知渠道是否有效
  6. 文档化：记录路由规则和接收人清单
  7. 监控自身：确保 Alertmanager 正常运行

十二、总结

Alertmanager 是告警管理的核心组件，合理配置路由和通知渠道可以：

  • 减少告警噪音，提高响应效率
  • 确保关键告警及时送达
  • 支持多渠道通知，提升可靠性
  • 通过抑制和静默，避免告警风暴

建议根据团队实际情况，逐步完善告警路由策略，建立高效的告警响应机制。