Grafana 监控可视化与仪表盘配置实战

一、概述

Grafana 是一款开源的数据可视化和分析平台，支持与 Prometheus、InfluxDB、Elasticsearch、MySQL 等多种数据源集成。本文介绍 Grafana 的安装配置、仪表盘设计及运维实践。

二、安装与部署

2.1 Ubuntu/Debian 安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 添加 GPG 密钥
sudo apt-get install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

# 添加仓库
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

# 安装
sudo apt-get update
sudo apt-get install grafana

# 启动服务
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

2.2 CentOS/RHEL 安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 添加仓库
cat > /etc/yum.repos.d/grafana.repo << EOF
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

# 安装
sudo yum install grafana

# 启动服务
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

2.3 Docker 部署

1
2
3
4
5
6
7
8
9
docker run -d \
  --name=grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  -v grafana-config:/etc/grafana \
  -e "GF_SECURITY_ADMIN_USER=admin" \
  -e "GF_SECURITY_ADMIN_PASSWORD=admin123" \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource" \
  grafana/grafana:latest

2.4 Docker Compose 部署（含 Prometheus）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
    networks:
      - monitoring

volumes:
  prometheus-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

三、配置详解

3.1 主配置文件

Grafana 主配置文件位于 /etc/grafana/grafana.ini：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[server]
protocol = http
http_addr = 0.0.0.0
http_port = 3000
domain = grafana.example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/

[database]
type = sqlite3
path = grafana.db

# 或使用 MySQL
# type = mysql
# host = 127.0.0.1:3306
# database = grafana
# user = grafana
# password = secret

[security]
admin_user = admin
admin_password = admin123
secret_key = SW2YcwTIb9zpOOhoPsMm

[auth.anonymous]
enabled = false

[users]
allow_sign_up = false
allow_org_create = false

[log]
mode = console file
level = info

[alerting]
enabled = true
execute_alerts = true

3.2 数据源配置

通过 UI 配置：

  1. 登录 Grafana → Configuration → Data Sources
  2. 点击 “Add data source”
  3. 选择数据源类型（Prometheus、MySQL 等）
  4. 填写连接信息并测试

通过配置文件 provisioning：

创建 /etc/grafana/provisioning/datasources/prometheus.yml：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: "15s"
      queryTimeout: "60s"
      httpMethod: POST

  - name: MySQL
    type: mysql
    url: mysql:3306
    database: monitoring
    user: grafana
    secureJsonData:
      password: mysql_password
    jsonData:
      maxOpenConns: 10
      maxIdleConns: 5
      connMaxLifetime: 14400

3.3 自动配置仪表盘

创建 /etc/grafana/provisioning/dashboards/default.yml：

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    folderUid: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards

将 JSON 仪表盘文件放入 /etc/grafana/provisioning/dashboards/ 目录即可自动加载。

四、仪表盘设计

4.1 面板类型

面板类型 适用场景
Graph/Time series 时间序列数据趋势
Stat 单个数值指标展示
Gauge 仪表盘式百分比展示
Bar gauge 条形进度展示
Table 表格数据展示
Heatmap 热力图分布
Pie chart 占比分布
Alert list 告警列表
Logs 日志展示
Trace 链路追踪

4.2 常用 PromQL 查询示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# CPU 使用率
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# 磁盘使用率
(1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})) * 100

# 网络流量（接收）
rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])

# 网络流量（发送）
rate(node_network_transmit_bytes_total{device!~"lo|veth.*"}[5m])

# 磁盘 IO 读取
rate(node_disk_read_bytes_total[5m])

# 磁盘 IO 写入
rate(node_disk_written_bytes_total[5m])

# 系统负载
node_load1, node_load5, node_load15

# 连接数统计
node_netstat_Tcp_CurrEstab

# 进程数
node_procs_running, node_procs_blocked

# 容器 CPU 使用率
rate(container_cpu_usage_seconds_total{image!=""}[5m])

# 容器内存使用率
container_memory_usage_bytes{image!=""}

# Pod 状态
kube_pod_status_phase

# 部署副本数
kube_deployment_status_replicas_available

4.3 变量配置

仪表盘变量支持动态过滤：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "current": {},
        "refresh": 1
      },
      {
        "name": "instance",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(node_uname_info, instance)",
        "refresh": 2,
        "multi": true,
        "includeAll": true
      },
      {
        "name": "interval",
        "type": "interval",
        "query": "1m,5m,10m,30m,1h,6h,12h,1d",
        "current": {
          "text": "5m",
          "value": "5m"
        }
      }
    ]
  }
}

使用变量：$instance 或 ${instance}

4.4 告警配置

告警规则示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# 在 Prometheus 中配置告警规则
groups:
  - name: node_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "高 CPU 使用率"
          description: "{{ $labels.instance }} CPU 使用率超过 80%，当前值 {{ $value }}%"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "高内存使用率"
          description: "{{ $labels.instance }} 内存使用率超过 85%，当前值 {{ $value }}%"

      - alert: DiskSpaceLow
        expr: (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs"} / node_filesystem_size_bytes{fstype!~"tmpfs"})) * 100 > 90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "磁盘空间不足"
          description: "{{ $labels.instance }} {{ $labels.mountpoint }} 磁盘使用率超过 90%"

Grafana 告警通知渠道配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# /etc/grafana/provisioning/notifiers/alertmanager.yml
apiVersion: 1

notifiers:
  - name: Alertmanager
    type: alertmanager
    uid: alertmanager
    is_default: true
    settings:
      url: http://alertmanager:9093
      timeout: 30

  - name: Email
    type: email
    uid: email
    is_default: false
    settings:
      addresses: ops@example.com
      singleEmail: true

  - name: Webhook
    type: webhook
    uid: webhook
    settings:
      url: http://webhook-server:8080/alert
      httpMethod: POST
      username: webhook_user
      secureJsonData:
        password: webhook_password

五、常用仪表盘模板

5.1 系统概览仪表盘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
{
  "dashboard": {
    "title": "系统概览",
    "panels": [
      {
        "type": "stat",
        "title": "CPU 使用率",
        "targets": [
          {
            "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "thresholds": {
              "steps": [
                {"color": "green", "value": null},
                {"color": "yellow", "value": 60},
                {"color": "red", "value": 80}
              ]
            }
          }
        }
      },
      {
        "type": "stat",
        "title": "内存使用率",
        "targets": [
          {
            "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100"
          }
        ]
      },
      {
        "type": "stat",
        "title": "磁盘使用率",
        "targets": [
          {
            "expr": "(1 - (node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})) * 100"
          }
        ]
      },
      {
        "type": "timeseries",
        "title": "CPU 趋势",
        "targets": [
          {
            "expr": "100 - (avg by(instance)(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
          }
        ]
      },
      {
        "type": "timeseries",
        "title": "内存趋势",
        "targets": [
          {
            "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes"
          }
        ]
      },
      {
        "type": "timeseries",
        "title": "网络流量",
        "targets": [
          {
            "expr": "rate(node_network_receive_bytes_total[5m])",
            "legendFormat": "接收 {{device}}"
          },
          {
            "expr": "rate(node_network_transmit_bytes_total[5m])",
            "legendFormat": "发送 {{device}}"
          }
        ]
      }
    ]
  }
}

5.2 Kubernetes 集群仪表盘

关键指标查询：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 集群 Pod 总数
count(kube_pod_info)

# 按命名空间统计 Pod
count by(namespace) (kube_pod_info)

# 节点资源使用率
sum by(node) (rate(container_cpu_usage_seconds_total[5m])) / sum by(node) (kube_node_status_capacity_cpu_cores) * 100

# Pod 重启次数
sum by(pod, namespace) (rate(kube_pod_container_status_restarts_total[1h]))

# 持久卷使用率
1 - (sum by(persistentvolumeclaim) (kubelet_volume_stats_available_bytes) / sum by(persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes)) * 100

5.3 应用监控仪表盘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# HTTP 请求速率
sum(rate(http_requests_total[5m])) by (service, status)

# 请求延迟（P95）
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# 错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

# JVM 内存使用
jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} * 100

# 数据库连接池
hikaricp_connections_active / hikaricp_connections_max * 100

六、权限管理

6.1 角色类型

角色 权限
Viewer 查看仪表盘和数据源
Editor 创建/编辑仪表盘
Admin 管理数据源、用户、组织

6.2 文件夹权限

1
2
3
4
5
6
7
8
9
10
11
# 通过 API 设置文件夹权限
curl -X POST -H "Content-Type: Application/json" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{
    "items": [
      {"role": "Viewer", "permission": 1},
      {"role": "Editor", "permission": 2},
      {"teamId": 1, "permission": 2}
    ]
  }' \
  http://grafana:3000/api/folders/<folder_uid>/permissions

6.3 团队管理

1
2
3
4
5
6
7
8
9
10
11
# 创建团队
curl -X POST -H "Content-Type: Application/json" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{"name": "DevOps", "email": "devops@example.com"}' \
  http://grafana:3000/api/teams

# 添加团队成员
curl -X POST -H "Content-Type: Application/json" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{"userId": 2}' \
  http://grafana:3000/api/teams/<team_id>/members

七、运维实践

7.1 备份与恢复

1
2
3
4
5
6
7
8
9
10
11
12
13
# 备份 Grafana 数据
# SQLite 默认路径：/var/lib/grafana/grafana.db
# MySQL 需备份数据库

# 停止服务
sudo systemctl stop grafana-server

# 备份数据目录
tar -czf grafana-backup-$(date +%Y%m%d).tar.gz /var/lib/grafana /etc/grafana

# 恢复
tar -xzf grafana-backup-YYYYMMDD.tar.gz -C /
sudo systemctl start grafana-server

7.2 性能优化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# grafana.ini 优化配置

[database]
# 使用 MySQL 替代 SQLite（生产环境）
type = mysql
host = 127.0.0.1:3306
database = grafana

[dataproxy]
# 增加超时时间
timeout = 60
dial_timeout = 10
keep_alive_seconds = 30

[alerting]
# 调整告警评估间隔
max_annotation_age = 7d
max_annotations_to_keep = 100

[unified_alerting]
# 启用统一告警
enabled = true

[panels]
# 限制面板查询
disable_sanitize_html = false

7.3 监控 Grafana 自身

1
2
3
4
5
6
7
# Grafana 自身指标
grafana_http_request_duration_seconds_count
grafana_http_request_duration_seconds_sum
grafana_active_service_users
grafana_stat_totals_dashboard
grafana_stat_totals_datasource
grafana_stat_totals_user

7.4 日志配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[log]
mode = console file
level = info
filters = 
# 可添加特定模块的日志级别
# sqlstore:debug
# auth:info

[log.file]
log_rotate = true
max_lines = 1000000
max_size_shift = 28
daily_rotate = true
max_days = 7

八、故障排查

8.1 常见问题

问题 可能原因 解决方案
数据源连接失败 网络不通/认证错误 检查网络和凭据
仪表盘不显示数据 PromQL 错误/时间范围 检查查询和时间设置
告警不触发 告警规则错误/通知渠道 检查规则和通知配置
页面加载缓慢 查询复杂/数据量大 优化查询/增加缓存
用户无法登录 LDAP/认证配置错误 检查认证配置

8.2 调试命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 查看 Grafana 日志
sudo journalctl -u grafana-server -f

# 检查服务状态
sudo systemctl status grafana-server

# 查看监听端口
sudo netstat -tlnp | grep 3000

# 测试 API
curl -u admin:admin123 http://localhost:3000/api/health

# 查看已安装插件
grafana-cli plugins list

九、最佳实践总结

  1. 使用 provisioning 配置：通过文件管理数据源和仪表盘，便于版本控制
  2. 合理设置刷新间隔：避免过于频繁的查询增加负载
  3. 使用变量提高复用性：一个仪表盘适配多个环境
  4. 设置合理的告警阈值：避免告警疲劳
  5. 定期备份配置：特别是生产环境
  6. 限制匿名访问：生产环境关闭匿名访问
  7. 使用 HTTPS：配置反向代理启用 HTTPS
  8. 监控 Grafana 自身：确保监控系统本身健康

十、参考资源