ELK Stack 日志收集与分析实战
一、背景与目标
在分布式系统和微服务架构中，日志分散在各个节点和服务上，传统的手动查看日志方式效率低下。ELK Stack（Elasticsearch + Logstash + Kibana）提供了一套完整的日志收集、存储、分析和可视化解决方案。
本文目标：
- 搭建完整的 ELK 日志收集平台
- 配置 Filebeat 采集应用日志
- 设计合理的日志索引策略
- 实现日志可视化与告警
二、架构设计
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Application│ │ Application│ │ Application│
│ Server 1 │ │ Server 2 │ │ Server N │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ Filebeat Agent │
└─────────────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Logstash │
│ (过滤、解析、丰富日志) │
└─────────────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Elasticsearch │
│ (存储、索引、搜索) │
└─────────────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Kibana │
│ (可视化、仪表盘、告警) │
└─────────────────────────────────────────────────────┘
|
三、环境准备
3.1 系统要求
| 组件 |
最低配置 |
推荐配置 |
| Elasticsearch |
4GB RAM, 2 CPU |
16GB RAM, 8 CPU |
| Logstash |
2GB RAM, 1 CPU |
4GB RAM, 2 CPU |
| Kibana |
2GB RAM, 1 CPU |
4GB RAM, 2 CPU |
3.2 版本选择
四、安装部署
4.1 安装 Elasticsearch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list
apt update && apt install -y elasticsearch=${ELK_VERSION}
cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: elk-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: true
xpack.security.enrollment.enabled: false
EOF
systemctl enable elasticsearch && systemctl start elasticsearch
|
4.2 安装 Logstash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
| apt install -y logstash=${ELK_VERSION}
cat > /etc/logstash/conf.d/app-logs.conf << 'EOF'
input {
beats {
port => 5044
}
}
filter {
if [message] =~ /^\{.*\}$/ {
json {
source => "message"
target => "parsed"
}
}
mutate {
add_field => { "processed_at" => "%{@timestamp}" }
}
mutate {
remove_field => ["host", "agent"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "app-logs-%{+YYYY.MM.dd}"
user => "elastic"
password => "${ES_PASSWORD}"
}
}
EOF
systemctl enable logstash && systemctl start logstash
|
4.3 安装 Kibana
1
2
3
4
5
6
7
8
9
10
11
12
13
| apt install -y kibana=${ELK_VERSION}
cat > /etc/kibana/kibana.yml << EOF
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "${ES_PASSWORD}"
xpack.encryptedSavedObjects.encryptionKey: "$(openssl rand -hex 32)"
EOF
systemctl enable kibana && systemctl start kibana
|
4.4 安装 Filebeat（客户端）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| apt install -y filebeat=${ELK_VERSION}
cat > /etc/filebeat/filebeat.yml << 'EOF'
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/application/*.log
json.keys_under_root: true
json.add_error_key: true
json.message_key: message
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
output.logstash:
hosts: ["logstash-server:5044"]
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat.log
EOF
systemctl enable filebeat && systemctl start filebeat
|
五、索引生命周期管理（ILM）
5.1 创建 ILM 策略
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
| curl -X PUT "localhost:9200/_ilm/policy/app-logs-policy" -H 'Content-Type: application/json' -u elastic:${ES_PASSWORD} -d'
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50GB",
"max_age": "7d"
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"set_priority": {
"priority": 50
},
"shrink": {
"number_of_shards": 1
}
}
},
"cold": {
"min_age": "30d",
"actions": {
"set_priority": {
"priority": 0
},
"freeze": {}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}'
|
5.2 创建索引模板
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| curl -X PUT "localhost:9200/_index_template/app-logs-template" -H 'Content-Type: application/json' -u elastic:${ES_PASSWORD} -d'
{
"index_patterns": ["app-logs-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.lifecycle.name": "app-logs-policy",
"index.lifecycle.rollover_alias": "app-logs"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"message": { "type": "text" },
"level": { "type": "keyword" },
"service": { "type": "keyword" },
"host": { "type": "keyword" }
}
}
}
}'
|
5.3 创建初始索引
1
2
3
4
5
6
7
8
| curl -X PUT "localhost:9200/app-logs-000001" -H 'Content-Type: application/json' -u elastic:${ES_PASSWORD} -d'
{
"aliases": {
"app-logs": {
"is_write_index": true
}
}
}'
|
六、Kibana 可视化配置
6.1 创建数据视图
- 访问 Kibana (http://server-ip:5601)
- 进入 Stack Management → Data Views
- 创建数据视图
app-logs-*
- 选择
@timestamp 作为时间字段
6.2 常用查询示例
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| # 查看 ERROR 级别日志
level: "ERROR"
# 查看特定服务的日志
service: "user-service"
# 查看最近 1 小时的错误日志
level: "ERROR" AND @timestamp > now-1h
# 统计各服务错误数量
# 使用 Visualize → Data Table
# Aggregation: Terms (service)
# Filter: level: "ERROR"
# 查看特定错误堆栈
message: "*NullPointerException*"
|
6.3 创建仪表盘
推荐仪表盘组件：
- 日志量趋势图 - Line chart，按时间统计日志数量
- 日志级别分布 - Pie chart，展示 INFO/WARN/ERROR 比例
- 服务日志排行 - Bar chart，Top 10 服务的日志量
- 错误日志列表 - Data table，最近 50 条 ERROR 日志
- 响应时间分布 - Histogram，API 响应时间分布
七、告警配置
7.1 创建告警规则
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|
curl -X PUT "localhost:9200/_watcher/watch/error-rate-alert" -H 'Content-Type: application/json' -u elastic:${ES_PASSWORD} -d'
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"indices": ["app-logs-*"],
"body": {
"query": {
"bool": {
"must": [
{ "term": { "level": "ERROR" } },
{ "range": { "@timestamp": { "gte": "now-5m" } } }
]
}
},
"size": 0,
"aggs": {
"error_count": { "value_count": { "field": "_id" } }
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.aggregations.error_count.value": {
"gt": 100
}
}
},
"actions": {
"send_email": {
"email": {
"to": ["ops-team@example.com"],
"subject": "【告警】ERROR 日志数量异常",
"body": "过去 5 分钟内 ERROR 日志数量：{{ctx.payload.aggregations.error_count.value}}"
}
}
}
}'
|
八、性能优化建议
8.1 Elasticsearch 优化
1
2
3
4
5
6
7
|
-Xms8g
-Xmx8g
indices.memory.index_buffer_size: 20%
thread_pool.write.queue_size: 1000
|
8.2 Logstash 优化
1
2
3
4
|
pipeline.workers: 4
pipeline.batch.size: 125
pipeline.batch.delay: 50
|
8.3 Filebeat 优化
1
2
3
4
5
6
7
8
|
filebeat.inputs:
- type: log
close_inactive: 5m
close_eof: false
clean_inactive: 24h
max_bytes: 10MB
line_limit: 10000
|
九、常见问题排查
9.1 日志不入库
1
2
3
4
5
6
7
8
9
|
systemctl status filebeat
filebeat test output
curl localhost:9600/_node/stats/pipelines
curl -u elastic:${ES_PASSWORD} localhost:9200/_cat/indices?v
|
9.2 查询性能慢
- 检查索引分片数量是否合理
- 避免使用通配符前缀查询
- 为常用字段建立 keyword 类型
- 使用时间范围过滤减少扫描量
9.3 磁盘空间不足
- 调整 ILM 策略，缩短保留时间
- 启用索引压缩
- 增加冷节点存储
- 定期删除无用索引
十、总结
ELK Stack 是企业级日志管理的标准解决方案。通过合理配置：
- Filebeat 负责轻量级日志采集
- Logstash 负责日志处理和丰富
- Elasticsearch 负责存储和检索
- Kibana 负责可视化和告警
关键成功因素：
- 合理的索引生命周期管理
- 适当的资源分配和性能调优
- 有意义的可视化仪表盘
- 及时有效的告警规则
参考文档：
