OpenTelemetry 分布式追踪链路部署与运维实战

一、背景与概述

1.1 为什么需要分布式追踪

在微服务架构中，一个用户请求往往会经过多个服务的处理。当系统出现性能问题或故障时，传统的监控手段（如指标监控、日志监控）难以快速定位问题所在的服务或调用链路。分布式追踪（Distributed Tracing）技术应运而生，它能够：

  • 可视化调用链路：清晰展示请求在各服务间的流转路径
  • 定位性能瓶颈：识别耗时最长的服务或操作
  • 快速故障定界：确定故障发生的具体服务节点
  • 分析依赖关系：发现服务间的隐式依赖

1.2 OpenTelemetry 简介

OpenTelemetry（简称 OTel）是 CNCF 旗下的开源项目，由 OpenTracing 和 OpenCensus 合并而成，提供了统一的可观测性信号（追踪、指标、日志）采集标准。其核心优势包括：

  • 厂商中立：不绑定特定后端，可对接 Jaeger、Zipkin、Prometheus 等多种后端
  • 语言覆盖全面：支持 Java、Go、Python、Node.js、.NET 等主流语言
  • 自动插桩：多数语言支持自动 instrumentation，无需修改业务代码
  • 统一 API：一套 API 同时支持追踪、指标、日志采集

二、架构设计

2.1 整体架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
┌─────────────────────────────────────────────────────────────────┐
│                        应用层                                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ 服务 A    │  │ 服务 B    │  │ 服务 C    │  │ 服务 D    │        │
│  │ (Java)   │  │ (Go)     │  │ (Python) │  │ (Node)   │        │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘        │
│       │             │             │             │               │
│       └─────────────┴──────┬──────┴─────────────┘               │
│                            │                                     │
│                    OTel SDK (自动/手动插桩)                        │
└────────────────────────────┼─────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                    OTel Collector 层                             │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              OTel Collector (Gateway)                    │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐      │    │
│  │  │   Receivers │  │  Processors │  │   Exporters │      │    │
│  │  │  (OTLP/gRPC)│  │  (批处理/   │  │ (Jaeger/    │      │    │
│  │  │  (OTLP/HTTP)│  │   采样/     │  │  Prometheus/│      │    │
│  │  │  (Zipkin)   │  │   过滤)     │  │  Elasticsearch)│   │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘      │    │
│  └─────────────────────────────────────────────────────────┘    │
└────────────────────────────┼─────────────────────────────────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
       ┌──────────┐   ┌──────────┐   ┌──────────┐
       │  Jaeger  │   │Prometheus│   │Elasticsearch│
       │ (追踪存储)│   │ (指标存储)│   │ (日志存储)  │
       └──────────┘   └──────────┘   └──────────┘

2.2 核心组件说明

组件 作用 部署建议
OTel SDK 应用内嵌，采集追踪数据 随应用部署
OTel Collector 接收、处理、导出追踪数据 独立部署，可集群
Jaeger/Tempo 追踪数据存储与查询 独立部署
Prometheus 指标存储与查询 独立部署

三、部署实践

3.1 部署 OTel Collector

OTel Collector 是 OpenTelemetry 的核心组件，负责接收、处理和导出可观测性数据。

3.1.1 Docker Compose 部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# docker-compose-otel.yaml
version: '3.8'

services:
  # Jaeger - 追踪后端
  jaeger:
    image: jaegertracing/all-in-one:1.54
    container_name: jaeger
    ports:
      - "16686:16686"  # Jaeger UI
      - "14250:14250"  # gRPC 接收
      - "14268:14268"  # HTTP 接收
      - "9411:9411"    # Zipkin 兼容
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - MEMORY_MAX_TRACES=100000
    restart: unless-stopped

  # OTel Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.95.0
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8888:8888"   # Prometheus 指标
      - "13133:13133" # Collector 健康检查
    depends_on:
      - jaeger
    restart: unless-stopped

  # Prometheus - 指标后端
  prometheus:
    image: prom/prometheus:v2.50.1
    container_name: prometheus
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
    restart: unless-stopped

volumes:
  prometheus_data:

3.1.2 Collector 配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
# otel-collector-config.yaml
receivers:
  # OTLP 接收器 (gRPC + HTTP)
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Zipkin 兼容接收器
  zipkin:
    endpoint: 0.0.0.0:9411

  # Prometheus 接收器 (拉取模式)
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['0.0.0.0:8888']

processors:
  # 批处理 - 提高导出效率
  batch:
    timeout: 1s
    send_batch_size: 1024
    send_batch_max_size: 2048

  # 采样 - 降低存储成本
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    policies:
      - name: error-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-policy
        type: latency
        latency:
          threshold_ms: 1000
      - name: probabilistic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

  # 资源属性处理
  resource:
    attributes:
      - key: service.namespace
        value: production
        action: upsert
      - key: deployment.environment
        value: prod
        action: upsert

  # 属性过滤 - 敏感信息脱敏
  attributes:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      - key: db.statement
        action: hash

exporters:
  # Jaeger 导出器
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  # Prometheus 导出器
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel

  # 日志导出器 (调试用)
  logging:
    loglevel: info
    sampling_initial: 5
    sampling_thereafter: 200

extensions:
  # 健康检查
  health_check:
    endpoint: 0.0.0.0:13133

  # Prometheus 指标
  pprof:
    endpoint: 0.0.0.0:1777

  # ZPages 调试
  zpages:
    endpoint: 0.0.0.0:55679

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      processors: [resource, attributes, tail_sampling, batch]
      exporters: [jaeger, logging]
    metrics:
      receivers: [otlp, prometheus]
      processors: [resource, batch]
      exporters: [prometheus, logging]
    logs:
      receivers: [otlp]
      processors: [resource, attributes, batch]
      exporters: [logging]

3.2 应用接入

3.2.1 Java 应用 (Spring Boot)

方式一：Java Agent 自动插桩（推荐）

1
2
3
4
5
6
7
8
9
10
11
# 下载 Java Agent
wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.36.0/opentelemetry-javaagent.jar

# 启动应用时添加 Agent
java -javaagent:./opentelemetry-javaagent.jar \
     -Dotel.service.name=my-spring-app \
     -Dotel.service.namespace=production \
     -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
     -Dotel.traces.sampler=parentbased_traceidratio \
     -Dotel.traces.sampler.arg=0.1 \
     -jar my-application.jar

方式二：手动 SDK 集成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<!-- pom.xml -->
<dependencies>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.36.0</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
        <version>1.36.0</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
        <version>1.36.0</version>
    </dependency>
    <!-- Spring Boot 自动配置 -->
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-spring-boot-starter</artifactId>
        <version>2.1.0-alpha</version>
    </dependency>
</dependencies>
1
2
3
4
5
6
7
8
9
10
11
# application.yml
opentelemetry:
  exporter:
    otlp:
      endpoint: http://otel-collector:4317
  service:
    name: my-spring-app
    namespace: production
  traces:
    sampler: parentbased_traceidratio
    sampler-arg: 0.1

3.2.2 Go 应用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package main

import (
    "context"
    "log"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() func() {
    ctx := context.Background()

    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("my-go-app"),
            semconv.ServiceNamespace("production"),
        )),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1),
        )),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return func() {
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        tp.Shutdown(ctx)
    }
}

func main() {
    shutdown := initTracer()
    defer shutdown()

    // 业务代码...
}

3.2.3 Python 应用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# 配置 TracerProvider
resource = Resource.create({
    "service.name": "my-python-app",
    "service.namespace": "production",
})

tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True),
        max_export_batch_size=512,
        schedule_delay_millis=5000,
    )
)

trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(__name__)

# 使用追踪
@tracer.start_as_current_span("process_order")
def process_order(order_id):
    with tracer.start_as_current_span("validate_order"):
        # 验证订单逻辑
        pass
    
    with tracer.start_as_current_span("charge_payment"):
        # 支付逻辑
        pass
    
    with tracer.start_as_current_span("update_inventory"):
        # 库存更新逻辑
        pass
1
2
3
# 自动插桩方式
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

3.3 Kubernetes 部署

3.3.1 使用 Helm Chart 部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# 添加 OpenTelemetry Helm 仓库
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# 创建 values.yaml
cat > otel-values.yaml << EOF
mode: deployment
presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeStateMetrics:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  
  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024
    k8sattributes:
      extract:
        metadata:
          - k8s.pod.name
          - k8s.namespace.name
          - k8s.deployment.name
      filter:
        node_from_env_var: K8S_NODE_NAME
  
  exporters:
    jaeger:
      endpoint: jaeger-collector:14250
      tls:
        insecure: true
  
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [k8sattributes, batch]
        exporters: [jaeger]
EOF

# 部署
helm install otel-collector open-telemetry/opentelemetry-collector \
  -f otel-values.yaml \
  -n monitoring

3.3.2 应用 Sidecar 注入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    instrumentation.opentelemetry.io/endpoint: http://otel-collector:4317
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        sidecar.opentelemetry.io/inject: "true"
    spec:
      containers:
      - name: my-app
        image: my-app:latest
        env:
        - name: OTEL_SERVICE_NAME
          value: "my-app"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector:4317"
        - name: OTEL_TRACES_SAMPLER
          value: "parentbased_traceidratio"
        - name: OTEL_TRACES_SAMPLER_ARG
          value: "0.1"

四、运维管理

4.1 采样策略配置

采样是降低追踪存储成本的关键手段。根据场景选择合适的采样策略：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# 组合采样策略示例
tail_sampling:
  decision_wait: 10s  # 等待时间，确保收集完整链路
  num_traces: 10000   # 缓存追踪数
  
  policies:
    # 1. 错误追踪 - 全部保留
    - name: keep-errors
      type: status_code
      status_code:
        status_codes: [ERROR]
    
    # 2. 慢请求 - 全部保留
    - name: keep-slow
      type: latency
      latency:
        threshold_ms: 2000
    
    # 3. 特定服务 - 全部保留
    - name: keep-critical-services
      type: string_attribute
      string_attribute:
        key: service.name
        values:
          - payment-service
          - order-service
        enabled_regex_matching: false
    
    # 4. 其他 - 概率采样
    - name: probabilistic-sampling
      type: probabilistic
      probabilistic:
        sampling_percentage: 5

4.2 性能调优

4.2.1 Collector 资源限制

1
2
3
4
5
6
7
8
# Kubernetes 资源限制
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi

4.2.2 批处理参数优化

1
2
3
4
5
6
7
processors:
  batch:
    # 增大批处理大小，提高吞吐量
    send_batch_size: 2048
    send_batch_max_size: 4096
    # 缩短超时时间，降低延迟
    timeout: 500ms

4.2.3 内存管理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 启用内存限制
extensions:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300

service:
  extensions: [memory_limiter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger]

4.3 高可用部署

4.3.1 Collector 集群部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Kubernetes StatefulSet 部署
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: otel-collector
spec:
  serviceName: otel-collector
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector
  template:
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.95.0
        args: ["--config=/etc/otel-collector-config.yaml"]
        ports:
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 4318
          name: otlp-http
        volumeMounts:
        - name: config
          mountPath: /etc/otel-collector-config.yaml
          subPath: config.yaml
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

4.3.2 负载均衡配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Service 配置
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  type: ClusterIP
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318
  selector:
    app: otel-collector
---
# Ingress 配置 (外部访问)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: otel-collector-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: otel.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: otel-collector
            port:
              number: 4318
  tls:
  - hosts:
    - otel.example.com
    secretName: otel-tls-secret

4.4 监控与告警

4.4.1 Collector 自身监控

1
2
3
4
5
6
7
8
9
10
# Prometheus 监控配置
scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8888']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'otelcol_.*'
        action: keep

4.4.2 关键告警规则

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# prometheus-rules.yaml
groups:
  - name: otel-collector
    rules:
      # Collector 宕机告警
      - alert: OtelCollectorDown
        expr: up{job="otel-collector"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "OTel Collector 实例宕机"
          description: "实例 {{ $labels.instance }} 已宕机超过 1 分钟"

      # 批处理队列积压
      - alert: OtelCollectorBatchQueueBacklog
        expr: rate(otelcol_processor_batch_batch_send_size_sum[5m]) < 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OTel Collector 批处理队列积压"
          description: "批处理速率过低，可能存在导出瓶颈"

      # 导出失败率高
      - alert: OtelCollectorExportFailure
        expr: rate(otelcol_exporter_sent_spans{state="failed"}[5m]) / rate(otelcol_exporter_sent_spans_total[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "OTel Collector 导出失败率高"
          description: "导出失败率超过 10%，请检查后端服务"

      # 内存使用过高
      - alert: OtelCollectorHighMemory
        expr: process_runtime_go_memstats_heap_alloc_bytes / process_runtime_go_memstats_heap_sys_bytes > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OTel Collector 内存使用过高"
          description: "内存使用率超过 80%，考虑扩容或优化配置"

4.5 故障排查

4.5.1 常见问题及解决方案

问题 可能原因 解决方案
追踪数据丢失 采样率过低 调整采样策略，提高关键服务采样率
Collector 内存溢出 批处理过大 减小 batch size，增加内存限制
链路不完整 跨服务传播失败 检查 HTTP Header 传播，确保 Trace Context 正确传递
Jaeger 查询慢 数据量过大 增加索引，优化存储配置
应用性能下降 SDK 同步阻塞 使用 BatchSpanProcessor，调整导出间隔

4.5.2 调试技巧

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 查看 Collector 日志
kubectl logs -f deployment/otel-collector -n monitoring

# 启用详细日志
# 在配置中添加:
service:
  telemetry:
    logs:
      level: debug

# 使用 ZPages 调试
kubectl port-forward svc/otel-collector 55679:55679
# 访问 http://localhost:55679/debug/tracez

# 检查 Collector 健康状态
curl http://localhost:13133

# 查看 Collector 指标
curl http://localhost:8888/metrics

五、最佳实践

5.1 Span 命名规范

1
2
3
4
5
6
7
8
9
# 推荐格式：{操作类型}.{资源/目标}
✅ HTTP GET /api/users → HTTP GET /api/users
✅ DB SELECT users → DB SELECT users
✅ RPC payment-service/Charge → RPC payment-service/Charge

# 避免
❌ /api/users (缺少操作类型)
❌ SELECT * FROM users WHERE id=? (包含敏感信息)
❌ function_123 (无意义名称)

5.2 属性标注规范

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 标准属性 (Semantic Conventions)
http.method: GET
http.url: https://api.example.com/users/123
http.status_code: 200
db.system: mysql
db.statement: SELECT * FROM users WHERE id = ?
db.operation: SELECT
rpc.system: grpc
rpc.service: UserService
rpc.method: GetUser

# 自定义属性 (业务相关)
user.id: 12345
order.id: ORD-2026-001
tenant.id: tenant-abc

5.3 安全考虑

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 敏感信息脱敏配置
processors:
  attributes:
    actions:
      # 删除敏感 Header
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      
      # 脱敏 SQL 语句
      - key: db.statement
        action: hash
      
      # 删除个人身份信息
      - key: user.email
        action: delete
      - key: user.phone
        action: delete

5.4 成本控制

  1. 合理采样：生产环境建议 1%-10% 采样率，错误和慢请求全量保留
  2. 数据保留：追踪数据保留 7-30 天，根据合规要求调整
  3. 存储优化：使用压缩存储，定期清理过期数据
  4. 资源限制：为 Collector 设置合理的 CPU/内存限制

六、总结

OpenTelemetry 作为云原生可观测性的事实标准，为分布式追踪提供了统一的采集和导出方案。通过合理的架构设计、采样策略和运维管理，可以在保证可观测性的同时控制成本。

核心要点回顾：

  1. 架构设计：应用 → Collector → 后端的三层架构，解耦采集与存储
  2. 采样策略：错误全量、慢请求全量、其他概率采样
  3. 性能调优：批处理参数、内存管理、资源限制
  4. 高可用：Collector 集群部署、负载均衡、健康检查
  5. 安全合规：敏感信息脱敏、访问控制、数据保留策略

通过本文的实践指南，您可以快速搭建生产级的 OpenTelemetry 分布式追踪系统，为微服务架构的可观测性提供坚实基础。