OpenTelemetry

The Pipeline

Every OTel deployment looks like the same five-stage pipeline: receive, process, route, export, ingest. The shape doesn't change; only the components plugged into each stage do.

Key Numbers

The OTLP Protocol

OTLP (OpenTelemetry Protocol) is the wire format. Protobuf schemas define metrics, logs, traces, and (in development) profiles. Available over gRPC (port 4317) and HTTP/protobuf or HTTP/JSON (port 4318).

{`# Top-level OTLP messages
message ExportTraceServiceRequest    { repeated ResourceSpans   resource_spans = 1; }
message ExportMetricsServiceRequest  { repeated ResourceMetrics resource_metrics = 1; }
message ExportLogsServiceRequest     { repeated ResourceLogs    resource_logs = 1; }

# Each ResourceSpans groups spans by Resource (service.name, k8s.pod.name, ...)
message ResourceSpans {
  Resource   resource    = 1;       // shared service-level attributes
  repeated ScopeSpans scope_spans = 2;  // grouped by instrumentation library
}

# gRPC service definition
service TraceService {
  rpc Export(ExportTraceServiceRequest) returns (ExportTraceServiceResponse);
}

# HTTP/protobuf
POST /v1/traces HTTP/1.1
Content-Type: application/x-protobuf


# HTTP/JSON (less efficient but easier for browser RUM)
POST /v1/traces HTTP/1.1
Content-Type: application/json
{"resourceSpans": [...]}`}

Resource Detection

Every signal carries a Resource — a set of attributes that describe the producer. service.name, service.version, k8s.pod.name, cloud.region, host.id. The SDK auto-detects these from the environment.

{`# Environment-based detection
OTEL_SERVICE_NAME=billing-api
OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,service.version=v1.42.0,deployment.environment=prod

# K8s resource detector populates automatically:
#   k8s.pod.name        from $POD_NAME
#   k8s.namespace.name  from $POD_NAMESPACE
#   k8s.node.name       from $NODE_NAME
#   k8s.container.name  from runtime API

# Cloud detectors hit instance metadata services:
#   cloud.provider     = aws | gcp | azure
#   cloud.region       = us-east-1
#   cloud.account.id
#   host.id            = i-0abcdef...

# Resource attributes are global - applied to every span/metric/log from this process`}

Processors: Where the Pipeline Logic Lives

Processors transform telemetry between receive and export. They run in pipeline order and can drop, modify, enrich, batch, or sample.

Collector Pipeline Config

The full mental model: receivers feed processors feed exporters, all stitched together by named pipelines.

{`# otelcol.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  prometheus:
    config:
      scrape_configs:
        - job_name: 'k8s-pods'
          kubernetes_sd_configs: [{ role: pod }]

processors:
  memory_limiter: { check_interval: 1s, limit_mib: 4000 }
  batch: { send_batch_size: 8192, timeout: 1s }
  attributes:
    actions:
      - { key: user.email, action: delete }    # PII redaction
      - { key: deployment.environment, value: prod, action: insert }
  tail_sampling:
    decision_wait: 30s
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency,     latency: { threshold_ms: 500 } }
      - { name: rest,   type: probabilistic, probabilistic: { sampling_percentage: 1 } }

exporters:
  otlphttp/tempo:    { endpoint: http://tempo:4318 }
  prometheusremotewrite: { endpoint: http://mimir:9090/api/v1/push }
  otlphttp/loki:     { endpoint: http://loki:3100/otlp }

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, attributes, tail_sampling, batch]
      exporters:  [otlphttp/tempo]
    metrics:
      receivers:  [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters:  [prometheusremotewrite]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, attributes, batch]
      exporters:  [otlphttp/loki]`}

OpAMP: Fleet Management

OpAMP (Open Agent Management Protocol) is the answer to "how do I manage 10,000 collectors without SSH?". A central control plane pushes config, certificates, and binary updates to every agent. Each agent reports back its health, version, and currently active config hash.

{`# Collector OpAMP extension (subset)
extensions:
  opamp:
    server:
      ws: { endpoint: wss://opamp.example.com/v1/opamp }
    capabilities:
      reports_effective_config:    true
      accepts_remote_config:       true
      reports_status:              true
      accepts_packages:            true       # binary updates
      reports_package_statuses:    true

# Server pushes a new config; collector validates and reloads
# Server pushes a new collector binary (signed); collector swaps and restarts
# Collector reports back its actual running config hash for audit

# Vendor support:
#   - BindPlane (Observiq) - OpAMP server with config UI
#   - Grafana Agent Cloud
#   - Various commercial SaaS`}

Semantic Conventions

The semconv spec standardizes attribute names and values. http.method, not method. db.system=postgresql, not db_kind. Backends rely on these names for built-in dashboards.

Semconv has had breaking changes (http.method → http.request.method). Check the maturity tier (Stable / Experimental / Deprecated) before relying on a name.

Tradeoffs & Gotchas

FAQ