Three Pillars (Plus One)

Three Different Shapes

Each pillar has a fundamentally different cost-per-question and shape of answer.

Key Numbers

Metrics: When to Use

Metrics answer aggregate questions cheaply. They lose information about individual events but compress beautifully and stay queryable for years.

• Health questions. "Is the error rate elevated right now?" needs counters and gauges, not log search.
• SLI computation. SLOs are defined as ratios of metric counts. Compute them from logs and you'll never afford the query.
• Long retention. Per-second metrics for a year cost gigabytes. The same fidelity in logs would cost terabytes.
• Alerts. Thresholds and burn rates evaluate cheaply against time series. Querying logs every 30 seconds for an alert is a money pit.
• Dashboards. Painting a graph from a metric is microseconds. Painting one from logs is seconds — users notice.

{`# Good metrics: low-cardinality, aggregable
http_requests_total{method,status,route}        # 5 x 40 x 200 = ~40K series
http_request_duration_seconds_bucket{route,le}  # 200 x 10 buckets = 2K series

# Bad metrics: identifiers as labels
http_requests_total{user_id, request_id, ...}   # unbounded, kills TSDB`}

Logs: When to Use

Logs preserve event-level information. Anything you want to remember about a single request, transaction, or operation goes in a log line. Modern logs are structured: not text, but JSON or logfmt with typed fields.

{`# Bad: unstructured log
[2024-08-12 14:23:11] ERROR: Failed to charge user [email protected] $42.99 (insufficient_funds)

# Good: structured log
{
  "ts":         "2024-08-12T14:23:11Z",
  "level":      "ERROR",
  "msg":        "charge failed",
  "user_id":    "u_abc123",
  "user_email": "[email protected]",
  "amount_usd": 42.99,
  "reason":     "insufficient_funds",
  "trace_id":   "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id":    "00f067aa0ba902b7",
  "service":    "billing",
  "version":    "v1.42.0"
}`}

The trace_id is the magic field. With it, every log line can be pivoted to its trace, and every trace span can pull the log lines emitted during it. Without it, logs are a search index that can only answer "show me logs matching X" — a much weaker question.

Traces: When to Use

Traces preserve causal structure: which call invoked which, with what parameters, taking how long. They're the only signal that captures the shape of a single request through a distributed system.

• "Why is this request slow?" The span tree shows exactly which child span ate the latency.
• "What does this request actually do?" The trace is documentation of the runtime behavior.
• "Where did this exception come from?" Spans carry exception events; the trace shows the call chain.
• "What's the dependency graph?" Aggregating traces produces a service-to-service map.

Traces are sampled because at full fidelity they cost more than logs. But unlike metrics, sampled traces still preserve the question shape — you just have a 1% sample of the request shapes, not aggregate counts.

Profiles: The Fourth Pillar

Continuous CPU and memory profiling captures stack-frame frequencies across the fleet at low cost. It answers questions metrics, logs, and traces can't: "which line of code is consuming all this CPU?"

• Cross-cutting performance. A function used in 50 places shows up in profiles by total cost, not 50 separate spans.
• Hot leaves. JSON parsing eating 30% CPU? Profile sees it. Trace doesn't — it's not a span.
• Memory leaks. Heap profile shows live allocations by stack — the leaking call site is right there.
• Diff before/after. Compare last week's flamegraph to this week's. The diff highlights regressions a metric never would.

Exemplars: Linking Metrics to Traces

An exemplar is a trace_id attached to a metric sample, typically a histogram bucket. Click the spike on the latency dashboard, drill into a trace from that exact moment.

{`# OpenMetrics exemplar format - trace_id + span_id attached to a histogram bucket
http_request_duration_seconds_bucket{le="0.5",route="/api"} 12492 # {trace_id="4bf92f...",span_id="00f067..."} 0.473 1660354527.123

# In Prometheus + Grafana + Tempo:
#   - Prometheus stores the exemplar pointer with the bucket
#   - Grafana surfaces "view trace" links on the histogram
#   - Click jumps to Tempo with the trace_id pre-filled

# OTel SDK enables exemplars by default for histograms when:
#   - the span is currently sampled
#   - the metric is OTel-emitted (not Prometheus-direct)`}

Exemplars are the cheapest way to bridge the metrics-traces gap. Each histogram bucket holds at most one exemplar at a time (the latest), so the storage cost is tiny — but the navigation value is huge.

Vendor-Neutral Stack

The full unified-observability stack today, all vendor-neutral and OTel-friendly.

Tradeoffs

FAQ