Observability Internals

The Three Pillars (And the Fourth)

Observability has historically rested on three signal types. Metrics are pre-aggregated numerical time-series — counters, gauges, histograms. They are cheap to store (one number plus a few label values per scrape interval) and great for alerting and dashboards, but they are dimensional and lose individual events. Logs are structured records of discrete events. Expensive to store at high cardinality but irreplaceable for "what exactly happened to this request" questions. Traces are causally-linked records of work spanning processes — span trees that show how a request flowed through services.

A fourth pillar has emerged: continuous profiling. Sampled CPU/memory profiles from production with negligible overhead, stored as flame graphs over time. Pyroscope, Parca, and Polar Signals proved you can run pprof-style profilers continuously and answer questions metrics and traces can't ("which line of code is using my CPU"). OpenTelemetry's profiling signal is the formalisation.

The interesting question isn't "which pillar?" but how they correlate. Exemplars attach a sample trace_id to a histogram bucket: a high-latency p99 metric link directly to the slow trace. Logs include trace_id in their structured fields so a single trace ID jumps from "slow request" to "the exact log line where it stalled." This is the wholesale shift from "three pillars" to "one signal graph with multiple projections."

OpenTelemetry: The Standard

OTel is three things: a specification (data model + semantic conventions), a set of SDKs in every major language, and a Collector binary that receives, processes, and exports telemetry. The wire protocol, OTLP (OpenTelemetry Protocol, gRPC or HTTP), is the canonical format. Every vendor accepts OTLP at the front door now.

The semantic conventions are arguably more important than the protocol. They define standardised attribute names: http.request.method, db.system.name, messaging.kafka.partition. When everyone agrees these names, dashboards built for one service work for all of them; correlation across services becomes trivial. Vendor SDKs that stick OTel-incompatible attribute names on spans break this for everyone.

Auto-instrumentation libraries (Java agent, Python opentelemetry-instrument, .NET, Node) attach to popular frameworks and emit OTLP without code changes. Manual instrumentation fills gaps the auto layer can't see. The combination usually gives you 80% coverage in an afternoon.

Distributed Tracing and Context Propagation

A trace is a directed graph of spans, each representing a unit of work (an HTTP request, a DB query, an internal function). Spans have a start, an end, attributes, events, and a parent span ID. The trace is identified by a 128-bit trace_id; spans by a 64-bit span_id. The whole graph is reconstructed by joining child spans to their parents.

For a trace to span processes, the trace context must travel with the request. The W3C Trace Context standard defines two headers: traceparent (the binary identifiers in a fixed format) and tracestate (vendor-specific data). All OTel-compliant libraries inject and extract these headers automatically across HTTP, gRPC, and most messaging clients. The older B3 headers (Zipkin) are still common in legacy environments.

A subtle correctness pitfall: if your trace context propagation is incomplete (you call a service that doesn't know how to forward the headers), spans become orphans. They show up as separate small traces. The fix is auto-instrumentation everywhere or explicit propagation in any custom client. Treat the first lost trace boundary as a P0 — the entire blast radius of "we don't know what's happening" lives downstream of it.

Sampling: Head-Based vs Tail-Based

Recording every span at scale is impossibly expensive. Sampling is mandatory. The two strategies have very different costs and properties.

Head-based sampling: decide at trace start whether to keep the trace, before any work happens. Cheap (no buffering), simple (the decision propagates through the parent context so all spans in a trace are kept or dropped together), but blind — you can't make the decision based on whether the trace turned out interesting. A 1% sample rate misses 99% of rare bugs by definition.

Tail-based sampling: buffer all spans for a trace, decide at the end whether to keep. Lets you sample 100% of errors and slow traces while keeping baseline at 1%. Expensive: the sampler must hold every span in memory until the trace completes (or a timeout fires). Requires a sampling decision service that sees all spans for a trace ID — usually a sticky-routed Collector with a tail-sampling processor.

Hybrid: head-sample the boring 99%, tail-sample errors and outliers. This is the dominant production pattern. Honeycomb's "refinery" and the OTel Collector's tail_sampling processor implement it.

The Cardinality Problem

A metric series is identified by metric name plus the set of label values. Each unique combination is a new time-series. A counter http_requests_total{method, path, status, user_id} with even modest cardinality on each label can produce billions of series. Prometheus, Mimir, VictoriaMetrics — all hit a wall around 10–50M active series before query latency, ingest, and storage costs explode.

Cardinality discipline: never put unbounded values in labels (user_id, request_id, trace_id, customer email). Aggregate at ingest where possible. If you need user-level breakdowns, do it on logs or traces with exemplars, not metrics. The number-one cause of "our observability bill is bigger than our compute bill" is undisciplined labels.

Modern systems (Mimir, VictoriaMetrics, ClickHouse-backed metrics stores) handle higher cardinality than classic Prometheus because they use better storage engines (label-indexed columnar) — but no one handles unbounded cardinality. Pre-aggregation and recording rules are still the answer.

Logs: Structured, Correlated, Aggregated

Modern logging is structured: JSON or logfmt with fixed fields. Free-text logs are searchable but un- queryable; structured logs are queryable like a database. The OTel logs SDK ships log records with span context attached, so every log line in a request handler automatically carries the trace ID.

The aggregation pipeline is canonical: app → stdout → log shipper (Vector, Fluent Bit, OTel Collector with logs receiver) → storage (Loki, Elasticsearch, ClickHouse, Splunk, S3+Athena). Loki's design point is "index labels only, store raw log content compressed" — dramatically cheaper than Elasticsearch for log volume but slower for full-text search. ClickHouse-backed log stores (SigNoz, Quickwit, vendor stacks) split the difference.

eBPF: Observability Without Instrumentation

eBPF programs run in the kernel, hooked to events: syscalls, network packets, function entries. They can produce metrics, traces, and profiles without touching application code. Cilium Tetragon, Pixie, Parca, and Hubble all leverage eBPF for system-call-level visibility.

The win is huge for closed-source binaries, polyglot environments, and "drop-in observability" claims. The limit is that the kernel sees syscalls, not application semantics — eBPF can tell you a syscall took 5 ms but not which user-level operation that syscall was serving. Combined with OTel auto-instrumentation, the two cover complementary blind spots.

SLI, SLO, SLA, Error Budgets

The Google SRE framework that became universal. SLI: Service Level Indicator — a measurable metric like "p99 latency of the checkout endpoint" or "error rate of writes to user table." SLO: Service Level Objective — a target ("p99 below 300 ms 99.9% of the time over 28 days"). SLA: Service Level Agreement — a contractual SLO with consequences for violation.

An error budget is the inverse of an SLO: if you commit to 99.9%, you have 43 minutes of allowed downtime per month. Burn-rate alerts (Google's multi-window multi-burn-rate pattern, MWMBR) page humans only when you're about to blow the budget — not on every blip. A 14.4× burn rate over a 1-hour window means "in the next 5 days at this rate you'll exhaust the month's budget" — alert. A 1.5× burn rate over 6 hours means "you'll exhaust the budget in 18 days at this rate" — also worth paging because the issue is sustained.

The three classic monitoring frameworks: RED (Rate, Errors, Duration — for request- driven services), USE (Utilization, Saturation, Errors — for resources), and the four golden signals (latency, traffic, errors, saturation — Google's mash-up). They all describe the same idea: instrument both the request flow and the resources serving it.

Tradeoffs and When Less is More

Modern observability is expensive. A typical bill at scale: 30–60% of compute cost. Most of it is metric cardinality and log volume that nobody queries. The discipline is to keep alerting metrics surgical, sample traces aggressively, ship logs only when you need them, and use exemplars to bridge between cheap signals and expensive ones.

The opposite mistake is also common: instrument too lightly, ship to a single tool, then realise after an incident that you have no idea what happened. The right floor for a production service is: RED metrics on every endpoint, structured logs at info+, distributed tracing at 1% head sampling + 100% errors, and continuous profiling. Everything beyond that is workload-specific.