High-Cardinality Metrics

Cardinality Explosion (Visualized)

Each label dimension multiplies. The same metric name can produce one series or one billion, depending only on what you label.

Key Numbers

What Counts as a Time Series

A time series in Prometheus is uniquely identified by the metric name plus the full set of label key/value pairs. Two samples differ in any label and they are different series, each with its own chunk file, its own index entry, and its own RAM footprint.

{`# These are FOUR distinct time series:
http_requests_total{method="GET", status="200", route="/api/v1/orders"}
http_requests_total{method="GET", status="200", route="/api/v1/users"}
http_requests_total{method="POST", status="201", route="/api/v1/orders"}
http_requests_total{method="GET", status="500", route="/api/v1/orders"}

# Adding user_id="12345" creates a series for EACH user EACH route EACH status.
# A 10M-user app with 200 routes and 40 statuses => 80 BILLION potential series.`}

The series identifier is hashed with FNV-64 to produce a series ID, which becomes a key in the in-memory index. Even if a series sees only one sample per day, its index entry stays resident as long as that sample is within the retention window.

The Cost Inside Prometheus

Cardinality cost is not just RAM. It hits four subsystems simultaneously, each with a different threshold.

Detecting the Worst Offenders

Prometheus exposes its own internals as metrics. A handful of PromQL queries find the labels destroying your TSDB before they fall over.

{`# Top metrics by cardinality
topk(10, count by (__name__)({__name__=~".+"}))

# Top label keys by distinct values across the whole TSDB
topk(20, count by (label) (label_values_count))

# How many series each scrape target is producing
topk(10, count by (job, instance) ({__name__=~".+"}))

# Series churn rate (new series being created per second)
rate(prometheus_tsdb_head_series_created_total[5m])

# Active head series (the live cardinality)
prometheus_tsdb_head_series

# WAL truncation lag - high values mean the head can't keep up
prometheus_tsdb_wal_truncations_failed_total`}

The tsdb-analyze CLI tool does the same offline against a snapshot, printing per-metric cardinality, series count by label key, and the top label values. Run it weekly — cardinality drift is gradual and easy to miss until the OOM.

Label Dropping & Recording Rules

The cheapest fix is to never ingest the offending label. The next cheapest is to pre-aggregate at scrape time so the high-cardinality series are dropped from the TSDB after their useful information has been extracted.

{`# prometheus.yml - drop user_id at scrape time
scrape_configs:
  - job_name: 'api'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_requests_total'
        action: labeldrop
        regex: 'user_id|request_id|trace_id'

# Or replace high-cardinality URL paths with route templates
      - source_labels: [path]
        regex: '/users/[0-9]+/orders/[a-f0-9-]+'
        target_label: path
        replacement: '/users/:id/orders/:uuid'`}

Recording rules pre-compute aggregations at evaluation time and store the result as a new (lower-cardinality) series. The original high-cardinality series can then be dropped during compaction or held only briefly.

{`# rules.yml
groups:
  - name: api_rollups
    interval: 30s
    rules:
      - record: http_requests:rate5m_by_route
        expr: sum by (route, status) (rate(http_requests_total[5m]))
      - record: http_request_duration:p99_by_route
        expr: histogram_quantile(0.99,
                sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))`}

When to Use Logs or Traces Instead

The strongest cardinality discipline is "if it identifies a single request or user, it goes on a log line or a trace span, never on a metric label."

Cortex / Mimir Cardinality Limits

Multi-tenant Prometheus systems impose hard limits per tenant. These prevent one noisy tenant from taking down the cluster.

{`# mimir.yaml - per-tenant runtime overrides
overrides:
  tenant-a:
    max_global_series_per_user: 1000000          # hard ceiling
    max_global_series_per_metric: 100000         # per-metric ceiling
    max_label_names_per_series: 30               # reject samples with too many labels
    max_label_value_length: 2048
    ingestion_rate: 100000                       # samples/sec
    ingestion_burst_size: 200000

  tenant-b:
    max_global_series_per_user: 100000
    max_global_series_per_metric: 10000
    out_of_order_time_window: 10m`}

When a tenant exceeds max_global_series_per_user, new series are rejected at the distributor with HTTP 429. Existing series keep ingesting. This is a hard wall designed to protect the cluster, but it produces angry pages from the tenant who hit it — they need pre-warning via cardinality alerts.

Tradeoffs

FAQ