TiDB Internals

Architecture Overview

Key Numbers

Why TiDB Exists

The Three Tiers

A TiDB deployment has exactly three component types, each scaled independently:

The split lets you run, say, 30 tidb-servers (CPU-bound query execution), 6 TiKV stores with NVMe (I/O bound storage), 3 PD nodes (tiny, only metadata), and 4 TiFlash nodes (analytics workloads). Compare to CockroachDB where every node runs SQL + storage + gossip together — simpler to deploy but harder to scale heterogeneously.

TiKV: Range-Sharded RocksDB

Every TiKV store runs two RocksDB instances per data directory: kv holds the user data, raft holds Raft logs. The global key space is partitioned into Regions, each a contiguous [start_key, end_key) range with a default target of 96 MB and a split threshold of 144 MB.

// PD's region metadata for a table row
Region {
  id: 12_345,
  start_key: t\x80\x00\x00\x00\x00\x00\x00\x07_r\x80\x00\x00\x00\x00\x00\xa1\x00,
  end_key:   t\x80\x00\x00\x00\x00\x00\x00\x07_r\x80\x00\x00\x00\x00\x01\xa1\x00,
  region_epoch: { conf_ver: 3, version: 22 },
  peers: [
    { id: 12_346, store_id: 1, role: Voter },     // Raft leader (probably)
    { id: 12_347, store_id: 2, role: Voter },
    { id: 12_348, store_id: 3, role: Voter },
    { id: 12_349, store_id: 7, role: Learner }    // TiFlash learner
  ]
}

The t\x80... prefix encodes table ID 7. The _r separator distinguishes data rows from indexes (which use _i). Inside the row prefix, the row's primary key is encoded in TiDB's memcomparable format so RocksDB's bytewise-sorted SST files yield correct SQL ordering. Every Region runs an independent Raft group; region_epoch.version bumps on every split/merge so stale routes are rejected by TiKV.

The Placement Driver (PD)

PD is the cluster's brain. A 3- or 5-node etcd-backed Raft group, it holds every Region's location, allocates monotonically increasing timestamps (the TSO, fetched per transaction), and runs the scheduler that moves Region replicas around to balance load.

• TSO allocator. Hands out (physical_ms, logical_counter) pairs. tidb-servers batch up to 10 000 TSO requests per RPC. Latency: ~1ms in-DC, dominates short txns.
• Region heartbeats. Every TiKV leader heartbeats its Region to PD every 60s with size, key count, read/write QPS. PD uses these to decide moves.
• Schedulers. balance-region-scheduler evens storage; balance-leader-scheduler evens leadership; hot-region-scheduler reacts to write hot spots; evict-leader-scheduler drains a node before maintenance.
• Placement rules. Constrain replicas by labels: e.g., "3 voters spread across 3 zones, 1 learner in zone D for TiFlash."

Percolator Transactions

TiDB transactions follow the Google Percolator paper. Snapshot reads are easy: each transaction takes a start_ts from PD and reads the most recent committed version <= start_ts for every key. Writes are a 2-phase protocol with a "primary" key that linearizes the commit.

// Pseudo-code of a Percolator commit
fn commit(write_set: Vec<(Key, Value)>) -> Result<()> {
  let start_ts = pd.get_tso();
  let primary    = write_set[0].0.clone();    // pick the first key

  // PHASE 1 — prewrite every key
  for (key, value) in &write_set {
    tikv.prewrite(key, value, primary.clone(), start_ts)?;
    // writes a Lock record at (key, start_ts) pointing to primary
    // FAILS if any other lock or any write > start_ts exists
  }

  // Get commit_ts AFTER all prewrites (so commit_ts > start_ts of any
  // overlapping prewrite that locked the same key first)
  let commit_ts = pd.get_tso();

  // PHASE 2 — commit primary first, then secondaries
  tikv.commit(primary.clone(), start_ts, commit_ts)?;       // durable point
  for (key, _) in &write_set[1..] {
    tikv.commit(key.clone(), start_ts, commit_ts)?;          // best-effort
  }
  Ok(())
}

The clever bit. If the coordinator crashes after committing the primary but before committing all secondaries, the transaction is still semantically committed — any future reader who encounters a stale lock on a secondary will look up the primary's commit record and roll the secondary forward (or back if the primary was rolled back). This means there is no central commit log, and the system recovers from any single failure without a coordinator.

The cost. Each key in the write set is at least 2 RPCs. A 100-key transaction across 5 Regions on 3 stores is ~200 RPCs. TiDB has an "async commit" mode (since 5.0) that returns to the client after prewrite completes (skipping the commit_ts roundtrip), and a "1PC" fast path for transactions whose entire write set fits in one Region.

Coprocessor Pushdown

Naively, tidb-server would fetch every row from TiKV and filter/aggregate in Go. That would be unworkable — the network would saturate. Instead, TiDB pushes execution fragments down into TiKV via the coprocessor interface: predicates, projections, simple aggregates, top-N, and even some hash join build-sides execute inside the TiKV process.

EXPLAIN ANALYZE SELECT category, COUNT(*)
FROM orders
WHERE created_at >= '2025-01-01' AND status = 'shipped'
GROUP BY category;

  HashAgg            (final agg in tidb-server)
  └─ TableReader     (fetches partial aggregates)
     └─ HashAgg      ◀ pushed down — runs in TiKV coprocessor
        └─ Selection ◀ pushed down — predicate evaluated in TiKV
           └─ TableFullScan

With pushdown, only the per-Region partial aggregates cross the network. For TiFlash, the entire MPP plan can run in the columnar layer with shuffle exchanges between TiFlash nodes, returning a final result to tidb-server. This is what makes HTAP queries on TiDB competitive with dedicated OLAP systems on the same hardware.

TiFlash: The Columnar Replica

A standard TiKV Region has 3 voting Raft peers. Add a learner and you get a 4th replica that participates in Raft log replication but doesn't vote. TiFlash is exactly that: a learner whose state machine is a columnar storage engine derived from ClickHouse.

-- Add 2 TiFlash replicas for the orders table
ALTER TABLE orders SET TIFLASH REPLICA 2;

-- Check status
SELECT TABLE_SCHEMA, TABLE_NAME, AVAILABLE, PROGRESS
FROM information_schema.tiflash_replica;

-- Force the optimizer to use TiFlash
SELECT /*+ READ_FROM_STORAGE(TIFLASH[orders]) */
  category, SUM(total)
FROM orders
GROUP BY category;

Because TiFlash is in the Raft group, reads are snapshot-consistent with TiKV under the same Percolator timestamp. The optimizer maintains a cost model that knows how much each engine charges for a scan (TiFlash cheap, TiKV expensive for full scans) versus a point lookup (TiKV cheap, TiFlash expensive). It picks per-table-access, so a single query can mix engines.

Online Schema Changes

TiDB implements the F1 paper's "online async schema change" algorithm. A DDL goes through five public states — none → delete-only → write-only → write-reorganization → public — and at every transition there are at most two adjacent states active across the cluster simultaneously. tidb-servers poll the schema version and lazy-load.

• Add column. Non-blocking. The new column is null/default until backfilled in the write-reorg state.
• Add index. Non-blocking. Backfill scans the table in batches; concurrent writes update both old and new state per F1.
• Drop column. Goes through delete-only first, then physical removal happens in the GC pass.
• Lossy data type changes (e.g., VARCHAR(10) → VARCHAR(5)): require an offline migration. TiDB is honest about it and rejects them.

Tradeoffs & When To Use

TiDB vs CockroachDB

FAQ