Cassandra Internals

Architecture Overview — The Token Ring

Key Numbers

Why Cassandra Exists

The Wide-Column Data Model

Cassandra's data model has three keys: a partition key decides which node owns the data; a clustering key orders rows within a partition; the rest are regular columns. CQL makes this look like SQL but the physical layout is what matters.

CREATE TABLE messages (
  user_id    UUID,
  msg_time   TIMEUUID,
  sender     TEXT,
  body       TEXT,
  attachment BLOB,
  PRIMARY KEY ((user_id), msg_time)        -- (PK), CK
) WITH CLUSTERING ORDER BY (msg_time DESC)
  AND compaction = { 'class': 'TimeWindowCompactionStrategy',
                     'compaction_window_size': 1,
                     'compaction_window_unit': 'DAYS' }
  AND default_time_to_live = 2592000;       -- 30 days

-- Logical row layout for one user's partition:
-- user_id=alice  | msg_time=t1 | sender=bob | body=...| attachment=...
--                | msg_time=t2 | sender=eve | body=...| attachment=null
--                | msg_time=t3 | ...

-- Physical layout on disk (one SSTable, partition contiguous):
-- alice : t1:sender=bob, t1:body=..., t2:sender=eve, t2:body=..., ...

Three rules of thumb that fall out of this layout:

• Partition by access pattern. All rows for the same partition key live together on one node — perfect for "give me the last 50 messages for user X" — but a partition that grows unbounded (say, all events ever) becomes a hotspot and a memory bomb at compaction time. Aim for partitions under ~100 MB.
• Clustering keys define on-disk sort order, so range queries within a partition are pure sequential reads.
• WHERE without partition key requires ALLOW FILTERING, which scans every node. The system warns you because it almost always means the schema is wrong for the query.

The Token Ring & Vnodes

Cassandra hashes every partition key (default Murmur3Partitioner) onto a 64-bit token. The unsigned token space is conceptually a ring; each node owns one or more contiguous ranges of it. To find a key's replicas, walk the ring clockwise starting at the key's token and pick the first RF distinct nodes (constrained by datacenter and rack with NetworkTopologyStrategy).

Without vnodes, each node owns exactly one range. Bootstrapping a new node means streaming data from its two neighbors only — slow, and hot-spotted on those two source nodes. With vnodes (default 16 in 4.x, was 256 in 3.x), each physical node owns many small ranges scattered around the ring. Bootstrapping streams from many sources in parallel, and load is distributed even with heterogeneous node sizes (give a bigger node more vnodes).

# cassandra.yaml
num_tokens: 16                               # vnodes per physical node
allocate_tokens_for_local_replication_factor: 3
endpoint_snitch: GossipingPropertyFileSnitch  # rack/DC awareness
partitioner: org.apache.cassandra.dht.Murmur3Partitioner

The 256-vnode default in 3.x was a known mistake — too many vnodes meant every range query touched too many nodes, multiplying coordinator latency. 4.x dropped to 16 with smarter token allocation that keeps load even.

The Write Path

A write hits the coordinator (any node — clients pick one) which forwards it to all RF replicas. Each replica's local path is the classic LSM:

CLIENT
  └─ coordinator (any node)
       └─ partitioner.getToken(partition_key)
       └─ ring.getReplicas(token, RF)
       └─ for each replica:
            ├─ append to commit log (sequential, fsync per group)
            ├─ insert into memtable (in-memory sorted map per table)
            └─ ack to coordinator
       └─ wait for CL acks (e.g., LOCAL_QUORUM = 2 of 3)
       └─ return to client

ASYNC: when memtable hits ~1 GB or 60s timer
       ├─ flush memtable → new SSTable on disk (immutable)
       ├─ build bloom filter, partition index, summary
       └─ truncate commit log segments fully covered by SSTables

Hinted handoff. If a replica is down, the coordinator stores a hint (the write, addressed to the down replica) in a local hints file. When the replica returns within max_hint_window_in_ms (default 3 hours), the coordinator replays the hints. After 3 hours, hints are dropped — the replica catches up via read repair or anti-entropy.

SSTables & Compaction Strategies

SSTables are immutable sorted runs of (partition_key, clustering_key, column) → value entries. A single Cassandra table on a single node typically has dozens to hundreds of SSTables. Reads must consult all of them (filtered by bloom filter and min/max key range) and merge results, with later versions winning. Compaction is the background process that merges SSTables to keep this manageable.

Choosing wrong is one of Cassandra's classic foot-cannons. A messaging app that picked STCS will get progressively slower reads as old messages pile up. The same app on TWCS drops yesterday's window in O(1) once every row has expired.

The Read Path

Reads at QUORUM contact ceil(RF/2)+1 replicas and merge results. Each replica's local read is multi-stage:

READ for (partition_key, clustering_key)
  1. Check row cache (off-heap, default off — usually skipped)
  2. Check key cache: maps partition_key → SSTable offset
  3. For each candidate SSTable (filtered by min/max partition token):
       a. Check bloom filter (FP rate ~1%, kept in memory)
       b. If bloom positive, binary search partition summary
       c. Read partition index entry from disk
       d. Seek to offset, read partition data, filter by clustering_key
  4. Merge results from all SSTables + memtable, latest version wins
  5. Apply tombstones

Read repair. When the coordinator gets responses from multiple replicas at QUORUM, it compares them. If they disagree (latest timestamp wins), it pushes the merged result to the stale replicas synchronously before returning. There's also read_repair_chance for digest-only reads — the coordinator occasionally asks a replica to send a checksum instead of the value, and triggers a full read+merge if it disagrees.

Tunable Consistency: R + W > N

Every read and write picks a consistency level. The system gives you strong consistency for that operation if and only if R + W > N, where N is the replication factor.

The most common production choice is RF=3, R=W=LOCAL_QUORUM: strong consistency within a datacenter, no WAN dependency for local operations, tolerates 1 node failure transparently. Cross-DC consistency is achieved by writes propagating asynchronously between DCs (one coordinator forwards to a coordinator in each DC).

Gossip & the Phi Failure Detector

Cassandra has no leader, no master, no central registry. Cluster membership is maintained by gossip: every second, each node picks 1-3 random peers and exchanges a state digest containing version-clocked tuples (heartbeat, status, schema version, load). After log(N) gossip rounds, news has reached the entire cluster.

Failure detection uses the Phi Accrual detector. Rather than a fixed timeout ("dead at 5s"), each node tracks the inter-arrival time history of gossip messages from each peer. The detector emits a continuously-valued phi (suspicion level) — when phi exceeds phi_convict_threshold (default 8), the node is marked DOWN. This adapts to network conditions: a node consistently slow is given more leeway than a node that goes silent abruptly.

Anti-Entropy Repair

Read repair and hinted handoff handle the common case, but eventually replicas can drift. Periodic repair reconciles them by computing Merkle trees per token range and exchanging hashes.

# Run on every node, every (gc_grace_seconds - safety) — e.g., weekly for default 10d grace
nodetool repair -pr keyspace1                    # primary range only

# Incremental repair (4.0+) — only un-repaired SSTables
nodetool repair -inc -pr keyspace1

# How it works internally:
# 1. Each replica builds a Merkle tree over the keyspace ranges
# 2. Replicas exchange Merkle tree hashes
# 3. For ranges where hashes differ, replicas stream the data
# 4. Newest timestamps win on merge

Without periodic repair, tombstones older than gc_grace_seconds can be purged on some replicas before they've reached others, causing "deleted" data to resurrect (the infamous "zombie data" problem). For this reason, Cassandra operations include scheduled repair as a hard requirement, not a luxury.

Tradeoffs & When To Use

Cassandra vs ScyllaDB

FAQ