RocksDB Tuning

Block Cache: First and Most Important Knob

Where decompressed blocks live. Determines read latency.

BlockBasedTableOptions table_opts;
table_opts.block_cache = NewLRUCache(24L << 30);  // 24 GB
table_opts.cache_index_and_filter_blocks = true;
table_opts.pin_l0_filter_and_index_blocks_in_cache = true;
table_opts.cache_index_and_filter_blocks_with_high_priority = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_opts));

cache_index_and_filter_blocks puts the SST's index and bloom filter blocks into the cache (rather than always loading them on file open). Without it, every SST open reads index+filter from disk; with it, they stay hot. The "high priority" variant ensures they're not evicted by streaming data block churn.

pin_l0_filter_and_index_blocks_in_cache never evicts L0 filter/index blocks. Since L0 is checked on every Get, this prevents L0 filter loading from latency spikes.

Sizing: start at 25% of RAM. Watch the BLOCK_CACHE_HIT_RATIO ticker. If <90% on a read-heavy workload, increase. Above 95% is diminishing returns. Memtable + block cache + OS overhead should sum to ≤80% of RAM, leaving page cache headroom (or use direct I/O and claim more for the block cache).

Memtable and Write Buffers

Big memtables = fewer flushes = lower WA. Up to a point.

cf_opts.write_buffer_size = 256 << 20;        // 256 MB per memtable
cf_opts.max_write_buffer_number = 4;            // 4 memtables max
cf_opts.min_write_buffer_number_to_merge = 2;   // merge 2 before flush

// global RAM budget across all CFs
db_opts.write_buffer_manager.reset(new WriteBufferManager(2 << 30));  // 2 GB pool

Each CF's memtable holds writes until it fills write_buffer_size. Then it becomes immutable and a new memtable starts; the old one is flushed asynchronously to L0. Larger memtables mean (a) fewer flushes, (b) each flush produces a bigger L0 SST, (c) better duplicate-key elimination at flush time (writes to the same key are coalesced before disk).

max_write_buffer_number caps how many memtables can be pending flush. Hit it and writes block. min_write_buffer_number_to_merge can be raised to 2 to coalesce two memtables in one flush — saves L0 file count.

The WriteBufferManager pools memtable budget across CFs. Without it, an idle CF still holds onto its full memtable size. With it, idle CFs cost nothing. For deployments with many CFs, always use the manager.

Levels and Multipliers

Where the LSM tradeoff is set.

cf_opts.compaction_style = kCompactionStyleLevel;
cf_opts.num_levels = 7;
cf_opts.max_bytes_for_level_base = 256 << 20;        // L1 = 256 MB
cf_opts.max_bytes_for_level_multiplier = 10;          // L2 = 2.5 GB, etc.
cf_opts.target_file_size_base = 64 << 20;             // 64 MB SSTs
cf_opts.target_file_size_multiplier = 1;              // same size all levels

cf_opts.level0_file_num_compaction_trigger = 4;
cf_opts.level0_slowdown_writes_trigger = 20;
cf_opts.level0_stop_writes_trigger = 36;

The 10x multiplier is well-tested. Lowering to 5x reduces space amp (tombstones reach bottom faster) at cost of higher write amp. Raising to 20x reduces write amp at cost of more levels.

L0 stall triggers protect against cascade failure: if compaction can't keep up, slowdown kicks in at 20 L0 files and stops writes at 36. If you see writes stalling at L0 stop in production, your compaction can't keep up — increase parallel compaction threads or move to faster disk.

Compaction Threads and Rate Limiter

Throughput on the maintenance path; impact on the foreground.

db_opts.max_background_jobs = 8;        // total bg threads (flush + compact)
db_opts.max_subcompactions = 4;          // intra-compaction parallelism

// Rate limit compaction I/O so it doesn't starve user reads
db_opts.rate_limiter = NewGenericRateLimiter(
  80L * 1024 * 1024,    // 80 MB/s sustained
  100 * 1000,           // 100ms refill interval
  10                    // bonus burst factor
);

max_background_jobs is the total compaction+flush thread count. Set to core_count for compaction-heavy workloads. max_subcompactions lets a single compaction job split into chunks (parallel writers within one compaction). Useful for large compactions on multi-core hosts.

The rate limiter caps compaction's I/O bandwidth. Without it, a major compaction can saturate the disk and your p99 read latency explodes. Set it to ~80% of disk's sustained bandwidth. The bonus burst factor allows brief spikes for small compactions.

Compression Strategy

Different levels, different tradeoffs.

cf_opts.compression_per_level = {
  kNoCompression,        // L0: latency-sensitive
  kLZ4Compression,       // L1: balanced
  kLZ4Compression,       // L2
  kZSTDCompression,      // L3+: storage density
  kZSTDCompression,
  kZSTDCompression,
  kZSTDCompression
};

cf_opts.bottommost_compression = kZSTDCompression;
cf_opts.bottommost_compression_opts.level = 6;    // higher ZSTD level for biggest level

L0 is the hot tier: every Get checks all L0 files, and the data is recently written. Skip compression there — saves CPU on the hot path, costs little disk because L0 is small. L1-L2 use LZ4 (fast decompression, modest compression). L3+ use ZSTD; the deepest level uses ZSTD level 6 for maximum density since data here rarely gets read.

For datasets that compress well (JSON, text, low-cardinality columns), this strategy can cut disk usage 3-5x with minimal latency impact. For binary or already-compressed data, compression is a small loss — consider kNoCompression everywhere if you've verified your data doesn't compress.

Why TiKV Uses RocksDB and CockroachDB Switched to Pebble

Forks exist because the same kernel doesn't fit every host language.

TiKV (the Rust-based KV layer of TiDB) calls into RocksDB via Rust FFI. The C++/Rust boundary is well-tuned and FFI overhead is minimal. TiKV uses RocksDB's WritePrepared mode for percolator transactions, three CFs per region, and aggressive tuning for write-heavy OLTP workloads. They contribute back to RocksDB rather than fork.

CockroachDB originally used RocksDB via Cgo, but Cgo's call overhead (~100ns per crossing) hurt them on hot paths. In 2018 they began Pebble — a from-scratch Go reimplementation of RocksDB's LSM. Pebble is now their default. The advantages: zero Cgo overhead, better integration with Go's runtime (concurrency, profiling, GC), and the freedom to add Cockroach-specific features (their MVCC layer, SSTable format extensions). The disadvantage: parallel maintenance burden — RocksDB and Pebble are both moving targets, and porting features between them is real work.

YugabyteDB uses RocksDB from C++; Cassandra's row cache uses Cassandra's own SSTable code (not RocksDB); Kafka uses RocksDB for some materialized state. The pattern: RocksDB is the LSM kernel of choice for non-Go systems; Go projects increasingly use Pebble.

FAQ