DuckDB Storage Format
Row Groups, Column Segments, Zone Maps & Parquet Integration
File Layout Overview
A .duckdb file splits table data into row groups of roughly 122,880 rows each. Within every row group, each column is stored as a contiguous compressed segment. This columnar layout means analytical queries that touch only a few columns skip the rest entirely.
Compression Methods
DuckDB picks a compression codec per column segment based on the data's statistical profile. Select a method below to see how it encodes sample data.
Zone Map Pruning
Each column segment stores its minimum and maximum value. When a query includes a filter, DuckDB checks these zone maps and skips entire row groups that cannot contain matching rows. No separate index needed.
Checkpointing & WAL
DuckDB writes new data to a Write-Ahead Log first for crash safety. Periodically it checkpoints β flushing everything into the columnar .duckdb file. This keeps writes fast while maintaining the optimized columnar layout on disk.
Parquet Integration
DuckDB queries Parquet files as if they were native tables. It pushes projections and filters into the Parquet reader, reading only needed columns and row groups. This is why DuckDB became the go-to tool for querying data lake files.
Column Projection Pushdown
A query selecting 3 columns from a 50-column Parquet file reads only those 3 column chunks. The other 47 columns are never touched.
Row Group Filter Pushdown
Parquet files also store min/max statistics per row group. DuckDB uses these to skip row groups that fail the WHERE clause, identical to its native zone maps.
Parallel Row Group Reads
Each Parquet row group is assigned to a different thread via morsel-driven parallelism. On an 8-core machine, 8 row groups are processed simultaneously.
Row Groups as Parallelism Units
Row groups are the fundamental unit of parallel execution. Each row group can be processed by a different CPU core independently. No coordination or locking needed between threads scanning separate row groups.
Frequently Asked Questions
What is the .duckdb file format?
A .duckdb file is DuckDB's native persistent storage format. It organizes data into row groups (each holding roughly 120,000 rows by default). Within each row group, data is stored column-by-column in compressed segments. The file also contains metadata blocks, a write-ahead log pointer, and min/max zone maps for each column segment. This layout enables efficient analytical scans because queries can read only the columns and row groups they actually need.
How do zone maps (min/max indexes) work in DuckDB?
Every column segment in a row group stores the minimum and maximum value for that segment. When DuckDB encounters a filter like WHERE price > 500, it checks the zone map for each row group's price column. If the maximum value in a segment is 400, the entire row group is skipped without reading any actual data. This eliminates the need for separate B-tree or hash indexes for most analytical filter patterns.
What compression methods does DuckDB use?
DuckDB selects compression per column segment based on the data distribution. Methods include: constant encoding (all values identical), dictionary compression (few distinct values), bitpacking (small integers packed into fewer bits), run-length encoding (consecutive repeated values), frame-of-reference (values in a narrow range stored as offsets from a base), and general-purpose codecs like LZ4 and Zstd for data that doesn't match a specialized pattern.
Can DuckDB query Parquet files without importing them?
Yes. DuckDB reads Parquet files directly using its built-in Parquet reader. It pushes column projections and row group filters into the reader so it only loads the columns referenced in the query and skips row groups whose Parquet statistics fail the filter predicates. This makes DuckDB extremely popular for querying data lake files without any ETL step.
What is checkpointing in DuckDB?
DuckDB uses a Write-Ahead Log (WAL) for crash recovery. New writes go to the WAL first. Periodically, or when the WAL grows large enough, DuckDB performs a checkpoint: it flushes all in-memory and WAL data into the main .duckdb file in the columnar row-group format. After a successful checkpoint, the WAL is truncated. If DuckDB crashes before checkpointing, it replays the WAL on the next startup to recover committed transactions.