Memory Management
How an In-Process Database Shares RAM Without Hogging It
DuckDB runs inside your process. It shares the same address space as your Python script, your web server, or your data pipeline. Unlike a standalone database server that owns all available RAM, DuckDB needs to be a good memory citizen — using what it needs, releasing what it does not, and gracefully handling situations where the data outgrows the available memory.
The key mechanism is the buffer manager: a pool of fixed-size memory blocks that tracks every allocation, enforces configurable limits, and spills to temporary disk files when pressure gets too high. Combined with DuckDB's streaming execution model — where most operators only need enough memory for a single vector of 2048 rows — this architecture lets DuckDB process datasets far larger than RAM on a single machine.
The Embedded Memory Challenge
A server database like PostgreSQL owns its memory. DuckDB borrows it.
Shared Address Space
DuckDB runs as a library inside your application. If it allocates 12GB on a 16GB machine, your application gets the remaining 4GB minus OS overhead. There is no isolation boundary — both compete for the same physical pages.
Unpredictable Workloads
A user might run a simple SELECT that touches 100 rows, then follow it with a 10-way join over 500 million rows. The memory system must handle both extremes without manual tuning between queries.
No OOM Crashes
If DuckDB is embedded in a production web server, an out-of-memory crash takes down the entire service. The memory manager must degrade gracefully — slowing down rather than terminating.
Buffer Manager Architecture
All DuckDB memory flows through the buffer manager. It allocates blocks, tracks usage, and decides what stays in RAM and what gets evicted.
Memory Pool Visualization
Watch how blocks are allocated, pinned, and evicted as a query executes. When the pool fills up, unpinned blocks spill to disk so the query can continue.
Block Lifecycle
Every block in the buffer pool transitions through a defined set of states.
Spill-to-Disk Mechanism
When a query needs more memory than the configured limit, DuckDB does not fail. It writes intermediate results to temporary files and continues.
Sorting 100 million rows with a 2GB memory limit. The sort operator fills its buffer, writes sorted runs to disk, then merges them in a final pass. This is external merge sort — the same algorithm that databases have used for decades, integrated into DuckDB's pipeline-based execution.
Building a hash table on a 50-million-row table. When the hash table exceeds the memory limit, DuckDB partitions both sides of the join, spills partitions to disk, and processes them one at a time. Each partition fits in memory, so the join completes without requiring the full hash table to be resident.
GROUP BY over a high-cardinality column (millions of groups). The hash aggregate operator tracks its memory usage. When it approaches the limit, it partitions the groups, spills some partitions, and processes remaining ones in memory. Spilled partitions are loaded back one at a time in a second pass.
Streaming Execution and Memory
Most operators in DuckDB never accumulate data. They process one vector at a time, keeping memory usage constant regardless of table size.
Pipeline Breakers: Where Memory Accumulates
Allocation Strategy
DuckDB uses a purpose-built allocator that balances performance, predictability, and fragmentation avoidance.
| Aspect | DuckDB Approach | Typical Server DB |
|---|---|---|
| Block pool | Pre-allocated fixed-size blocks (256KB). Avoids external fragmentation. | Shared buffer pool with 8KB pages (PostgreSQL) or larger blocks (MySQL InnoDB 16KB). |
| Large allocations | Oversized requests fall through to the system allocator but are tracked against the memory limit. | Typically use the shared buffer pool or dedicated memory arenas. |
| Thread safety | Each thread has a local allocation cache. Contention is minimized because DuckDB's morsel-driven model gives each thread independent work units. | Centralized buffer pool with latches or lock-free structures. |
| Temp files | Automatically created in a configurable temp directory. Cleaned up when the connection closes or the query finishes. | Explicit tablespace configuration for temp tables. Requires DBA setup. |
Frequently Asked Questions
What happens when DuckDB exceeds the memory limit?
DuckDB does not crash or throw an out-of-memory error immediately. When the buffer manager reaches the configured memory limit, it evicts unpinned blocks from the buffer pool and writes them to temporary files on disk. Operators that need more memory than the limit — like large sorts or hash joins — transparently spill their intermediate data to these temp files and continue processing. The query completes, just slower than if everything fit in RAM.
How do I set a memory limit in DuckDB?
Use SET memory_limit='4GB'; (or any value) at the start of your session. You can also pass it as a configuration option when opening a connection. If you do not set a limit, DuckDB defaults to 80% of available system RAM. For embedded use cases where DuckDB shares memory with your application, setting an explicit limit prevents it from starving other components of memory.
Does DuckDB use memory-mapped files?
DuckDB uses a combination of standard heap allocation and its own buffer pool. It does not rely on mmap for general query processing the way some databases do. The buffer pool manages fixed-size blocks (typically 256KB) and handles eviction and spill-to-disk internally. This gives DuckDB predictable memory behavior and avoids the unpredictable page-fault patterns that mmap-based systems can suffer from under memory pressure.
How does streaming execution reduce memory usage?
DuckDB processes data in pipelines. A scan-filter-project pipeline only materializes one vector of 2048 rows at a time in each thread. The rows flow through the entire operator chain before the next batch is fetched. This means a simple filter query over a billion-row table might only need a few kilobytes of working memory. Only pipeline-breaking operators like sorts, hash join builds, and aggregations need to accumulate data, and those are the operators that spill to disk when necessary.
Can I monitor DuckDB's memory usage at runtime?
Yes. You can query the pragma_database_size() function to see current database size, and use CALL duckdb_temporary_files() to inspect spill files. The buffer manager tracks total allocated memory internally. In Python, you can also use standard process-level tools like psutil to monitor the RSS of the process DuckDB is embedded in, giving you a full picture of both DuckDB and application memory usage.