Delta Lake
ACID transactions on object storage — the lakehouse foundation
Delta Lake is an open-source storage layer that brings ACID transactions to object storage (S3, GCS, Azure Blob, ADLS). It solves the core problem with data lakes: raw object storage is cheap and scalable but provides no guarantees — no atomicity, no consistency, no protection from partial writes. A Spark job that dies mid-write leaves the lake in an indeterminate state. Delta Lake wraps object storage with a transaction log that makes every mutation atomic and consistent, and every read snapshot-isolated.
The table format is stored as Parquet data files plus a _delta_log/
directory. The transaction log is the source of truth: every change is a
numbered commit, readers see a consistent snapshot up to a specific version,
and writers use optimistic concurrency to avoid conflicting commits. This
same design powers Apache Iceberg and Apache Hudi — Delta Lake pioneered it
for the Spark ecosystem.
Why Delta Lake?
The core guarantees that make a lake behave like a database:
ACID Transactions
Every write is atomic. Multiple files are committed as a single unit — either all appear, or none do. Readers never see partial writes. This works on S3 because Delta Lake uses preconditions and atomic file operations to implement serializable isolation.
Time Travel
Every version of the table is accessible. Query VERSION AS OF 42 or TIMESTAMP AS OF '2026-01-01'. Roll back accidental DELETEs, audit changes, replay old data for experiments. History is retained for 30 days by default.
Schema Enforcement
Delta Lake validates writes against the table schema before committing. A column type mismatch, a missing required field, an extra column — all rejected at write time, not at read time. No silent data corruption from schema drift.
Z-Order Clustering
Cosort data across multiple columns using the Z-order space-filling curve. Maximizes data skipping for queries that filter on multiple columns. Combined with per-file statistics (min/max/null count), this dramatically reduces the files that need to be scanned.
Unified Batch & Streaming
Same table for batch ETL and streaming inserts. Spark Structured Streaming writes to Delta with exactly-once semantics. Autoloader handles raw cloud object storage ingestion. Change Data Feed exposes row-level changes for downstream consumers.
Schema Evolution
Add columns, drop columns, rename columns, change column types — with explicit options for how to handle existing data. Merge schema (add new columns from new files automatically) vs strict enforcement vs safe evolution with backfill.
How It Works
The transaction log is the table.
# The _delta_log/ directory structure
s3://warehouse/nyc-taxi/
├── _delta_log/
│ ├── 00000000000000000000.json ← table creation
│ ├── 00000000000000000001.json ← first data write
│ ├── ... (one JSON file per commit)
│ └── 00000000000000000100.checkpoint.parquet ← snapshot at v100
├── year=2024/ ← partitioned data files
│ ├── month=01/
│ │ ├── part-00000-abc123.parquet
│ │ └── part-00001-def456.parquet
│ └── month=02/
└── year=2025/
# A JSON commit contains actions
# 00000000000000000003.json:
[
{ "add": {"path": "year=2025/month=02/part-00000-...", "size": ..., "stats": "..."} },
{ "remove": {"path": "year=2024/month=01/part-99999-...", "deletionTimestamp": ...} }
]
# Reading version 42:
# 1. Find latest checkpoint at or before v42
# 2. Load checkpoint state
# 3. Replay commits checkpoint+1 through 42
# Result: exact set of files active at version 42
# Writing (optimistic concurrency):
# 1. Read latest version N from _delta_log/
# 2. Produce new data files
# 3. Write _delta_log/N+1.json (atomic precondition: file must not exist)
# 4. If precondition fails → someone else committed first → retry
The key insight: cloud object stores are read-after-write consistent for new objects.
Once a JSON commit file is written, all subsequent readers see it. The zero-padded
version numbers ensure lexicographic order matches numerical order, so listing
_delta_log/ returns commits in version order. This gives total ordering
without a coordinator.
Key Numbers
Comparison: Delta Lake vs Iceberg vs Hudi
| Feature | Delta Lake | Apache Iceberg | Apache Hudi |
|---|---|---|---|
| Origin | Databricks (2019) | Netflix / Apple (2017) | Uber (2016) |
| Commit format | JSON in _delta_log/ | Avro manifests in metadata/ | Avro timeline in .hoodie/ |
| Checkpoint format | Parquet action log | Parquet manifest list | Parquet metadata table |
| Schema evolution | Explicit merge/strict/assert | Add, drop, rename, reorder | In-place rewrite or non-destructive |
| Partition transforms | Identity, bucket, truncate | Identity, bucket, truncate, year/month/day/hour | Identity, bucket, truncate, date |
| Time travel | VERSION, TIMESTAMP AS OF | Snapshot ID, timestamp | Incremental queries via timeline |
| Primary engines | Spark, Databricks, Trino, DuckDB | Flink, Spark, Trino, Snowflake, BigQuery | Spark, Flink, Trino |
| Isolation | Serializable | Snapshot isolation | Copy-on-write or merge-on-read |
The three formats are functionally similar — all provide ACID on object storage, schema enforcement, time travel, and partition transforms. Delta Lake wins on Spark/Databricks ecosystem maturity and human-readable JSON commits. Iceberg wins on engine neutrality and more expressive partition transforms. Hudi wins on incremental processing and merge-on-read performance for mutable workloads.
Articles
Frequently Asked Questions
Why is Delta Lake called a 'lakehouse'?
A traditional data lake is just object storage: cheap, scalable, but no transactions, no schema enforcement, no reliability. A data warehouse gives you ACID, schemas, and BI tooling, but ties you to a proprietary format and costs 10-100x more per TB. Delta Lake sits in the middle: it runs on top of object storage (S3, ADLS, GCS) using open formats (Parquet) so the data is portable, while providing ACID transactions, schema enforcement, time travel, and a rich surface area for SQL engines (Spark, Databricks, Trino, Snowflake, DuckDB) to read it. The 'lakehouse' is the vision: data lake economics with warehouse reliability.
How does Delta Lake compare to Apache Iceberg?
Both are open table formats that sit on object storage and provide ACID transactions. Delta Lake originated from Databricks and is heavily optimized for Spark; Iceberg originated from Netflix and is engine-agnostic by design. Iceberg uses a separate metadata layer (metadata/ directory) with Avro-based manifest files, while Delta Lake puts everything in _delta_log/ at the table root with human-readable JSON commits. Iceberg's partition evolution is more flexible; Delta Lake's DML syntax (UPDATE, DELETE, MERGE) is more mature in Spark. Both support time travel, schema evolution, and partition transforms. The ecosystem matters more than the technical differences at this point.
What's the difference between COPY INTO and streaming writes?
COPY INTO is for bulk ingestion with idempotent semantics — if a file is already loaded, it's skipped. It handles retries gracefully and is the recommended path for recurring ETL batches. Streaming writes (from Spark Structured Streaming, Kafka, Kinesis) use micro-batches or continuous mode to write incrementally, using transaction versioning to ensure exactly-once semantics. COPY INTO is a one-shot load; streaming is a long-running process. COPY INTO doesn't require a running Spark cluster after the load, while streaming jobs need to stay alive.
Can Delta Lake handle streaming data at scale?
Yes. Delta Live Tables (Databricks) or Spark Structured Streaming writing to Delta is a common pattern for streaming ingestion. The transaction log handles the micro-batches, and the checkpoint interval controls how often snapshots are taken. For very high-throughput streams (millions of events/sec), the recommendation is to partition by event time, use bucketing for heavy shuffles, and tune the checkpoint interval — more frequent checkpoints mean faster recovery but more log entries. Autoloader (cloudFiles) is the recommended ingestion path for raw S3/GCS/Azure Blob event streams.
When should I use Z-ORDER clustering?
Z-ORDER is a space-filling curve that co-locates similar values across multiple columns. Use it when you have queries that filter on multiple columns and the table is large enough that file skipping matters (hundreds of GB+). The cost: Z-ORDER rewrites all data files in a partition, which is expensive and cannot be combined with other operations in the same transaction. Run it after bulk loads, not continuously. For high-cardinality columns with range queries (timestamps, IDs), Z-ORDER often helps more than partition pruning alone. Only Z-ORDER columns that appear in your WHERE clauses.
What's the relationship between Delta Lake and Databricks?
Delta Lake was created by Databricks and initially open-sourced under the Apache 2.0 license. Databricks ships Delta Lake as the default table format in their platform. However, Delta Lake is a standalone open-source project (github.com/delta-io/delta) that runs on Spark, Flink, Trino, dbt, DuckDB, and more. The Databricks-specific additions — Delta Live Tables, live tables, Unity Catalog integration,Change Data Feed — are proprietary. The core Delta Lake protocol, the transaction log format, and the DML syntax are fully open and vendor-neutral.