Apache Iceberg: Building Modern Data Lakehouse Architecture

Apache Iceberg: Building a Modern Data Lakehouse

Data lakes promised cheap, scalable storage for all your data. The reality was a “data swamp” — no schema enforcement, no ACID transactions, and queries that scanned entire directories. Apache Iceberg fixes this by adding a table format layer on top of data lake storage that brings database-like features: schema evolution, time travel, partition evolution, and ACID transactions — all on files stored in S3 or HDFS. Therefore, this guide covers how Iceberg works and how to build a production data lakehouse.

Why Data Lakes Need a Table Format

Traditional Hive-partitioned data lakes organize data by directory structure: s3://data/events/year=2026/month=03/day=09/. The query engine must list directories and files to figure out what to read. This breaks down at scale: listing millions of files in S3 is slow, schema changes require rewriting all data, and there’s no way to read a consistent snapshot while new data is being written.

Iceberg introduces a metadata layer that tracks exactly which files belong to each table, their schemas, partition information, and column-level statistics. Instead of listing directories, the query engine reads a compact metadata file that tells it exactly which data files to read. Moreover, each write creates a new snapshot — readers always see a consistent view of the table, even during concurrent writes. This is the same isolation guarantee you’d expect from a traditional database, but operating on files in object storage.

The table format is engine-agnostic: Spark, Trino, Flink, Dremio, Snowflake, and BigQuery can all read and write Iceberg tables. This engine independence means you’re not locked into a single query engine — use Spark for batch processing, Trino for interactive queries, and Flink for streaming, all operating on the same tables.

Apache Iceberg data lakehouse architecture and table format — Iceberg adds database-like features to data lakes — schema evolution, time travel, and ACID

The Three-Tier Metadata Architecture

To use Iceberg well, it helps to understand how the metadata is layered. The format is deliberately hierarchical, and each layer exists to avoid the expensive directory listings that plague Hive. At the top sits the catalog, which holds a single pointer per table to the current metadata file. That pointer swap is the atomic commit primitive — flipping it is how a write becomes visible to every reader at once.

Below the catalog is the metadata file (a JSON document) that records the table schema, partition specs, snapshot history, and a reference to the current manifest list. Each manifest list points to one or more manifest files, and those manifests finally enumerate the actual Parquet, ORC, or Avro data files along with per-file statistics. Because statistics like row counts, null counts, and min/max values live in the manifests, the engine can prune entire files without opening them.

This layering is what makes Iceberg fast. A query that filters on event_type = 'purchase' reads the manifest, checks the min/max bounds, and skips any data file that can’t contain matching rows. In practice, benchmarks published by the project show that for highly selective queries this file-level pruning eliminates the vast majority of scanned bytes compared with a naive directory scan. The trade-off is that every commit writes new metadata files, so write-heavy tables need the maintenance routines described later to keep the metadata tree compact.

Schema Evolution Without Rewriting Data

One of Iceberg’s most valuable features is safe schema evolution. You can add columns, rename columns, reorder columns, widen types (int to long), and make required columns optional — all without rewriting existing data files. The metadata layer tracks schema versions, and the query engine maps old data files to the current schema automatically.

The mechanism behind this is column IDs. Unlike Hive, which resolves columns by name or by physical position, Iceberg assigns every column a stable integer ID at creation time. A rename only changes the human-readable name in metadata; the underlying ID never moves, so old data files remain correctly addressable. As a result, renames are metadata-only operations, and a renamed column never silently maps to the wrong data.

-- Create an Iceberg table (Spark SQL)
CREATE TABLE lakehouse.events (
    event_id STRING,
    user_id STRING,
    event_type STRING,
    timestamp TIMESTAMP,
    properties MAP<STRING, STRING>
)
USING iceberg
PARTITIONED BY (days(timestamp), event_type)
TBLPROPERTIES (
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'zstd'
);

-- Schema evolution — no data rewrite needed
ALTER TABLE lakehouse.events ADD COLUMNS (
    session_id STRING,
    device_type STRING
);

ALTER TABLE lakehouse.events RENAME COLUMN properties TO metadata;

-- Old data files still work — missing columns return NULL
SELECT event_id, session_id  -- session_id is NULL for old rows
FROM lakehouse.events
WHERE timestamp > '2026-01-01';

In contrast, Hive tables require you to either rewrite all existing data files with the new schema (expensive and slow for terabyte-scale tables) or maintain complex schema-on-read logic in every query. Additionally, Iceberg enforces schema validation on writes — you can’t accidentally write data with the wrong column types, a common source of data quality issues in traditional data lakes.

Time Travel and Snapshot Isolation

Every write to an Iceberg table creates a new snapshot. You can query any historical snapshot by timestamp or snapshot ID — essential for debugging data issues, reproducing ML training datasets, and auditing changes. Concurrent readers always see a consistent snapshot, even while writers are adding new data.

-- Query data as it existed at a specific time
SELECT * FROM lakehouse.events
TIMESTAMP AS OF '2026-03-01 00:00:00';

-- Query a specific snapshot
SELECT * FROM lakehouse.events
VERSION AS OF 5847293048;

-- View snapshot history
SELECT * FROM lakehouse.events.snapshots;

-- View file-level changes between snapshots
SELECT * FROM lakehouse.events.changes
WHERE snapshot_id BETWEEN 5847293048 AND 5847293055;

-- Rollback to a previous snapshot (undo bad writes)
CALL lakehouse.system.rollback_to_snapshot(
    'events', 5847293048
);

-- Cherry-pick: apply changes from a specific snapshot
CALL lakehouse.system.cherrypick_snapshot(
    'events', 5847293055
);

Time travel is also powerful for regulatory compliance. If an auditor asks “what did the customer data look like on January 15th?”, you query that exact timestamp without maintaining separate audit tables. Furthermore, ML teams use time travel to create reproducible training datasets — freeze a snapshot ID in your training config, and you can always reproduce the exact same dataset months later.

The concurrency model underneath is optimistic concurrency control. When two writers commit at nearly the same time, each attempts the atomic pointer swap in the catalog. The first wins; the second detects that the base metadata it read is now stale, then retries its commit against the new state. For append-only workloads conflicts are rare, but for merge-heavy tables you should expect occasional retries and configure sensible retry limits to avoid pathological contention.

Data lakehouse time travel and snapshot management — Time travel enables auditing, debugging, and reproducible ML datasets from historical snapshots

Partition Evolution: Change Partitioning Without Rewriting

Partition evolution is uniquely powerful here. Traditional Hive tables encode partition values in directory paths — changing the partitioning scheme requires rewriting all data. Iceberg decouples partitioning from the physical file layout, allowing you to change partition strategies without touching existing data.

-- Start with daily partitioning
CREATE TABLE lakehouse.logs (
    log_id STRING,
    service STRING,
    level STRING,
    message STRING,
    timestamp TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(timestamp));

-- Data grows — switch to hourly partitioning for recent data
ALTER TABLE lakehouse.logs
ADD PARTITION FIELD hours(timestamp);

-- Drop the old daily partitioning
ALTER TABLE lakehouse.logs
DROP PARTITION FIELD days(timestamp);

-- New writes use hourly partitions
-- Old data (daily partitions) is still readable
-- Both partition schemes coexist in the metadata

This is transformative for growing datasets. You might start with monthly partitions when your table has 100GB, switch to daily at 10TB, and hourly at 100TB — all without downtime or data rewriting. The query engine uses metadata to apply the correct partition filter regardless of which scheme was active when the data was written.

Equally important is hidden partitioning. In Hive, a consumer querying WHERE timestamp BETWEEN ... must also know to add a redundant WHERE day = ... predicate or the engine scans everything. Iceberg derives partition values from the source column through partition transforms like days() or bucket(), so the engine prunes partitions automatically from the natural filter. Consequently, analysts write simpler SQL and stop accidentally triggering full-table scans.

Production Integration with Spark and Trino

A practical lakehouse uses multiple engines: Spark for heavy ETL and batch processing, Trino for fast interactive queries and dashboards, and Flink for real-time streaming ingestion. All three work with the same Iceberg tables through a shared catalog.

# PySpark — batch ETL with Iceberg
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.catalog.lakehouse",
            "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.lakehouse.type", "rest") \
    .config("spark.sql.catalog.lakehouse.uri",
            "http://iceberg-rest-catalog:8181") \
    .config("spark.sql.catalog.lakehouse.warehouse",
            "s3://my-lakehouse/warehouse") \
    .getOrCreate()

# Read Iceberg table
events = spark.table("lakehouse.events")

# Incremental read — only new snapshots since last run
new_events = spark.read.format("iceberg") \
    .option("start-snapshot-id", last_processed_snapshot) \
    .table("lakehouse.events")

# Merge (upsert) — ACID transaction
spark.sql("""
MERGE INTO lakehouse.customers target
USING updates source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
""")

# Maintenance — compact small files
spark.sql("""
CALL lakehouse.system.rewrite_data_files(
    table => 'events',
    options => map('target-file-size-bytes', '134217728')
)
""")

# Expire old snapshots (free storage)
spark.sql("""
CALL lakehouse.system.expire_snapshots(
    table => 'events',
    older_than => TIMESTAMP '2026-02-01 00:00:00',
    retain_last => 10
)
""")

Table maintenance is essential for production Iceberg deployments. Small files accumulate from streaming writes and need periodic compaction. Old snapshots consume storage and should be expired after your retention period. Orphan files from failed writes need cleanup. Consequently, schedule these maintenance operations as regular batch jobs — weekly compaction and daily snapshot expiration work well for most workloads.

Choosing a catalog implementation matters more than newcomers expect. The REST catalog shown above has become the recommended default because it cleanly decouples engines from the storage backend and centralizes commit coordination. Older options like the Hive Metastore catalog or the file-system “hadoop” catalog still work, but the hadoop catalog in particular relies on atomic renames that object stores like S3 do not natively guarantee, which can cause subtle commit issues under concurrency.

Data lakehouse production monitoring and maintenance — Regular maintenance — file compaction, snapshot expiration, and orphan cleanup — keeps performance optimal

When Iceberg Is the Wrong Choice

Iceberg is not a free win for every workload, and choosing it reflexively leads to disappointment. For low-latency point lookups and high-throughput transactional updates, an operational database such as PostgreSQL still wins decisively — Iceberg is optimized for large analytical scans, not single-row OLTP access patterns. If your “table” is a few gigabytes that one analyst queries occasionally, the metadata machinery is overhead you don’t need.

There is also a real operational cost. Someone has to own the maintenance jobs, monitor metadata growth, and tune file sizes; tables left unmaintained accumulate millions of tiny files and slow to a crawl. Streaming ingestion that commits every few seconds is especially prone to this. Competing table formats matter too: Delta Lake has tighter Databricks integration, while Apache Hudi historically led on fast upserts and record-level indexing, so the right answer depends on your existing engines and access patterns. For deeper context on storage and query trade-offs, see the SQL Query Optimization Guide.

Related Reading:

Resources:

In conclusion, Apache Iceberg transforms data lakes into reliable data lakehouses with database-grade features. Schema evolution, time travel, and partition evolution eliminate the most painful aspects of data lake management. Start with a single table, use your existing Spark or Trino infrastructure, schedule the maintenance jobs from day one, and expand as you see the operational benefits over traditional Hive-partitioned tables.