Data Lakehouse Architecture Guide

Data Engineering Data Engineering 8 min read 1618 words Beginner ExcellentWiki Editorial Team

The data lakehouse combines the flexibility of a data lake with the reliability of a data warehouse.

What is a Lakehouse?

Traditional architecture splits data:

Data Lake (raw files) --ETL--> Data Warehouse (structured tables)

Lakehouse merges these:

Data Lakehouse (ACID, schema, files on object storage)

Key difference: Lakehouse stores raw and structured data in the same system, with ACID transactions on raw file formats.

Why Lakehouse?

Capability	Data Lake	Data Warehouse	Lakehouse
Raw data storage	Yes	Limited	Yes
Structured tables	No (manual)	Yes	Yes
ACID transactions	No	Yes	Yes
Schema enforcement	No	Yes	Yes
BI tool support	No	Yes	Yes
ML/AI data access	Yes	Limited	Yes
Cost	Low (object storage)	High (compute + storage)	Low (object storage)
Open formats	Yes	No (proprietary)	Yes

Core Technologies

Delta Lake (Databricks)

Open-source storage layer on top of Parquet.

# Writing with Delta Lake
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .getOrCreate()

# Write with ACID
df.write \
    .format("delta") \
    .mode("overwrite") \
    .save("/data/lakehouse/orders")

# Read
df = spark.read.format("delta").load("/data/lakehouse/orders")

Apache Iceberg

Open table format for large analytic datasets.

-- Create Iceberg table
CREATE TABLE orders (
    order_id BIGINT,
    customer_id STRING,
    order_date DATE,
    amount DECIMAL(10,2)
)
USING iceberg
PARTITIONED BY (months(order_date));

-- Time travel query
SELECT * FROM orders FOR SYSTEM_VERSION AS OF 1234567890;
-- Or
SELECT * FROM orders FOR SYSTEM_TIME AS OF '2026-01-01 00:00:00';

Apache Hudi

Lakehouse technology focused on incremental processing.

ACID Transactions on Data Lakes

Traditional data lakes don’t support ACID. Lakehouse formats add:

Write: A writer starts a transaction
  → Reads current table state
  → Creates new data files
  → Commits (atomic metadata update)

Read: A reader opens the table
  → Reads latest committed snapshot
  → Reads corresponding files
  → Never sees partial writes

Transaction Operations

Operation	Description
Snapshot isolation	Reader sees consistent state at point-in-time
Atomic commit	Write succeeds completely or not at all
Concurrent writes	Multiple writers don’t corrupt data
Time travel	Query historical states
Rollback	Revert to previous state

Delta Lake Features

Schema Evolution

Add columns without rewriting data:

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "/data/orders")

# Add new column (no rewrite needed)
delta_table.alterTableAddColumns(
    "customer_segment STRING"
)

Schema Enforcement

Reject writes that don’t match the schema:

# This will fail: schema mismatch
df_with_wrong_schema.write \
    .format("delta") \
    .mode("append") \
    .save("/data/orders")

Change Data Feed

Track changes (inserts, updates, deletes):

# Read changes since version 100
df = spark.read.format("delta") \
    .option("readChangeFeed", "true") \
    .option("startingVersion", "100") \
    .load("/data/orders")

Lakehouse Architecture

Bronze-Silver-Gold Pattern

Bronze (raw)         → Silver (cleaned)    → Gold (aggregated)
│                       │                       │
│ Raw data from source  │ Deduplicated          │ Business-level
│ No transformations    │ Cleansed              │ aggregates
│ Schema may change     │ Joined                │ Ready for BI
│ Append only           │ Enriched              │
│                       │                       │
└─── Bronze layer ──────┴─── Silver layer ──────┴─── Gold layer ────

Bronze Layer

-- Bronze: Raw data ingestion
CREATE TABLE bronze_orders
USING delta
AS
SELECT *, current_timestamp() AS _ingested_at
FROM raw_orders_source

Silver Layer

-- Silver: Cleaned and deduplicated
CREATE TABLE silver_orders
USING delta
AS
SELECT DISTINCT
    order_id,
    customer_id,
    CAST(order_date AS DATE) AS order_date,
    amount / 100.0 AS amount_dollars,
    status,
    _ingested_at
FROM bronze_orders
WHERE order_id IS NOT NULL
  AND amount > 0

Gold Layer

-- Gold: Business aggregates
CREATE TABLE gold_daily_sales
USING delta
AS
SELECT
    order_date,
    COUNT(DISTINCT order_id) AS order_count,
    COUNT(DISTINCT customer_id) AS customer_count,
    SUM(amount_dollars) AS total_revenue
FROM silver_orders
GROUP BY order_date

Optimization

Compaction

Small files hurt performance. Compact them:

# Delta Lake optimize
spark.sql("OPTIMIZE bronze_orders")

Z-Ordering

Co-locate related data within files:

spark.sql("OPTIMIZE silver_orders ZORDER BY (customer_id, order_date)")

Vacuum

Remove old files that are no longer referenced:

# Remove files older than 7 days
spark.sql("VACUUM bronze_orders RETAIN 168 HOURS")

Lakehouse vs. Warehouse

What Is a Lakehouse?

A data lakehouse combines the flexibility of a data lake with the reliability and performance of a data warehouse. It stores data in open formats (Parquet, ORC, Avro) on object storage (S3, ADLS, GCS) and adds ACID transactions, schema enforcement, and performance optimization through a metadata layer.

Lakehouse Architectures

Apache Iceberg — Open table format that adds ACID transactions, time travel, schema evolution, and partition evolution to data lakes. Supports most query engines (Spark, Trino, Flink, Hive).

Delta Lake — Databricks’ open-source storage layer. Features ACID transactions, scalable metadata handling, and unified batch/streaming. Tightly integrated with Spark.

Apache Hudi — Open-source lakehouse framework focusing on incremental processing, upserts, and delete support. Popular for streaming ingestion workflows.

Lakehouse Benefits Over Traditional Architectures

FAQ

What is data engineering? Data engineering builds infrastructure and pipelines to collect, store, and process data at scale. It involves ETL/ELT processes, data warehousing, streaming systems, and ensuring data quality and accessibility for analytics and machine learning.

What is the difference between ETL and ELT? ETL (Extract-Transform-Load) transforms data before loading into the target system. ELT (Extract-Load-Transform) loads raw data first and transforms in-place. ELT leverages modern data warehouses like Snowflake and BigQuery for compute power.

What is a data pipeline? A data pipeline moves data from source systems to destinations, applying transformations along the way. Pipelines can be batch (scheduled intervals) or streaming (continuous processing). Tools include Airflow, dbt, and Spark.

Should I use a data lake or a data warehouse? Data lakes store raw data in native formats (cheap, flexible). Data warehouses store structured, curated data optimized for analytics (faster, more expensive). Modern architectures use both — lakehouse architecture combines their strengths.

What is the role of a data catalog? A data catalog indexes and documents available data assets — their schemas, lineage, quality metrics, and ownership. It helps data consumers discover, understand, and trust the data they work with.

What Is a Lakehouse?

Lakehouse Architectures

Apache Iceberg — Open table format that adds ACID transactions, time travel, schema evolution, and partition evolution to data lakes. Supports most query engines (Spark, Trino, Flink, Hive).

Delta Lake — Databricks’ open-source storage layer. Features ACID transactions, scalable metadata handling, and unified batch/streaming. Tightly integrated with Spark.

Apache Hudi — Open-source lakehouse framework focusing on incremental processing, upserts, and delete support. Popular for streaming ingestion workflows.

Lakehouse Benefits Over Traditional Architectures

Feature	Data Lake	Data Warehouse	Lakehouse
Data format	Raw/Semi-structured	Structured	Open (Parquet/ORC)
Storage cost	Low	High	Low
Query performance	Slow (raw files)	Fast	Fast (optimized)
ACID transactions	No	Yes	Yes
Schema enforcement	None	Strict	Flexible
ML/AI support	Direct	Limited	Direct

Time Travel and Versioning

Lakehouse table formats support time travel — querying data as it existed at a specific point in time:

-- Delta Lake time travel
SELECT * FROM orders VERSION AS OF 12345;
SELECT * FROM orders TIMESTAMP AS OF '2024-01-15 10:00:00';

-- Iceberg time travel
SELECT * FROM orders FOR SYSTEM_TIME AS OF '2024-01-15 10:00:00';

Time travel enables reproducing historical reports, auditing data changes, and recovering from accidental data modifications without restoring from backups.

Lakehouse Migration Strategy

Migrating from a data warehouse to a lakehouse should start with non-critical workloads. Pick a single business domain, set up the lakehouse table format, dual-write data to both systems, validate query results match, and gradually migrate consumers. The most common migration path is warehouse → lakehouse + warehouse (hybrid) → lakehouse for most workloads.

Time Travel and Versioning

Lakehouse table formats support time travel — querying data as it existed at a specific point in time:

-- Delta Lake time travel
SELECT * FROM orders VERSION AS OF 12345;
SELECT * FROM orders TIMESTAMP AS OF '2024-01-15 10:00:00';

-- Iceberg time travel
SELECT * FROM orders FOR SYSTEM_TIME AS OF '2024-01-15 10:00:00';

Time travel enables reproducing historical reports, auditing data changes, and recovering from accidental data modifications without restoring from backups.

Lakehouse Migration Strategy

Monitoring a Lakehouse

Metric	What It Tells
Table size (total files)	Storage growth
File count (small files)	Need compaction
Commit history	Write frequency
Version age	Cleanup needed
Query latency	Performance

Anti-Patterns

Anti-Pattern	Problem	Fix
Too many small files	Slow queries	Compact regularly
No partitioning	Full table scans	Partition by date
Over-partitioning	Too many directories	Use coarser partitions
No vacuum	Runaway storage costs	Set retention policy
Writing from 100+ streams	Transaction conflicts	Batch or reduce writers