Data Lakes: Architecture, Governance, and Best Practices

A data lake stores vast amounts of raw data in its native format. Unlike data warehouses, data lakes do not require schema-on-write — data is stored as-is and structured only when read. According to Gartner, organizations that implement a well-governed data lake see a 20-30% reduction in data management costs compared to traditional siloed approaches. However, without proper governance, a data lake quickly becomes a data swamp: an unmanageable collection of poorly documented, low-quality data with no reliable way to find, trust, or use the data it contains.

Data Lake vs Data Warehouse

The choice between a data lake and a data warehouse depends on the use case. Data lakes accept any data format — JSON, CSV, Parquet, Avro, images, video — making them suitable for data science exploration and machine learning. Data warehouses enforce a schema before loading, which ensures consistency and query performance for business intelligence. The Fundamentals of Data Engineering book by Joe Reis and Matt Housley describes data lakes as “schema-on-read” systems and data warehouses as “schema-on-write” systems, capturing the fundamental architectural difference.

The lakehouse architecture bridges the gap by adding ACID transactions and warehouse-quality query performance over object storage. Apache Iceberg, Delta Lake, and Apache Hudi are the leading open table formats that enable this pattern, providing features like snapshot isolation, time travel, and schema evolution that were previously only available in enterprise data warehouses.

Medallion Architecture

The bronze-silver-gold layering pattern is the industry standard for organizing data lake content, originally popularized by Databricks documentation. Each layer has a distinct purpose, audience, and retention policy.

Bronze Layer

The bronze layer holds raw data exactly as received from source systems. This data is immutable — never modified after ingestion. Bronze preserves the original data for reprocessing, auditing, and debugging. A key best practice is to store the ingestion metadata alongside the data, including source system, ingestion timestamp, pipeline version, and any validation flags.

s3://data-lake/bronze/orders/2024/06/01/orders_20240601_001.parquet

Bronze should retain data for 30-90 days depending on regulatory requirements and reprocessing needs. Partition by ingestion date to enable efficient pruning and lifecycle management. Use compression codecs like Snappy or Zstd to reduce storage costs without significantly impacting read performance.

Silver Layer

The silver layer contains cleaned, validated, and deduplicated data. Silver applies schema enforcement, type casting, and business rules. This is where data quality checks run and where data engineers resolve inconsistencies from source systems. Common operations include standardizing date formats, normalizing text fields, removing duplicate records, and validating foreign key relationships.

INSERT INTO silver.orders
SELECT
    order_id,
    customer_id,
    SAFE_CAST(amount AS DECIMAL(10,2)) AS amount,
    COALESCE(currency, 'USD') AS currency,
    PARSE_TIMESTAMP('%Y-%m-%d', order_date) AS order_date
FROM bronze.orders
WHERE order_id IS NOT NULL

Silver data is suitable for data scientists who need clean data for exploration and feature engineering. Retain silver data for 90-365 days. Partitioning strategies may differ from bronze — for example, partitioning by month rather than by day if query patterns are less granular.

Gold Layer

The gold layer contains business-level aggregates and data marts. These are denormalized, pre-joined, and pre-aggregated tables designed for BI tools and dashboards. Gold data has a stable schema with well-defined naming conventions and is consumed by analysts and business stakeholders who need query performance without writing complex joins.

CREATE TABLE gold.daily_revenue AS
SELECT
    order_date,
    SUM(amount) AS total_revenue,
    COUNT(DISTINCT customer_id) AS active_customers
FROM silver.orders
GROUP BY order_date

Gold data should be retained for 365+ days. Because gold tables are aggregated, they are compact and cost-effective to store long-term. Gold tables should be documented with business definitions for each column to ensure consistent interpretation across the organization.

File Formats for Data Lakes

The choice of file format significantly impacts query performance and storage costs. Parquet is the default choice for analytical workloads because of its columnar layout and excellent compression characteristics. The Apache Parquet documentation notes that Parquet files include schema metadata and statistics at the row group level, enabling predicate pushdown and efficient skipping of irrelevant data.

Columnar formats like Parquet store data by column rather than by row, which means queries that select only a few columns read only those columns from disk. Parquet also supports predicate pushdown — filters on column values are applied at the storage layer before data reaches the query engine, dramatically reducing I/O for selective queries.

Partitioning Strategies

Proper partitioning is the most impactful performance optimization for data lakes. The goal is to balance partition count with partition size. Too many small partitions cause metadata overhead and slow listing operations; too few large partitions force full scans of unnecessary data.

s3://data-lake/orders/year=2024/month=06/day=01/

Date-based partitioning aligns with most analytical query patterns. Common secondary partition keys include region, category, and customer segment. The key insight is to choose partition columns that match filter conditions in your most frequent queries. The Designing Data-Intensive Applications book by Martin Kleppmann emphasizes that partitioning strategies should be chosen based on access patterns rather than storage convenience.

Cataloging and Governance

Without a catalog, a data lake is a data swamp. A catalog provides a central registry of datasets, their schemas, and their locations. It enables data discovery, access control, and schema evolution tracking. Modern catalogs support REST APIs for integration with query engines and provide fine-grained access control through RBAC and attribute-based policies.

A catalog also enables schema evolution tracking. With open table formats, you can add, drop, or rename columns without rewriting existing data files. The catalog tracks schema versions and ensures query engines use the correct schema for each snapshot, preventing compatibility issues when schemas change over time.

Compaction and Maintenance

Over time, data lakes accumulate many small files, especially from streaming ingestion. Small files degrade query performance because the query engine spends more time listing and opening files than reading actual data. Compaction jobs rewrite small files into larger ones, typically targeting 128-512 MB per file after compression. In Iceberg and Delta Lake, compaction is a built-in operation that rewrites data files in the background without affecting concurrent reads through snapshot isolation.

def compact_table(spark, table_path):
    df = spark.read.parquet(table_path)
    df.coalesce(num_partitions).write.mode("overwrite").parquet(table_path)

Beyond compaction, regular maintenance includes removing orphan files, updating table statistics, and expiring old snapshots. Iceberg’s expire_snapshots procedure removes metadata and data files that are no longer referenced by any active snapshot, freeing storage space.

Lifecycle Management

Automated lifecycle policies reduce storage costs by moving data to cheaper tiers as it ages. A common policy moves bronze data from S3 Standard ($0.023/GB) to S3 Glacier Instant Retrieval ($0.004/GB) after 30 days, and to S3 Glacier Deep Archive ($0.001/GB) after 90 days.

Frequently Asked Questions

What is the difference between a data lake and a data swamp? A data lake has governance — a catalog, schema management, access controls, and data quality checks. A data swamp lacks these and becomes an unmanageable collection of data with no documentation, inconsistent formats, and unknown quality. The difference is not technical but operational; the same storage system can be either depending on how it is managed. Implementing a catalog and enforcing data quality checks from day one prevents swamp formation.

Should I migrate my data warehouse to a data lake? Many organizations adopt a lakehouse architecture that keeps the warehouse for structured BI workloads while adding a data lake for raw data, ML, and data science. Full migration from warehouse to lake is rarely necessary or beneficial. Consider a lakehouse approach using Iceberg or Delta Lake: you keep existing warehouse capabilities while adding data lake flexibility for new use cases.

What is the best compression codec for Parquet files? Snappy offers the best balance of compression speed and decompression speed for query performance, making it the default choice for most workloads. Zstd provides better compression ratios (15-25% smaller than Snappy) with slightly slower performance. Gzip achieves the highest compression but the slowest decompression, making it suitable for archival rather than active querying. The Apache Parquet documentation recommends Snappy for general use and Zstd when storage costs are a primary concern.

How do I handle schema changes in a data lake? Open table formats like Apache Iceberg support schema evolution natively. You can add columns, rename columns, and change column types without rewriting existing data. The catalog tracks schema versions, and query engines automatically resolve the correct schema based on the snapshot being read. Backward-compatible changes (adding nullable columns) are safe for all consumers; breaking changes require coordination with downstream consumers.

Why is my data lake query slow? The most common causes are too many small files (high listing overhead), lack of partitioning (full scans), and choosing a row-oriented format like JSON or CSV for analytical queries. Run compaction to merge small files, verify that partition columns match frequent query filters, and convert to columnar formats like Parquet or ORC. Using an open table format with built-in statistics can also improve query planning efficiency.

Apache Spark Guide — dbt Guide — Modern Data Stack Guide