Data Catalog and Discovery Guide

A data catalog is a searchable inventory of all your data assets. Without one, data is invisible — users don’t know what exists or where to find it.

What is a Data Catalog?

A centralized metadata repository that answers:

• What data exists?
• Where is it stored?
• What does it mean? (definitions, business context)
• Who owns it?
• How fresh is it?
• Who has used it?

Why You Need One

Metadata Types

Technical Metadata

Automatically extracted from data systems:

• Schema: Column names, types, nullability
• Statistics: Row count, distinct values, null percentages
• Lineage: Upstream sources, downstream dependencies
• Partitioning: How data is organized in storage
• Freshness: Last updated timestamp

Business Metadata

Human-curated context:

• Description: What this data means
• Definitions: Clear explanations of columns
• Tags: PII, golden_record, experimental
• Ownership: Team and individual responsible
• Certification: “This is an approved source” badge

Operational Metadata

Runtime information:

• Pipeline runs: When was this last refreshed
• Quality checks: Passing/failing tests
• SLAs: Expected freshness windows
• Access logs: Who has been querying this

Data Catalog Features

Search

Lineage

Shows how data flows through the system:

Source DB → Ingestion pipeline → Bronze table → Silver table → Gold table → BI dashboard
    ↑            ↑                     ↑               ↑              ↑
  PostgreSQL   Airflow DAG          Delta Lake      dbt model      Looker Studio

Why it matters:

• Impact analysis: “Which dashboards break if I change this column?”
• Root cause: “Why is this dashboard number wrong?” → trace back to source
• Compliance: “Where does this PII data come from?”

Data Quality Badges

Popular Data Catalog Tools

Recommendation: Start with DataHub (most features, active community) or dbt Docs (if you already use dbt).

Implementing a Catalog

Phase 1: Automated Ingestion (Week 1-2)

Connect to your data sources and automatically ingest metadata:

# Example: Extracting metadata with DataHub
from datahub.ingestion.run import Pipeline

pipeline = Pipeline.create({
    "source": {
        "type": "snowflake",
        "config": {
            "host": "account.snowflake.com",
            "database": "PROD",
            "username": "service_account",
            "password": "***"
        }
    },
    "sink": {
        "type": "datahub-rest",
        "config": {"server": "http://datahub.company.com:8080"}
    }
---)
pipeline.run()

Phase 2: Business Metadata (Week 3-4)

Add descriptions, owners, and tags:

| Task | Who | |

Data Catalog Components

A data catalog serves as the central inventory of an organization’s data assets. Modern data catalogs include:

Business glossary — Business definitions of data terms. A “customer” might be defined differently across teams; the catalog documents the canonical definition and relationship to physical data assets.

Technical metadata — Schema definitions, data types, partition schemes, file formats, and storage locations. This metadata enables data consumers to understand how to access and interpret the data.

Data lineage — The complete journey of data from source to destination. Lineage shows which transformations were applied, which jobs produced the data, and which downstream artifacts depend on it. This is critical for impact analysis — when a source system changes, lineage tells you which reports and models are affected.

Data quality metrics — Completeness, accuracy, timeliness, uniqueness, and consistency scores. Data quality badges help consumers trust the data they use. Poor quality data should be flagged and quarantined.

Active vs Passive Cataloging

Passive cataloging crawls data sources and automatically extracts metadata — schema, row counts, update timestamps. Active cataloging goes further by tracking usage patterns, query history, and data popularity. Active catalogs identify the most valuable data assets, unused data that can be retired, and optimization opportunities for frequently queried datasets.

Data Catalog Tools

Open-source options include Apache Atlas, Amundsen, and DataHub. Commercial options include Alation, Collibra, and Atlan. When choosing a catalog, evaluate integration with your data stack, automated metadata ingestion capabilities, search quality, data quality integration, and collaboration features (comments, ratings, ownership).

Implementing a Data Catalog

Start with a pilot focused on the most critical data assets and most frequent data consumers. Define metadata standards before ingestion — consistent naming, tagging conventions, and documentation templates. Assign data owners responsible for maintaining catalog entries. Establish a governance process for updates and additions. Treat the catalog as a living system that requires ongoing maintenance, not a one-time documentation project.

FAQ

What is data engineering? Data engineering builds infrastructure and pipelines to collect, store, and process data at scale. It involves ETL/ELT processes, data warehousing, streaming systems, and ensuring data quality and accessibility for analytics and machine learning.

What is the difference between ETL and ELT? ETL (Extract-Transform-Load) transforms data before loading into the target system. ELT (Extract-Load-Transform) loads raw data first and transforms in-place. ELT leverages modern data warehouses like Snowflake and BigQuery for compute power.

What is a data pipeline? A data pipeline moves data from source systems to destinations, applying transformations along the way. Pipelines can be batch (scheduled intervals) or streaming (continuous processing). Tools include Airflow, dbt, and Spark.

Should I use a data lake or a data warehouse? Data lakes store raw data in native formats (cheap, flexible). Data warehouses store structured, curated data optimized for analytics (faster, more expensive). Modern architectures use both — lakehouse architecture combines their strengths.

What is the role of a data catalog? A data catalog indexes and documents available data assets — their schemas, lineage, quality metrics, and ownership. It helps data consumers discover, understand, and trust the data they work with.

Data Catalog Components

A data catalog serves as the central inventory of an organization’s data assets. Modern data catalogs include:

Business glossary — Business definitions of data terms. A “customer” might be defined differently across teams; the catalog documents the canonical definition and relationship to physical data assets.

Technical metadata — Schema definitions, data types, partition schemes, file formats, and storage locations. This metadata enables data consumers to understand how to access and interpret the data.

Data lineage — The complete journey of data from source to destination. Lineage shows which transformations were applied, which jobs produced the data, and which downstream artifacts depend on it. This is critical for impact analysis — when a source system changes, lineage tells you which reports and models are affected.

Data quality metrics — Completeness, accuracy, timeliness, uniqueness, and consistency scores. Data quality badges help consumers trust the data they use. Poor quality data should be flagged and quarantined.

Active vs Passive Cataloging

Passive cataloging crawls data sources and automatically extracts metadata — schema, row counts, update timestamps. Active cataloging goes further by tracking usage patterns, query history, and data popularity. Active catalogs identify the most valuable data assets, unused data that can be retired, and optimization opportunities for frequently queried datasets.

Data Catalog Tools

Open-source options include Apache Atlas, Amundsen, and DataHub. Commercial options include Alation, Collibra, and Atlan. When choosing a catalog, evaluate integration with your data stack, automated metadata ingestion capabilities, search quality, data quality integration, and collaboration features (comments, ratings, ownership).

Implementing a Data Catalog

Start with a pilot focused on the most critical data assets and most frequent data consumers. Define metadata standards before ingestion — consistent naming, tagging conventions, and documentation templates. Assign data owners responsible for maintaining catalog entries. Establish a governance process for updates and additions. Treat the catalog as a living system that requires ongoing maintenance, not a one-time documentation project.

—|—| | Write table descriptions | Data engineers | | Add column-level descriptions | Data engineers + analysts | | Tag PII columns | Security/compliance | | Assign owners | Engineering managers | | Mark certified datasets | Data lead |

Phase 3: Adoption (Ongoing)

• Make it a habit: Require catalog check before creating new tables
• Integrate with workflow: Catalog link in Slack, email, dashboard
• Measure adoption: Monthly active searchers, number of documented assets

Data Discovery Culture

The “Check the Catalog” Rule

Before creating a new dataset, check the catalog:

1. Does this data already exist?
2. Can I extend an existing table instead?
3. Is there an approved source I should use?

Catalog Champions

Assign a metadata owner for each data domain:

Anti-Patterns

A good data catalog turns data from invisible to accessible.

Data Governance Guide — Data Quality Guide — Modern Data Stack Guide