Data Pipeline Monitoring and Observability

Data pipelines break. Monitoring ensures you know when they break and can fix them quickly.

The Five Pillars of Data Observability

Pipeline Metrics

Freshness

-- Check when data last arrived
SELECT
    source,
    MAX(loaded_at) AS last_loaded,
    CURRENT_TIMESTAMP - MAX(loaded_at) AS staleness
FROM monitoring.pipeline_metrics
GROUP BY source

Alert when: Staleness exceeds SLA (e.g., 1 hour for real-time, 24 hours for daily).

Volume

-- Detect sudden volume changes
SELECT
    date,
    COUNT(*) AS row_count,
    COUNT(*) - LAG(COUNT(*)) OVER (ORDER BY date) AS day_over_day_change
FROM silver.orders
GROUP BY date

Alert when: Volume drops >50% or spikes >200% compared to trailing 7-day average.

Distribution

def check_distribution(df, column, method='zscore'):
    if method == 'zscore':
        zscores = (df[column] - df[column].mean()) / df[column].std()
        anomalies = df[abs(zscores) > 3]
        if len(anomalies) > 0:
            send_alert(f"Anomalous values in {column}: {len(anomalies)} rows")

Alert when: Statistical distribution shifts beyond thresholds.

Schema

def detect_schema_drift(schema_file, actual_schema):
    expected = parse_schema(schema_file)
    added = actual_schema.keys() - expected.keys()
    removed = expected.keys() - actual_schema.keys()
    changed = {k for k in expected if k in actual_schema
               and expected[k].type != actual_schema[k].type}
    if added or removed or changed:
        send_alert(f"Schema drift detected: +{added}, -{removed}, ~{changed}")

Monitoring Tools

Alerting

Alert Severity

Alert Fatigue

Too many alerts = ignored alerts.

• Alert on symptoms, not causes
• Use threshold + duration (e.g., “staleness > 1 hour for 5 minutes”)
• Day-one alerts only (auto-snooze repeated alerts)
• Route alerts to the right team

Data Lineage

Understand data flow:

stg_orders (source) → fact_orders (warehouse) → rpt_daily_revenue (report)
                           ↓
                     ML_feature_store (training)

Lineage helps you:

• Find impacted reports when a source changes
• Trace errors to root cause
• Identify owners of downstream dependencies

Lineage Tools

• Apache Atlas: Enterprise metadata + lineage
• dbt docs: Auto-generates lineage graph (dbt docs generate)
• Marquez: Open-source lineage
• DataHub: Metadata platform with lineage

Pipeline Health Dashboard

Build a dashboard with:

1. Pipeline status: Green (healthy), Yellow (degraded), Red (down)
2. Freshness: Last successful run time per pipeline
3. Volume: 7-day trend with anomaly bands
4. Error rate: Failed tasks / total tasks
5. Latency: Time from data arrival to availability
6. SLA compliance: Percentage of runs hitting SLAs

Monitoring by Pipeline Type

Batch Pipelines (Daily/Hourly)

pipeline: daily_orders_etl
steps:
  - extract: complete → 10:00 AM
  - load: complete → 10:15 AM
  - transform: complete → 10:30 AM
sla: 11:00 AM
alert_if_not_complete_by: 11:30 AM

Streaming Pipelines (Real-Time)

pipeline: clickstream_processor
metrics:
  - events_per_second: avg 1000
  - latency_p99: < 100ms
  - consumer_lag: < 10000
alert_if:
  - events_per_second < 100 (for 5 min)
  - consumer_lag > 100000

Common Failure Modes

| Mode | Detection | Fix | |

Data Pipeline Observability

Monitoring a data pipeline goes beyond tracking whether jobs succeeded or failed. True observability answers four questions: is data arriving on time, is it complete, is it correct, and are there anomalies?

Pipeline Metrics

Track these metrics for every pipeline:

• Freshness — How old is the data? A daily sales report should be available by 8 AM. Measure the delay between data generation and availability.
• Volume — Expected vs actual row counts. A sudden 50% drop may indicate a source system issue or a processing bug.
• Schema drift — Unexpected column additions, removals, or type changes. Schema drift breaks downstream consumers silently.
• Latency — How long each pipeline stage takes. Detect gradual performance degradation before it causes SLA breaches.
• Error rate — Percentage of failed records. A low but persistent error rate may indicate systemic data quality issues.

Alerting Strategies

Structure alerts in tiers:

Tier 1 (Pager) — Pipeline completely failed, no data available. Immediate attention required. Tier 2 (Email/Slack) — Data delayed past SLA, volume anomaly detected. Action within hours. Tier 3 (Dashboard) — Schema drift detected, data quality score declining. Review during business hours.

Data Quality Monitoring

Implement data quality checks at the end of each pipeline stage:

• Not null checks on critical columns
• Referential integrity checks (every order_id exists in the orders table)
• Uniqueness checks on primary key columns
• Range checks on numeric fields (negative prices are invalid)
• Freshness checks (data is less than X hours old)

Automate these checks and alert when they fail. Build a data quality scorecard that tracks trends over time and gives stakeholders confidence in the data they consume.

Tools for Pipeline Observability

Open-source: Great Expectations (data quality), Apache Airflow (orchestration + monitoring), Prometheus + Grafana (metrics and dashboards), OpenLineage (lineage tracking). Commercial: Monte Carlo, Sifflet, Bigeye.

FAQ

What is data engineering? Data engineering builds infrastructure and pipelines to collect, store, and process data at scale. It involves ETL/ELT processes, data warehousing, streaming systems, and ensuring data quality and accessibility for analytics and machine learning.

What is the difference between ETL and ELT? ETL (Extract-Transform-Load) transforms data before loading into the target system. ELT (Extract-Load-Transform) loads raw data first and transforms in-place. ELT leverages modern data warehouses like Snowflake and BigQuery for compute power.

What is a data pipeline? A data pipeline moves data from source systems to destinations, applying transformations along the way. Pipelines can be batch (scheduled intervals) or streaming (continuous processing). Tools include Airflow, dbt, and Spark.

Should I use a data lake or a data warehouse? Data lakes store raw data in native formats (cheap, flexible). Data warehouses store structured, curated data optimized for analytics (faster, more expensive). Modern architectures use both — lakehouse architecture combines their strengths.

What is the role of a data catalog? A data catalog indexes and documents available data assets — their schemas, lineage, quality metrics, and ownership. It helps data consumers discover, understand, and trust the data they work with.

Data Pipeline Observability

Monitoring a data pipeline goes beyond tracking whether jobs succeeded or failed. True observability answers four questions: is data arriving on time, is it complete, is it correct, and are there anomalies?

Pipeline Metrics

Track these metrics for every pipeline:

• Freshness — How old is the data? A daily sales report should be available by 8 AM. Measure the delay between data generation and availability.
• Volume — Expected vs actual row counts. A sudden 50% drop may indicate a source system issue or a processing bug.
• Schema drift — Unexpected column additions, removals, or type changes. Schema drift breaks downstream consumers silently.
• Latency — How long each pipeline stage takes. Detect gradual performance degradation before it causes SLA breaches.
• Error rate — Percentage of failed records. A low but persistent error rate may indicate systemic data quality issues.

Alerting Strategies

Structure alerts in tiers:

Tier 1 (Pager) — Pipeline completely failed, no data available. Immediate attention required. Tier 2 (Email/Slack) — Data delayed past SLA, volume anomaly detected. Action within hours. Tier 3 (Dashboard) — Schema drift detected, data quality score declining. Review during business hours.

Data Quality Monitoring

Implement data quality checks at the end of each pipeline stage:

• Not null checks on critical columns
• Referential integrity checks (every order_id exists in the orders table)
• Uniqueness checks on primary key columns
• Range checks on numeric fields (negative prices are invalid)
• Freshness checks (data is less than X hours old)

Automate these checks and alert when they fail. Build a data quality scorecard that tracks trends over time and gives stakeholders confidence in the data they consume.

Tools for Pipeline Observability

Open-source: Great Expectations (data quality), Apache Airflow (orchestration + monitoring), Prometheus + Grafana (metrics and dashboards), OpenLineage (lineage tracking). Commercial: Monte Carlo, Sifflet, Bigeye.

—|—|—| | Source system down | Freshness alert | Retry with backoff | | Schema change | Schema drift alert | Update pipeline | | Data quality issue | Distribution alert | Backfill + fix | | Resource exhaustion | CPU/memory alert | Scale up workers | | Upstream pipeline delayed | Freshness cascade | Notify downstream | | Data corruption | Volume + distribution | Restore from backup |

Runbook Essentials

For every critical pipeline, document:

1. What does this pipeline do?
2. What does failure look like? (metrics, alerts)
3. Who is on call?
4. How do you triage? (first 5 steps)
5. How do you fix common failures?
6. Who to escalate to?

Proactive Monitoring

The goal is to catch issues before they affect users:

• Run tests on sample data before full processing
• Compare each run to trailing averages
• Track trends (gradual degradation is easy to miss)
• Simulate failures (chaos engineering for data)

Good monitoring makes data pipelines reliable. Alerts should be actionable, not noisy.

Data Quality Guide — dbt Guide — Data Orchestration Guide