Skip to main content

Data lineage

A record that shows where data came from, what happened to it along the way, and where it ended up, like a passport stamp history for data.

The Simple Version

A record that shows where data came from, what happened to it along the way, and where it ended up, like a passport stamp history for data.

Detailed Explanation

Data lineage captures both technical lineage (system-to-system data flows, ETL transformations, pipeline operations) and business lineage (which business processes generate, consume, and transform data). In AI systems, lineage documentation allows teams to trace a model's prediction back to the training data that influenced it, supporting explainability, bias investigations, and regulatory audits. W3C PROV-O is a standard ontology for representing provenance and lineage relationships. The EU AI Act's Annex IV requires lineage-equivalent documentation for training and validation datasets.

Key Characteristics

  • Captures both technical and business dimensions of data flow
  • Enables root-cause analysis when models produce unexpected outputs
  • Supports regulatory auditability and data quality investigations
  • Can be represented using standards such as W3C PROV-O

Why It Matters

Data platform teams investing in lineage tooling (e.g. Apache Atlas, OpenMetadata) create audit-ready documentation that dramatically reduces the effort of AI Act Annex IV compliance.

Real-World Analogy

Like the chain of custody documentation in forensics, every person who handled evidence, every transformation it underwent, and every location it was stored is recorded to ensure its integrity can be verified.

Common Misconceptions

  • Data lineage is only for compliance purposes, it is equally valuable for debugging data quality issues, impact analysis, and understanding model behaviour.
  • Lineage documentation must be manual, modern data engineering tools can capture lineage automatically as part of pipeline execution.

Related Terms

Related Articles

Sources & Further Reading