Data catalog
A searchable directory of all the data an organisation has, describing what each dataset contains, where it lives, how good it is, and who owns it.
The Simple Version
A searchable directory of all the data an organisation has, describing what each dataset contains, where it lives, how good it is, and who owns it.
Detailed Explanation
A data catalog aggregates metadata from source systems, data warehouses, data lakes, and external datasets to create a unified, searchable view of available data assets. Modern catalogs include active metadata (usage statistics, quality scores, lineage), social features (reviews, ratings, questions), and governance workflows (certification, access requests). For AI teams, a catalog dramatically reduces the time spent finding and evaluating suitable training data. AI-aware catalogs may include specific fields for annotation method, intended ML use, and associated model cards.
Key Characteristics
- Provides unified, searchable inventory across disparate data sources
- Includes metadata, lineage, quality, and governance information
- Modern catalogs incorporate active metadata from automated profiling
- AI-aware catalogs support model card linking and ML-specific metadata
Why It Matters
Data catalogs reduce the 'data archaeology' time that AI teams spend locating and evaluating datasets, shortening time-to-model while improving data quality through better-informed data selection.
Real-World Analogy
Like a library catalogue, rather than searching through every shelf to find a book on a topic, the catalogue tells you what is available, where it is, and whether it is in good condition.
Common Misconceptions
- A data catalog is the same as a data warehouse, a warehouse stores data; a catalog describes and indexes it.
- Implementing a catalog tool solves metadata management, the tool requires ongoing curation and governance processes to remain useful.