Data preparation
All the work done to clean and organise raw data before it is used to train an AI, removing errors, standardising formats, and making sure it is in a usable shape.
The Simple Version
All the work done to clean and organise raw data before it is used to train an AI, removing errors, standardising formats, and making sure it is in a usable shape.
Detailed Explanation
Data preparation is the most time-intensive phase of AI development, often accounting for 60–80% of project effort. It encompasses: data cleaning (removing duplicates, fixing errors, handling missing values), transformation (normalisation, encoding, aggregation), integration (combining data from multiple sources), reduction (sampling, dimensionality reduction), and augmentation (generating synthetic samples to address gaps). EU AI Act Annex IV requires documentation of data preparation steps for high-risk systems, including justification of choices that affect the training set composition. ISO/IEC 5259 Part 4 addresses data preparation within a process framework for AI data quality.
Key Characteristics
- Typically accounts for the majority of AI project time and effort
- Covers cleaning, transformation, integration, reduction, and augmentation
- Preparation choices materially affect model performance and bias
- Must be documented for EU AI Act Annex IV compliance
Why It Matters
Investing in reusable data preparation pipelines with version control and automated quality checks reduces the cost of iterative model development and ensures auditability of training data provenance.
Real-World Analogy
Like a chef preparing ingredients before cooking, washing vegetables, measuring portions, chopping and marinating, the quality of the final dish depends as much on preparation as on cooking technique.
Common Misconceptions
- Data preparation is a technical task with no governance implications, preparation choices (which records to exclude, how to handle missing values) embed assumptions that affect fairness and accuracy.
- More data preparation is always better, over-engineering preparation can introduce biases through inappropriate imputation or normalisation choices.