Dataset bias
When a training dataset is lopsided in a way that causes an AI to produce unfair or inaccurate outputs for some groups or situations.
The Simple Version
When a training dataset is lopsided in a way that causes an AI to produce unfair or inaccurate outputs for some groups or situations.
Detailed Explanation
Dataset bias can arise from: historical bias (data reflecting past discriminatory patterns), representation bias (some groups collected more than others), measurement bias (data collection instruments that produce systematically different measurements for different groups), aggregation bias (grouping populations that should be modelled separately), and labelling bias (annotators applying inconsistent or discriminatory labels). Dataset bias is a root cause of algorithmic bias and is addressed in EU AI Act Article 10, which requires providers to examine training data for relevant biases and implement data governance measures to mitigate them.
Key Characteristics
- Multiple sources: historical, representational, measurement, aggregation, and labelling bias
- Persists through training into model outputs as algorithmic bias
- Required to be examined and documented under EU AI Act Article 10
- Can be partially addressed through re-sampling, re-weighting, and data augmentation
Why It Matters
AI ethics and compliance teams must build systematic bias auditing into data preparation processes, documenting identified biases and mitigation steps as part of Annex IV technical documentation.
Real-World Analogy
Like a flawed medical study that recruited only patients from private hospitals, the resulting treatment recommendations will be subtly biased towards the health patterns of wealthier patients.
Common Misconceptions
- Bias can be removed entirely from a dataset, bias can be reduced and managed but not eliminated entirely; residual bias must be documented and monitored.
- Bias only matters for demographic attributes, bias can affect model performance across any dimension where the training data distribution mismatches deployment conditions.