Algorithmic bias
When an AI system consistently produces unfair results for certain groups of people, for example, being less accurate or less favourable for women, ethnic minorities, or older adults.
The Simple Version
When an AI system consistently produces unfair results for certain groups of people, for example, being less accurate or less favourable for women, ethnic minorities, or older adults.
Detailed Explanation
Algorithmic bias manifests in multiple forms: disparate treatment (the algorithm treats groups differently based on protected characteristics), disparate impact (a facially neutral algorithm produces disproportionate harm to protected groups), and feedback loop bias (biased outputs influence future data collection, amplifying the original bias). Root causes include dataset bias, proxy variable use, objective function misalignment, and feedback loops. The EU AI Act requires providers of high-risk systems to implement bias detection and mitigation measures. ISO/IEC 23894 provides guidance on AI risk management including bias risk.
Key Characteristics
- Can be caused by data, model design, or feedback loops
- Manifests as disparate treatment or disparate impact across demographic groups
- Legally significant under EU AI Act, GDPR, and anti-discrimination law
- Requires measurement and mitigation across the full model lifecycle
Why It Matters
Organisations deploying AI in hiring, lending, healthcare, or law enforcement face significant legal and reputational exposure from undetected algorithmic bias, bias audits and ongoing monitoring are becoming standard practice.
Real-World Analogy
Like a hiring manager who uses 'cultural fit' as a criterion that consistently selects applicants from similar backgrounds, the criterion sounds neutral but produces discriminatory outcomes.
Common Misconceptions
- Removing protected attributes from the model eliminates bias, protected attributes can be proxied through correlated variables, producing the same discriminatory outcomes.
- Bias is always intentional, most algorithmic bias arises unintentionally from data and design choices made without adequate diversity analysis.