Robustness
How well an AI system keeps working correctly when things change, when data is noisy, unusual inputs arrive, or someone tries to manipulate it.
The Simple Version
How well an AI system keeps working correctly when things change, when data is noisy, unusual inputs arrive, or someone tries to manipulate it.
Detailed Explanation
AI robustness encompasses distributional robustness (performance maintenance under input distribution shift), adversarial robustness (resistance to deliberately crafted inputs designed to cause failures), and reliability robustness (consistent performance across repeated uses). The EU AI Act requires high-risk AI systems to achieve appropriate levels of accuracy, robustness, and cybersecurity (Article 15). ISO/IEC 23894 includes robustness in its AI risk taxonomy. Robustness testing, including stress testing, distribution shift evaluation, and adversarial testing, is a core component of pre-deployment validation.
Key Characteristics
- Three dimensions: distributional, adversarial, and reliability robustness
- Required for high-risk systems under EU AI Act Article 15
- Assessed through stress testing, distribution shift evaluation, and adversarial testing
- Trade-offs exist between robustness and model accuracy
Why It Matters
AI systems deployed in safety-critical contexts must be robustness-tested against plausible real-world distribution shifts, including seasonal data patterns, population drift, and edge cases not well-represented in training data.
Real-World Analogy
Like testing a bridge not just under expected traffic loads but under storms, earthquakes, and corrosion scenarios, robustness testing ensures the system holds up across the range of conditions it may actually encounter.
Common Misconceptions
- High accuracy on a test set implies robustness, test sets may not represent distribution shift or adversarial conditions; separate robustness evaluation is required.
- Robustness and accuracy are equivalent, a highly accurate model may still fail catastrophically on out-of-distribution inputs.