A machine learning model's overall accuracy — the percentage of predictions it gets right across its entire test dataset — is one of the most commonly reported measures of how well it performs. But a single aggregate accuracy number can conceal a real, meaningful problem: a model can achieve a high overall accuracy score while performing considerably worse for one specific subgroup of people than it does for everyone else, with that subgroup's poorer performance simply averaged out and hidden within the impressive-looking overall figure. Fairness in AI treats this kind of subgroup performance gap as a genuine, measurable problem in its own right, distinct from a model's overall accuracy.
Averaging can hide a gap that matters enormously to the people inside it
If a subgroup makes up a relatively small share of a model's overall training and test data, that subgroup's own error rate can be quite poor while barely moving the model's overall reported accuracy figure at all, since the much larger remaining population's generally better performance dominates the overall average. This means a model that looks excellent by its single headline accuracy number can, in practice, be performing meaningfully worse for a specific group of real people — potentially defined by characteristics like race, gender or age — a disparity the aggregate number alone gives no visibility into whatsoever, and one that only becomes apparent once a model's performance is specifically broken down and reported separately for each relevant subgroup, rather than reported only as one single combined figure.
Fairness metrics measure the gap directly, but no single definition covers every case
Researchers have developed several distinct formal fairness metrics specifically designed to measure and expose this kind of subgroup disparity directly, rather than relying on overall accuracy alone to reveal it — different metrics ask subtly different questions, such as whether a model's error rate is equal across groups, or whether its positive predictions are equally reliable across groups, and a model can sometimes satisfy one of these fairness definitions while still failing another, since the different metrics aren't always simultaneously achievable together for a single model. This is precisely why AI ethics and fairness research doesn't treat "make it accurate" and "make it fair" as automatically the same goal achieved by the same single technique — a model can be highly accurate overall, by the most commonly reported measure, while still leaving a real, measurable fairness problem for a specific group of people entirely unaddressed.
What we're still unsure about
That aggregate accuracy can conceal meaningful subgroup performance gaps, and that different formal fairness metrics can conflict with each other, are both well-established, mathematically demonstrated findings in machine learning research. What remains genuinely unresolved, and is actively debated among researchers, ethicists and policymakers, is which specific fairness definition, or which combination of them, should actually be prioritised for a given real-world application, since the different metrics can genuinely trade off against each other and against overall accuracy in ways that don't have one single mathematically correct answer — the choice ultimately involves real value judgements about a specific application's own context and stakes, not a technical question fairness research alone can settle without that further judgement.
This sits inside Ethics & Fairness in AI, one of eight topics in AI/ML, one of seven domains in Computer Science, one of seventeen subjects the app can quiz you on.