How AI Algorithms Develop Bias, and What Safeguards Exist to Stop It

Artificial intelligence systems are increasingly used to screen job applicants, score loan applications, moderate online content, and flag suspects for police. When these systems produce systematically unequal outcomes for particular groups, researchers and regulators call the result algorithmic bias. The question of how that bias forms, and what can be done about it, has moved from academic papers into corporate boardrooms and legislatures.

Where the bias comes from

A 2022 report from the U.S. National Institute of Standards and Technology (NIST), titled Special Publication 1270, concluded that AI bias is not only a data problem. The report's authors identify three distinct sources of bias in AI systems.

• Computational and statistical bias — arising from training data or algorithms that are not representative of the population the system will affect.
• Human bias — introduced when designers, labelers, or users apply incomplete information or stereotypes while building or operating the system.
• Systemic bias — existing institutional practices and historical inequalities that get encoded into a system simply because they are present in the records it learns from.

NIST principal investigator Reva Schwartz said organizations often default to narrow technical fixes that fail to capture bias originating from the broader social context in which AI is built and deployed, according to the agency's own summary of the report.

Two documented cases

Amazon built an experimental résumé-screening tool starting in 2014, training it on roughly a decade of résumés the company had received. Because the technology sector has historically employed far more men than women, the majority of those training résumés came from male applicants. According to reporting by MIT Technology Review, the system taught itself to penalize résumés that contained the word "women's" and downgraded graduates of two all-women's colleges. Amazon engineers tried to correct the pattern, but by 2017 the company had lost confidence the tool was gender-neutral in other respects and stopped using it.

A separate 2018 study called Gender Shades, conducted by Joy Buolamwini of the MIT Media Lab and Timnit Gebru, then at Microsoft Research, tested three commercial facial-analysis systems on a dataset of more than 1,200 images classified by skin tone using the Fitzpatrick scale. According to MIT News' summary of the findings, error rates for lighter-skinned men never exceeded 0.8%, while error rates for darker-skinned women ranged from 20.8% to 34.7%, reaching as high as 46.8% for the darkest-skinned women in one system.

What safeguards exist

The European Union's AI Act, formally Regulation (EU) 2024/1689, entered into force on August 1, 2024, and is described by the European Commission as the first comprehensive legal framework for AI worldwide. For systems classified as high-risk — including those used in employment, education, and biometric identification — the regulation requires providers to use training, validation, and testing data of sufficient quality to minimize the risk of discriminatory outcomes, and to put risk-assessment and mitigation measures in place before the system reaches the market, per the Commission's own description of the rules. The Commission states that obligations for high-risk systems used in sensitive areas such as employment and biometrics become applicable on December 2, 2027, with rules for AI embedded in regulated products following on August 2, 2028.

In the United States, NIST's own response to its bias findings was to recommend a "socio-technical" approach: addressing human and systemic bias alongside computational bias by involving experts outside engineering and consulting affected communities, rather than relying on technical fixes alone, according to the agency's report summary.

Why this matters now

Hiring platforms, credit-scoring tools, and content-moderation systems are now common across many industries, which means the kinds of errors documented in the Amazon and Gender Shades cases can affect large numbers of people before anyone notices a pattern. The EU AI Act's phased deadlines through 2028, and NIST's ongoing bias-management guidance, indicate that oversight of these systems is still being built out rather than already complete.