Why AI Chatbots "Hallucinate": The Statistical Roots of Language-Model Errors

An AI hallucination is a response from a chatbot or language model that reads as fluent and plausible but is factually incorrect — a fabricated court case, an invented statistic, a wrong date presented with total confidence. The term has moved from a niche machine-learning phrase into everyday vocabulary as more people use tools like ChatGPT for research, writing and legal work.

What OpenAI's research says causes it

In a paper titled "Why Language Models Hallucinate," published on OpenAI's website and on the arXiv preprint server in September 2025, researchers Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang argue that hallucinations are not a mysterious glitch but a predictable outcome of how models are built and graded, according to OpenAI's own summary of the paper.

During pretraining, a model learns from text that shows only correct examples of language use — it is never explicitly shown a labeled list of false statements to avoid. The researchers describe this as turning hallucination into a problem of binary classification: if a model cannot reliably tell a true statement from a false one in its training data, errors will arise through ordinary statistical pressure, independent of how good the engineering is, OpenAI reports.

Why models guess instead of saying "I don't know"

The paper's central claim concerns what happens after pretraining, during evaluation. According to OpenAI, most widely used accuracy benchmarks score a confident wrong answer the same as an admission of uncertainty — both just count as "wrong" — while a confident correct answer scores best. The researchers compare this to a student guessing on a multiple-choice exam: guessing has a chance of being right, while leaving a question blank guarantees zero credit, as reported by OpenAI.

OpenAI's proposed fix is not a separate hallucination test but a change to how existing benchmarks are scored: penalizing confident errors more heavily than expressions of uncertainty, so that models are no longer optimized to bluff their way through gaps in knowledge, OpenAI states.

How much do hallucination rates vary by model

OpenAI's researchers report one concrete illustration of this trade-off: an evaluation of a model called GPT-5-thinking-mini brought its error rate down to 26%, compared with 75% for the earlier o4-mini model, according to the paper as reported by InfoQ. The researchers note that the lower-error model achieved this partly by answering with the equivalent of "I don't know" in more than half of its responses, rather than guessing, InfoQ reported.

A real-world consequence: fabricated legal citations

The practical cost of hallucinations has surfaced repeatedly in U.S. courts. California's 2nd District Court of Appeal fined Los Angeles-area attorney Amir Mostafavi $10,000 in September 2025 after finding that 21 of 23 case quotations in an appellate brief he filed were fabricated by ChatGPT, according to reporting by CalMatters. The court wrote that no legal filing should contain citations the submitting attorney "has not personally read and verified," CalMatters reported.

Mostafavi, who filed the brief in July 2023 without checking the AI-generated content, told the court, "I hope this example will help others not fall into the hole. I'm paying the price," according to CalMatters. The outlet also reported that California's Judicial Council has required courts statewide to either ban generative AI or adopt a usage policy by December 15, 2025, and that legal-technology trackers have logged more than 600 documented instances of AI-fabricated citations in U.S. court filings.

Key points

• Hallucinations are fluent, confident statements from a language model that are factually wrong, not random glitches.
• OpenAI's September 2025 paper attributes them partly to pretraining limits — models cannot always distinguish true from false statements in their training data.
• The bigger driver, per OpenAI, is evaluation design: benchmarks that reward confident guesses and penalize admissions of uncertainty equally to wrong answers.
• In one OpenAI-reported example, a newer model (GPT-5-thinking-mini) cut its error rate to 26% from 75% for an earlier model (o4-mini), partly by saying 'I don't know' more often, but the company says the underlying incentive problem requires changing how benchmarks are scored industry-wide.
• Fabricated AI-generated legal citations have already led to court sanctions in the United States, including a $10,000 fine in California in 2025.