How AI Text Detectors Work, Explained Through a Prize-Winning Novel Dispute

A detection tool's finding that a prize-winning novel was almost entirely machine-written has put AI text detectors under scrutiny. The case raises a practical question for writers, students and editors alike: what are these tools actually measuring, and how much can their verdicts be trusted?

What triggered the debate

The Peruvian author David Silva Rojas won the inaugural Almadía Ventosa-Arrufat International Short Novel Prize, worth €20,000, for his novel "Donde el agua trabaja," a story set on a wine estate in Peru's Ica region. A unanimous jury that included writers Cristina Rivera Garza, Selva Almada and Mónica Ojeda selected the work from 2,204 submissions across 22 countries, according to Infobae.

According to Infobae, which cited an analysis by the Spanish newspaper El País, running the roughly 26,000-word novel through the Pangram 4.1 detection tool attributed 99% of it to artificial intelligence. The same tool flagged 97% of interview answers Silva Rojas gave in September and 80% of a 924-word autobiographical text he submitted, while the interviewer's own words registered as fully human. Contest rules required entrants to declare their manuscripts were "not written, neither partially nor totally, with artificial intelligence tools," on penalty of disqualification.

Publisher Almadía suspended the novel's presentation events in Madrid, Barcelona, Seville and Cádiz, which had been set to begin on October 23, 2026, according to Proceso. The publisher said Silva Rojas had signed a contract declaring he had not used such tools and asked for time to investigate. Silva Rojas declined to comment when contacted by Infobae for his account.

How AI text detectors actually work

According to GPTZero's own published methodology, its tool scores text on three main signals: perplexity, or how predictable the word choices are to a language model; burstiness, meaning how much sentence length and style vary across a passage; and overall tone, checking whether writing reads as generic or repetitive. GPTZero says it layers a proprietary model with "hundreds of factors" on top of these base measures.

Pangram, the tool used in the novel case, works differently. Per Wikipedia's entry on the tool and Pangram's own blog, it is trained by pairing large volumes of human writing with AI-generated text matched by topic, length and tone, then uses a "mixture of experts" architecture to classify individual sections of a document before combining them into a passage-level score. Pangram requires at least 50 words of prose to produce a result and reports separate false-positive rates by writing domain — for example, close to 0% on medical papers versus higher rates on poetry and recipes.

Crucially, none of these tools claim to identify an author. They output a statistical estimate — a probability that a given passage resembles machine-generated text more than human text the model has seen in training.

Why no detector offers certainty

OpenAI shut down its own AI text classifier in July 2023, citing a low rate of accuracy, according to TechCrunch. In TechCrunch's informal test of seven AI-generated passages, OpenAI's tool correctly identified only one, while GPTZero identified five. OpenAI said at the time it was researching "more effective provenance techniques" instead.

Pangram's own figures show the same tension between confidence and error. Wikipedia's entry on the tool notes that independent evaluations found it missed about 10.1% of passages written by AI systems imitating a specific human author's style, and that it performed worse on text run through "humanizer" tools meant to disguise AI output. The same entry lists past disputes in which Pangram results were challenged, including novelist Mia Ballard, whose writing was 78% flagged as AI, and criticism from Wall Street Journal editor James Taranto, who called the tool a "defamation machine" after it flagged three staff articles — though two of those articles' authors said they had used AI only to help revise their drafts.

What the novel's case does and does not show

The Pangram 4.1 score is a strong signal that prompted Almadía to pause the novel's rollout and open an investigation, but a detection score alone does not constitute proof of how a text was produced. As of the publisher's statement reported by Proceso, Almadía had not announced a final finding, and Silva Rojas had not given his account on the record. The contest's rule against AI use — and the signed declaration tied to it — is what exposes the author to potential disqualification if an investigation confirms the suspicion, separate from what any single detector score can establish on its own.