How Speech Recognition and Voice-to-Text AI Actually Works

Every time a phone transcribes a voicemail, a smart speaker answers a spoken question, or a captioning app displays live subtitles, the same basic pipeline runs in the background: sound is converted into a visual pattern, and a neural network reads that pattern to guess the words. This explainer breaks down how that pipeline works, which systems are in wide use, and where the technology still makes mistakes.

From sound wave to spectrogram

Speech-to-text systems do not read raw audio directly. According to OpenAI's technical paper for its Whisper model, posted on arXiv, audio is first converted into a form a neural network can process efficiently before any transcription happens. Developer documentation for Whisper, published on GitHub by OpenAI, describes the model as 'a general-purpose speech recognition model' that handles multilingual transcription, translation into English, spoken-language identification and voice activity detection — all through one model rather than several separate stages.

The common technique across modern systems is to turn the audio into a mel spectrogram: a chart that shows which frequencies are present at each moment in the recording, scaled to roughly match how the human ear perceives pitch. That image-like representation, rather than the raw waveform, is what gets fed into the neural network.

Two different approaches: cloud-scale and on-device

Not all speech recognizers are built the same way, and the differences matter for how and where they can run.

OpenAI's Whisper, described in the arXiv paper 'Robust Speech Recognition via Large-Scale Weak Supervision,' was trained on 680,000 hours of multilingual and multitask audio collected from the internet, using an encoder-decoder transformer architecture similar to the one behind text-generation models such as GPT. The encoder processes the spectrogram; the decoder then predicts text tokens one at a time. The paper's authors report that the resulting models 'generalize well to standard benchmarks' in zero-shot settings, meaning without being fine-tuned on those specific benchmarks, and that 'when compared to humans, the models approach their accuracy and robustness.'

A different design goal shaped Google's on-device recognizer for its Gboard keyboard, described in a March 12, 2019 post on the Google Research blog. Instead of a large model running on remote servers, Google built what it called an 'all-neural, on-device speech recognizer' using a technique called an RNN transducer (RNN-T). The system compresses down to about 80 megabytes — versus almost 2 gigabytes for the search graph used by Google's traditional production models — by applying parameter quantization, a technique that the post says delivered roughly a fourfold reduction compared with the trained floating-point version (about 450 megabytes before compression). Because the whole model fits on the phone, Google said it 'offers the same accuracy as the traditional server-based models' while working 'even when you are offline,' avoiding the delay or interruptions that come from sending audio to a remote server.

Why this matters for accessibility

Speech recognition only works well for speech patterns it has seen enough of in training. Google's AI-for-social-good team launched Project Euphonia in a May 7, 2019 blog post, saying standard models are built from large volumes of what the post calls 'typical' speech and therefore perform poorly for people whose speech is altered by conditions the post names as stroke, ALS, multiple sclerosis, traumatic brain injury and Parkinson's; Team Gleason, an ALS advocacy group that has worked with the project, separately lists cerebral palsy among the conditions that can make this technology harder to use. The project asks volunteers with atypical speech to record phrases so researchers can build more inclusive models. A 2022 follow-up post on the same blog announced the project's research was expanding to four more languages — French, Hindi, Japanese and Spanish — in addition to its original English-language work, and said the broader effort had grown into Project Relate, described in a separate November 2021 Google blog post as an Android app, launched at the time in closed beta, designed to learn a user's individual speech patterns and build a personalized recognition model.

Where the technology still fails: 'hallucinations'

Automatic transcription is not simply slow or occasionally wrong — researchers have documented cases where models invent content that was never spoken, a failure mode generally called a hallucination. Reporting by Fortune, published October 26, 2024, cited researchers Allison Koenecke of Cornell University and Mona Sloane of the University of Virginia, who found that of the hallucinations they identified in Whisper transcripts, nearly 40% contained content the researchers classified as harmful or concerning, including fabricated violent statements and invented racial commentary that was not present in the original audio. The same report said Whisper-based transcription from a company called Nabla was already in use by more than 30,000 clinicians across 40 health systems, with roughly 7 million medical visits transcribed by that point, even though OpenAI had cautioned against using Whisper in 'high-risk domains' where transcription errors could affect decisions.

Where people encounter this daily

• Phone keyboards: on-device dictation (e.g., Gboard's offline recognizer) converts speech to typed text without sending audio to a server.
• Smart speakers and voice assistants: cloud-based or hybrid models interpret spoken commands.
• Accessibility tools: apps built on personalized models, such as Project Relate, help transcribe speech for people with atypical speech patterns.
• Medical and professional transcription: Whisper-based commercial products transcribe consultations and meetings, a use case now facing scrutiny over hallucination rates.