How AI Image Generation Works: Diffusion Models Explained

Several AI image generators that emerged from 2022 onward — including OpenAI's DALL-E 2, Midjourney and Stability AI's Stable Diffusion — are built on a technique called diffusion. Rather than drawing an image stroke by stroke, these systems learn to reverse a mathematical process that gradually destroys an image with noise, then run that process backward starting from random static.

The core idea: learning to undo noise

Diffusion models for image generation were formalized in a 2020 paper, 'Denoising Diffusion Probabilistic Models,' by Jonathan Ho, Ajay Jain and Pieter Abbeel, posted on the research repository arXiv. The method, described as inspired by non-equilibrium thermodynamics, works in two stages. In the 'forward' stage, used only during training, small amounts of random (Gaussian) noise are added to a real photo over many steps until the image becomes indistinguishable from static. In the 'reverse' stage, a neural network is trained to predict and subtract that noise, step by step, until the original image is recovered.

Once trained, the network no longer needs a real photo to start from. It can begin with a canvas of pure random noise and run the same denoising steps it learned during training, producing a brand-new image that did not exist before. The 2020 paper reported that on a dataset called CIFAR-10 its model reached strong image-quality scores and, on larger 256x256 images, matched the sample quality of ProgressiveGAN, an earlier generative technique based on a different architecture.

Where the neural network does the work: U-Net and text guidance

The network that performs the denoising is typically a U-Net, a type of architecture that shrinks an image down to capture overall structure and then rebuilds it back up to full resolution. In Stable Diffusion's own technical documentation on GitHub, the model is described as using an 860-million-parameter U-Net to run the diffusion process, together with a separate autoencoder that compresses images into a smaller mathematical representation and decompresses the result back into a full image.

To follow a written prompt, the system needs a way to connect words to pixels. According to the same Stable Diffusion documentation, the model uses a frozen CLIP ViT-L/14 text encoder — a component trained separately to match text and images — to turn a prompt into a numerical representation that guides each denoising step, so the network removes noise in a way that steers the emerging image toward what the words describe.

• Training: real images are progressively corrupted with random noise over many steps until only static remains.
• Learning: a U-Net neural network is trained to predict and remove the noise added at each step.
• Text conditioning: a separate text encoder (such as CLIP) turns a prompt into a numerical guide for the denoising network.
• Generation: starting from random noise, the trained network removes noise step by step, guided by the text, until an image appears.
• Decoding (latent models only): a compressed internal representation is converted back into a full-resolution image.

Pixel space versus latent space: why Stable Diffusion runs on a laptop

Not all diffusion models work the same way internally. According to a December 2022 MIT Technology Review account of the technology's rise, DALL-E 2 performs its noise-removal process directly on full-resolution image pixels, while Stable Diffusion instead works on a compressed, 'latent' representation of the image inside the neural network — an approach the outlet says let Stable Diffusion run on ordinary personal computers rather than requiring large server infrastructure.

The GitHub repository for Stable Diffusion, maintained by the CompVis research group with Stability AI and Runway, states the model's latent diffusion approach was built on the group's earlier paper, 'High-Resolution Image Synthesis with Latent Diffusion Models,' presented at the CVPR 2022 computer-vision conference, and was trained on images from the LAION-5B dataset.

How the major diffusion-based image generators compare

MIT Technology Review reported that DALL-E 2 reached one million users within about two and a half months of its April 2022 launch, and that Stable Diffusion's DreamStudio service passed one million users in under half that time after its August 2022 release — figures the outlet cited as evidence of how quickly the diffusion approach reached a mass audience that year.

What the 2020–2022 research established

The underlying mechanism — noise added during training, noise removed during generation, guided by a text encoder — formed the basis of the major image generators that emerged in 2022, as described in the 2020 paper by Ho, Jain and Abbeel and the 2022 reporting and documentation reviewed here. Individual companies have since layered additional components, such as larger U-Nets, bigger text encoders or prompt-rewriting systems, on top of this same diffusion framework.