How Neural Networks Learn: The Math Behind ChatGPT and Gemini

Every time a chatbot answers a question or a photo app recognises a face, a neural network is running a learning process that follows the same basic steps, regardless of the task. According to IBM, that process rests on two linked techniques: backpropagation, which calculates how much each internal parameter contributed to an error, and gradient descent, the optimisation method that uses those calculations to correct the parameters.

What a Neural Network Actually Is

Stanford's Institute for Human-Centered AI (HAI) describes a neural network as a computational model inspired by the structure of the human brain, built from interconnected layers of artificial 'neurons' that process and transmit information. Each neuron takes in numeric inputs, multiplies them by adjustable values called weights, adds a bias term, and passes the result through a non-linear activation function before sending it to the next layer, per IBM's description of the architecture. Stanford HAI notes that this layered structure is what allows neural networks to extract patterns from large amounts of data, forming the basis of deep learning.

Step One: The Forward Pass

Training begins with weights and biases set to random values, IBM explains. During the 'forward pass,' input data moves through the network layer by layer: each neuron computes its weighted sum, applies its activation function, and forwards the output, until the final layer produces a prediction. At this early stage, because the weights are random, that prediction is essentially a guess.

Step Two: Measuring the Error

The network then compares its prediction to the correct answer using a loss function, a formula that quantifies how far off the output was, according to IBM. Different tasks use different formulas — mean squared error is common for predicting numbers, while cross-entropy is used for classification tasks such as sorting images into categories.

Step Three: Backpropagation Traces the Blame

The 'backward pass' is where learning happens. IBM describes backpropagation as working from the output layer back toward the input layer, using a calculus technique called the chain rule to determine how much each individual weight contributed to the overall error. The result is a gradient: a set of values, one for every weight and bias, indicating the direction and size of the adjustment that would reduce the error.

IBM points to the efficiency of this approach: for a network with one million weights, testing each parameter individually by trial and error would require more than a million separate forward passes through the network. Backpropagation reaches the same information in just two passes — one forward, one backward.

Step Four: Gradient Descent Updates the Weights

Once the gradient is known, gradient descent moves each weight a small step in the direction that lowers the error, IBM explains. The size of that step is controlled by the 'learning rate': a higher learning rate speeds up training but risks overshooting the best values, while a lower rate is more precise but slower to converge. The four steps — forward pass, loss calculation, backward pass, weight update — repeat over and over, often across millions of examples, until the loss stops meaningfully decreasing.

From Lab Technique to ChatGPT and Gemini

Large language models apply this same backpropagation-and-gradient-descent cycle at a much larger scale, across two distinct stages. The first, pretraining, exposes the model to massive amounts of unlabeled text — corpora containing hundreds of billions of tokens, according to RBC Borealis — and repeatedly asks it to predict the next word in a sequence. A research blog from RBC Borealis describes this foundation model as fundamentally a network trained to 'predict the next word in a partially complete input string,' which teaches it grammar, facts and general patterns but does not by itself make it a reliable assistant.

The second stage, fine-tuning, adapts the pretrained network to specific behaviour. RBC Borealis outlines three common approaches: supervised fine-tuning on human-written example answers; reinforcement learning from human feedback (RLHF), in which a separate reward model trained on human preferences guides further weight updates; and newer methods such as direct preference optimisation, which skip the separate reward model but still rely on backpropagation. This fine-tuning stage is what turns a general next-word predictor into a system that follows instructions and declines certain requests, rather than simply continuing text in whatever direction is statistically likely.

Why the Mechanism Matters

Because both pretraining and fine-tuning rely on the same underlying backpropagation-and-gradient-descent loop, improvements to that core technique — faster hardware, better loss functions, more efficient optimisation — affect every downstream application at once, from image recognition to conversational assistants. Stanford HAI notes that neural networks now form the foundation of deep learning systems used in facial recognition, voice assistants and autonomous vehicles, all of which are trained through variations of the same four-step cycle described above.