How Large Language Models Work: The Simple Idea Behind ChatGPT

In a seven-minute video made for an exhibit at the Computer History Museum, 3Blue1Brown — the channel famous for animating mathematics — distills what a large language model actually is. No jargon, no hype: just the core machinery, from next-word prediction to the transformer, with enough concrete numbers to make the scale sink in. Here is the explanation, in full.

The Core Idea: A Machine That Predicts the Next Word

The whole thing starts with a thought experiment. Imagine you come across a short movie script describing a scene between a person and their AI assistant. The script contains what the person asks the AI, but the AI's response has been torn off. Now suppose you also have a magical machine that can take any piece of text and give a sensible prediction of what word comes next. You could finish the script by feeding in what you have, seeing what the machine predicts as the first word of the AI's answer, and then repeating the process over and over with a growing script until the dialogue is complete.

When you interact with a chatbot, this is exactly what is happening.

A large language model is, at bottom, a sophisticated mathematical function that predicts what word comes next for any piece of text. It does not predict one word with certainty; instead, it assigns a probability to every possible next word. To build a chatbot, you lay out text describing an interaction between a user and a hypothetical AI assistant, append whatever the user types as the first part of the interaction, and then have the model repeatedly predict the next word that such an assistant would say in response. That prediction is what gets presented to the user.

One detail matters for how the output feels: the text looks much more natural if the model is allowed to select less likely words at random along the way. The practical consequence is that even though the model itself is deterministic, a given prompt typically produces a different answer each time it is run.

Training on the Whole Internet

Models learn to make these predictions by processing an enormous amount of text, usually pulled from the internet. To put it in human terms: if a person tried to read the amount of text used to train GPT-3, reading non-stop, 24 hours a day, seven days a week, it would take more than 2,600 years. And larger models trained since then use much, much more.

Training is a bit like tuning the dials on a big machine. A language model's behavior is entirely determined by a huge set of continuous values, usually called parameters or weights. Changing those parameters changes the probabilities the model assigns to the next word for any given input. What makes a language model "large" is that it can have hundreds of billions of these parameters.

No human ever deliberately sets those parameters. They begin at random, which means the model initially outputs gibberish. They are then repeatedly refined using many example pieces of text. A training example might be just a handful of words, or it might be thousands; either way, the procedure is the same: pass all but the last word of the example into the model, and compare the model's prediction with the true last word. An algorithm called backpropagation then tweaks all the parameters so that the model becomes a little more likely to choose the true last word, and a little less likely to choose every other word.

Do this across many trillions of examples and something remarkable happens: the model not only becomes more accurate on the training data, it also starts making reasonable predictions on text it has never seen before.

The Staggering Cost of Computation

Given the number of parameters and the amount of training data, the scale of computation involved is mind-boggling. Here is the illustration the video offers: imagine you could perform one billion additions and multiplications every single second. How long would it take to do all the operations involved in training the largest language models? A year? Ten thousand years? The answer is well over 100 million years.

That staggering computation is only possible because of special computer chips optimized to run many operations in parallel — GPUs.

From Text Predictor to Assistant: Human Feedback

Pre-training is only part of the story. The goal of auto-completing a random passage of internet text is very different from the goal of being a good AI assistant. So chatbots undergo a second kind of training, just as important, called reinforcement learning with human feedback, or RLHF. Workers flag unhelpful or problematic predictions, and their corrections further change the model's parameters, nudging the model toward the kinds of predictions users actually prefer.

The Transformer: Reading Everything at Once

Not all language models can be parallelized easily. Prior to 2017, most language models processed text one word at a time. Then a team of researchers at Google introduced a new architecture called the transformer. Transformers do not read text from start to finish; they soak it all in at once, in parallel.

The very first step inside a transformer — and most other language models, for that matter — is to associate each word with a long list of numbers. Training only works with continuous values, so language has to be encoded as numbers, and each list of numbers may, in some way, encode the meaning of its word.

What makes transformers unique is a special operation called attention. Attention lets all these lists of numbers talk to one another and refine the meanings they encode based on the surrounding context, all in parallel. For example, the numbers encoding the word "bank" might be adjusted based on context to encode the more specific notion of a riverbank.

Transformers typically also include a second kind of operation, the feed-forward neural network, which gives the model extra capacity to store patterns about language learned during training. Data repeatedly flows through many iterations of these two fundamental operations, and with each pass the hope is that every list of numbers is enriched to encode whatever information is needed to make an accurate prediction of the word that follows.

At the very end, a final function is applied to the last vector in the sequence — which by now has been influenced by all the context from the input text, plus everything the model learned during training — to produce the prediction of the next word. As before, the output is a probability for every possible next word.

Emergent Behavior: Why the Model Is Hard to Explain

Researchers design the framework for each of these steps, but the specific behavior of the model is an emergent phenomenon, a product of how those hundreds of billions of parameters get tuned during training. That makes it incredibly challenging to determine why the model makes the exact predictions it does.

What is easy to observe is the result: when you use a large language model's predictions to autocomplete a prompt, the words it generates are uncannily fluent, fascinating, and even useful.

Key Takeaways

• Next-Word Prediction: A large language model is a next-word predictor: it assigns probabilities to every possible next word and keeps sampling to build a response.
• Scale of Training: Models are trained on internet-scale text — GPT-3 alone would take a human over 2,600 years of non-stop reading — and refined by backpropagation over trillions of examples.
• Human Guidance: Pre-training is followed by reinforcement learning with human feedback, which shapes the model into a helpful assistant.
• Parallel Processing: The transformer architecture, introduced by Google researchers in 2017, made modern scale possible by processing text in parallel rather than one word at a time.
• Emergent Capabilities: The model's behavior is emergent; nobody can fully explain why it makes the exact predictions it does — even though the outputs are remarkably fluent.