How Do Large Language Models Work? Prediction, Attention and a Great Deal of Text

The training objective is almost embarrassingly simple. Text is chopped into tokens, which are roughly word fragments, so that a common word is one token and a rare one is several, and the model is shown a sequence and asked to predict the token that follows. Its prediction is compared with the actual next token, the error is used to adjust billions of internal numerical parameters very slightly, and the process is repeated across trillions of tokens of text. Nothing about grammar, facts, reasoning or style is specified anywhere. What emerges is a system whose internal parameters encode enormously detailed statistical structure about how language is used, and because accurate prediction of text written by people requires some representation of what the text is about, capabilities that were never the objective appear as a side effect of getting better at the objective.

The design that made this practical is the transformer, introduced in a 2017 paper from Google whose title announced that attention is all you need. Its key mechanism, self-attention, lets every token in a sequence look at every other and decide how much each one matters for interpreting it, which is how a model resolves what a pronoun refers to twenty words earlier. The essential pieces:

• •Embeddings, which convert each token into a long list of numbers positioned in a space where related meanings sit near each other
• •Attention heads, many per layer, each computing which earlier tokens are relevant to each current one and blending their information accordingly
• •Feed-forward layers between attention steps, which do most of the actual storage of learned patterns
• •Dozens to hundreds of such layers stacked, so that later layers work on increasingly abstract representations
• •A final step producing a probability for every possible next token, from which one is sampled
• •Crucially, attention processes a whole sequence in parallel rather than one word at a time, which is what made training on internet-scale data feasible on modern hardware

A model trained only on next-token prediction is not useful as an assistant, because it continues text rather than answering. Two further stages produce the behaviour people interact with. Instruction tuning trains the model on examples of requests paired with good responses, teaching it that the appropriate continuation of a question is an answer rather than more questions. Then a preference stage, usually some form of reinforcement learning from human feedback, has people compare pairs of model outputs, trains a separate model to predict those preferences, and adjusts the main model to produce outputs the preference model scores highly. This is where helpfulness, refusal behaviour, tone and formatting come from, and it is also where a good deal of the difficulty lies, since optimising for what human raters approve of is not the same as optimising for what is true or useful, and models can learn to produce confident, well-formatted answers that rate well and are wrong.

The failure modes follow from the design rather than from bugs. A model produces the most plausible continuation, and plausibility and accuracy are different properties, so a request for a citation yields something shaped exactly like a citation, with a realistic author, journal and year, that does not exist; this is usually called hallucination and is better understood as the system doing precisely what it was trained to do. Knowledge is frozen at the end of training unless the model is given documents to read at the time of the request. The context window, the amount of text the model can attend to at once, is finite, and material outside it is simply gone. Arithmetic and multi-step logic are unreliable because they require exact procedures rather than plausible continuations, though performance improves substantially when a model is prompted to work through steps explicitly or given access to a calculator. And because the training data reflects what people have written, the biases in that material are learned along with everything else.

Two questions are open and not merely unsettled by the public. The first is why scale works as well as it does: performance improves predictably with more parameters, more data and more compute, following empirical relationships known as scaling laws, and certain capabilities appear abruptly at particular scales rather than improving gradually, which nobody predicted and which is still being argued about, including whether the abruptness is real or an artefact of how it is measured. The second is what is happening inside. The field of interpretability has made real progress, identifying circuits that perform specific functions and features corresponding to recognisable concepts, but a model with hundreds of billions of parameters is not understood in the way a piece of engineered software is, and its designers cannot in general say why a particular output was produced. That gap between capability and understanding is the central fact about the technology, and it is why claims in either direction, that these systems merely retrieve text or that they think in any familiar sense, should be treated with suspicion.

A large language model is trained to predict the next fragment of text across trillions of tokens, and the capabilities it shows are side effects of doing that well rather than things it was taught. The transformer architecture makes it practical, using self-attention so every token can weigh every other, and further training on instruction examples and human preference comparisons turns a text continuer into an assistant. Fabricated citations, frozen knowledge and unreliable arithmetic follow directly from optimising for plausibility, and why scale produces these abilities is not fully understood.