Follow one prompt through the machine
You have all the parts now: tokens, embeddings, attention. Time to watch them run together.
Our prompt: "The capital of France is"
Station 1 — Tokenize
The chopping machine turns text into IDs.
"The capital of France is" → [464, 3139, 286, 4881, 318]
Station 2 — Embed
Each ID becomes a vector — a point on the map of meaning.
464 → [0.02, -0.41, ...] ← ~4,096 numbers each
Station 3 — Add position
A problem: attention is a set operation. It compares everything to everything with no inherent notion of order. To raw attention, "dog bites man" and "man bites dog" are the same bag of words.
So we mix position information into each vector. Now France doesn't just mean
France — it means France, in slot 4. (Modern models mostly use rotary
embeddings, RoPE, which encode position as a rotation. The principle is what
matters: order has to be injected, not assumed.)
Station 4 — The stack
This is the engine room. The same block, repeated 32, 80, 120 times. Every block does two things:
- Attention — every token looks at every other token and pulls in what's relevant. Gathering context.
- A feed-forward network (MLP) — each token, now aware of its neighbours, gets processed on its own through a small network. Thinking about what it gathered.
Attention moves information between tokens. The MLP does the work within a token. Gather, then think. Gather, then think. Eighty times over.
Two structural details make deep stacks trainable at all:
- Residual connections. Each block adds its output rather than replacing it. Picture a conveyor belt (the "residual stream") carrying each token's representation, with every block reaching in to annotate it. Nothing is destroyed; meaning accumulates. Without this, gradients vanish and deep networks refuse to learn.
- Layer normalisation. Keeps the numbers in a sane range so nothing explodes over 80 layers of compounding.
By the top of the stack, the vector sitting at the final position — is — has
absorbed the entire prompt. It isn't the word "is" any more. It's something
closer to "the slot right after someone asked for France's capital."
Station 5 — Predict
That final vector gets multiplied out into one score (a logit) for every token in the vocabulary. All ~100,000 of them. Softmax turns those scores into probabilities:
" Paris" → 0.87
" located" → 0.04
" a" → 0.02
" Lyon" → 0.001
...and ~100,000 others
That's the entire output of the model. Not a sentence. Not an answer. A probability distribution over what comes next.
Station 6 — Sample, append, repeat
Pick a token from that distribution. Glue it onto the end of the input. Run the whole thing again.
"The capital of France is" → " Paris"
"The capital of France is Paris" → "."
"The capital of France is Paris." → <end>
This loop is autoregression, and it's why chatbots type at you word by word instead of blinking a finished paragraph into existence. You're not watching a cosmetic animation. You're watching the machine run, once per token.