Transformers, End to End

You have all the parts now: tokens, embeddings, attention. Time to watch them run together.

Our prompt: "The capital of France is"

Station 1 — Tokenize

The chopping machine turns text into IDs.

"The capital of France is"  →  [464, 3139, 286, 4881, 318]

Station 2 — Embed

Each ID becomes a vector — a point on the map of meaning.

464  →  [0.02, -0.41, ...]   ← ~4,096 numbers each

Station 3 — Add position

A problem: attention is a set operation. It compares everything to everything with no inherent notion of order. To raw attention, "dog bites man" and "man bites dog" are the same bag of words.

So we mix position information into each vector. Now France doesn't just mean France — it means France, in slot 4. (Modern models mostly use rotary embeddings, RoPE, which encode position as a rotation. The principle is what matters: order has to be injected, not assumed.)

Station 4 — The stack

This is the engine room. The same block, repeated 32, 80, 120 times. Every block does two things:

1. Attention — every token looks at every other token and pulls in what's relevant. Gathering context.
2. A feed-forward network (MLP) — each token, now aware of its neighbours, gets processed on its own through a small network. Thinking about what it gathered.

Attention moves information between tokens. The MLP does the work within a token. Gather, then think. Gather, then think. Eighty times over.

Two structural details make deep stacks trainable at all:

• Residual connections. Each block adds its output rather than replacing it. Picture a conveyor belt (the "residual stream") carrying each token's representation, with every block reaching in to annotate it. Nothing is destroyed; meaning accumulates. Without this, gradients vanish and deep networks refuse to learn.
• Layer normalisation. Keeps the numbers in a sane range so nothing explodes over 80 layers of compounding.

By the top of the stack, the vector sitting at the final position — is — has absorbed the entire prompt. It isn't the word "is" any more. It's something closer to "the slot right after someone asked for France's capital."

Station 5 — Predict

That final vector gets multiplied out into one score (a logit) for every token in the vocabulary. All ~100,000 of them. Softmax turns those scores into probabilities:

" Paris"    →  0.87
" located"  →  0.04
" a"        →  0.02
" Lyon"     →  0.001
...and ~100,000 others

That's the entire output of the model. Not a sentence. Not an answer. A probability distribution over what comes next.

Station 6 — Sample, append, repeat

Pick a token from that distribution. Glue it onto the end of the input. Run the whole thing again.

"The capital of France is"          →  " Paris"
"The capital of France is Paris"    →  "."
"The capital of France is Paris."   →  <end>

This loop is autoregression, and it's why chatbots type at you word by word instead of blinking a finished paragraph into existence. You're not watching a cosmetic animation. You're watching the machine run, once per token.