The Attention Mechanism

Read this:

The trophy didn't fit in the suitcase because it was too big.

What's "it"? The trophy. Obvious.

Now change one word:

The trophy didn't fit in the suitcase because it was too small.

Now "it" is the suitcase. You flipped your interpretation without noticing you did any work.

Nothing in the grammar told you. Both readings parse fine. You resolved it by understanding what fitting means — and you did it by letting the word "it" reach back across the sentence and check the other words against each other.

That reaching back is attention.

What came before, and why it choked

Older models (RNNs) read left to right, squeezing everything they'd seen into one fixed-size summary — like reading a book through a keyhole while frantically taking notes on a single index card. By the end of a long sentence, the beginning had been overwritten. And because word 50 couldn't be read until word 49 was done, it was hopelessly serial — no way to use a GPU properly.

In 2017 a paper landed with a title that was also a thesis: Attention Is All You Need. Throw out the sequential reading. Let every word look at every other word, all at once, and let the model learn which looks matter.

That's the transformer. Nearly everything you've heard of is built on it.

Query, Key, Value

Attention has a reputation for being maths-heavy. The mechanism underneath is something you already do every day.

The conference analogy. You're at a conference, wearing a name badge, wanting to find someone useful.

• Your Query is what you're looking for: "I need someone who knows Rust."
• Everyone's Key is their badge — the advertisement of what they offer: "Rust compiler team", "marketing", "catering".
• Their Value is what you actually get if you talk to them: the real contents of their knowledge.

You scan the room, compare your query against every badge, and pay attention in proportion to the match. Mostly the Rust person. A bit to the C++ person — adjacent, might help. Basically none to catering.

Every token does exactly this, simultaneously.

Each token emits three vectors, all derived from its embedding by learned weights:

Query (Q)  — "here's what I'm looking for"
Key   (K)  — "here's what I'm advertising"
Value (V)  — "here's what you get if you attend to me"

The four steps.

1. Score. Compare every query against every key with a dot product — a cheap way of asking "how aligned are these two directions?" (Remember the map: same direction, related meaning.)
2. Scale. Divide by the square root of the dimension. Housekeeping — big vectors produce huge dot products, which make the next step degenerate into picking one winner and ignoring everything else.
3. Softmax. Turn the scores into weights that sum to 1. This is the "attention" — a budget of focus, spread across the sentence. Give more to one word and you necessarily give less to another.
4. Mix. Add up everyone's Value vectors, weighted by that budget.

The output for each token is a blend of the whole sentence, weighted by relevance to that token.

So when it runs this, its query — "what noun am I standing in for?" — scores high against trophy and suitcase, and the mix pulls their meaning into it. The word it stops being an empty placeholder and starts carrying the trophy with it. That's the "drifting" from the embeddings topic, made concrete.

Because it's too big, the balance tips toward trophy. Change it to too small and the same machinery tips the other way. Nobody wrote that rule. It's in the learned weights that produce Q and K.