Context Windows & Why They Matter

Time to dismantle the most persistent illusion in this whole field.

When you chat with an AI, it feels like it remembers you. It refers back to what you said. It picks up threads. It seems to be carrying the conversation along with you.

It isn't. Not even slightly.

The mental model

Imagine a brilliant consultant with total amnesia. Every single time you speak to them, they've been rebooted — no memory of you, of this project, of the last five minutes. To function at all, they need a folder containing the entire conversation so far, which they read from scratch before every reply.

The context window is how thick that folder can be.

That's not a simplification. That's mechanically what happens.

Every time you send a message, your app re-sends the whole conversation — the system prompt, every previous turn, yours and the model's, plus any documents — as one big block of tokens. The model reads all of it, generates a reply, and is then wiped. It retains nothing. The next message re-sends everything again, plus one more turn.

The model isn't remembering. Your app is re-reading the folder aloud, every time. The illusion of memory is a trick performed by the software around the model.

Once this clicks, a whole class of confusing behaviour turns into arithmetic.

What's in the folder

Everything competes for the same budget:

┌─ Context window (say, 128,000 tokens) ──────────┐
│  System prompt / instructions                   │
│  Tool & function definitions                    │
│  Retrieved documents (RAG chunks)               │
│  The entire conversation so far                 │
│  Your newest message                            │
│  ← room reserved for the model's reply →        │
└─────────────────────────────────────────────────┘

That last line surprises people: the output shares the window. The reply gets written into the same space. Fill the window to the brim with input and there's nowhere for an answer to go.