The model never sees your words
Here's the first thing that surprises people: a language model has never read a word in its life. Not one.
Before your text reaches the model, it passes through a tokenizer — a chopping machine that cuts your sentence into chunks called tokens. The model only ever sees those chunks, as numbers.
The rubber stamp set
Imagine you can only write using a set of pre-printed rubber stamps. You have
about 100,000 of them. Common words earned their own stamp — the, and,
cat. But there's no stamp for every word in existence, so rarer words have
to be built from smaller stamps:
"tokenization" → "token" + "ization"
"strawberry" → "str" + "aw" + "berry"
"Chukwuemeka" → "Ch" + "uk" + "w" + "eme" + "ka"
The model reads in stamps and writes in stamps. It never handles letters, the way you never handle individual atoms when you pick up a cup.
Where the stamps come from
Nobody sat down and wrote the list. It's learned from a mountain of text by an algorithm (usually byte-pair encoding) that starts with single characters and repeatedly glues together the pair that shows up most often. Do that tens of thousands of times and frequent words fuse into single tokens while rare ones stay in pieces.
So the vocabulary is a popularity contest. Common text gets efficient stamps. Unusual text gets spelled out the slow way.
A rule of thumb worth memorising
For ordinary English: 1 token ≈ 4 characters ≈ ¾ of a word.
So ~750 words ≈ ~1,000 tokens. That single ratio is the one you'll use over and over — it's how you estimate cost, and how you estimate whether something fits.
One subtlety that trips people up: the leading space is part of the token.
"cat" and " cat" are two different stamps with two different numbers. The
model learned that spaces live at the front of words, not between them.