cutaway · How It Reads

Step 1 · Make a guess first

How does it hold onto your text?

Type a line below. Before you turn it into numbers, make a call: when this model takes in "hi there," how does it actually store it?

As the words themselves As one number per letter As a single number for the whole line

The line to turn into numbers (change it to anything you like):

Step 2 · Your text, as numbers

Every character becomes one number

What the model actually reads

That is tokenization: text in, a list of numbers out, and the same numbers turned back into text when the model writes. From here on the model never touches a letter. It only ever sees these IDs, and it has no idea that a "word" is even a thing.

This is not a teaching simplification. It is the literal first step inside every model you use. When you send a prompt to ChatGPT, the very first thing it does, before it "understands" anything at all, is turn your text into numbers exactly like this.

Quick check: from here on, what does the model actually work with?

Your letters The list of numbers

You finished Lesson 1 · Next up

Lesson 2: Watch It Learn →

You have seen the numbers go in. Now press start and watch the model teach itself, from pure noise to real sentences, one tiny correction at a time.

Step 3 · There is more than one way to do this

Tiny vocabulary, or short text. Pick one.

This model takes the simplest path: one number per character, a vocabulary of just 65. That keeps the vocabulary tiny but makes every line long. The big models do it differently. They chop text into common chunks ("sub-words"), which keeps each line short but needs a vocabulary of around 50,000. Same job, opposite trade.

Real models group characters into tokens, short chunks; cutaway uses single characters so you can watch every step. Here is the same line, chopped both ways:

cutaway · one character at a time

Shakespeare␣wrote

17 characters · vocabulary of 65

real models (GPT, Claude) · subword tokens

Shakespeare wrote

roughly 3 tokens · vocabulary near 50,000

Two ways to tokenize: character-level, used by nanoGPT, with a vocabulary of 65 and very long sequences; versus sub-word, used by OpenAI, with a vocabulary near 50,000 and short sequences. A balance scale weighs vocabulary size against sequence length. — Character-level (this model) vs. sub-word (GPT-class models): vocabulary size traded against text length.