cutaway ↖ LINCHPIN · AI Innovation Lab
Lesson 3 · What It Pays Attention To

It does not read every
letter equally.

When the model guesses the next character, it leans harder on some earlier letters than others. That leaning is called attention, and it is the heart of how these models work. You are about to see exactly what the trained model looks at.

Step 1 · Make a guess first

Before you look

Give the model a line of text. It will get ready to guess the next character. The question: when it decides, does it spread its attention evenly across all the earlier letters, or focus on just a few?

Spread evenly Focus on a few

The line to feed it (change it to anything you like):

See the real code, line by line
Attention Math From: model.py · CausalSelfAttention
Why it mattersThis is the mechanism that lets a model weigh some earlier letters more than others. It is what made modern AI work.
Key conceptScore how relevant each earlier letter is, turn the scores into percentages, then pull most from the ones that matter.
What to look for"F.softmax" turns raw scores into the exact percentages that glowed in the heatmap above.
att = (q @ k.transpose(-2, -1)) / math.sqrt(head_dim)
causal = torch.tril(torch.ones(T, T, device=x.device)).view(1, 1, T, T)
att = att.masked_fill(causal == 0, float("-inf"))
att = F.softmax(att, dim=-1)
self.last_attn = att.detach()
y = att @ v
Hover or tap any line above. Its plain-English explanation appears right here.