The film
About 4:34, hosted by Tallow. It asks you two questions and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.
Nothing loads until you press play.
The lab: the tower
Switch off any head or MLP of the attention model, or make a room replace the stream instead of adding to it, and score it on its held-out rows. With everything on it copies 99% of far-back words; attention and the MLP hold 8,448 and 8,384 of its 20,864 numbers.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
The transformer block
The transformer block. Two floors, one stream.
Every word's list of numbers climbs the tower. On each floor, two rooms read it and add something to it. Tonight, what those rooms are, and what each one adds.
The stream
Start with a word's 32 numbers from the word table, its embedding, and add to them, number by number, 32 more from a second table for its place in the sentence. Call that the stream.
Each room reads the stream, works out something, and adds its answer back on. Nothing is thrown away. That adding is called the residual connection.
Before each room reads, it takes a copy of the stream rescaled to a steady size. That is layer norm, and in this model, as in GPT-2, it comes first in each room. The stream itself is left alone.
Two rooms per floor
The first room is attention: each word looks back along the sentence and takes in what it needs. This model has 2 heads per floor, two separate ways of looking.
The second room is a small network that works on each word alone, called the MLP or feed-forward layer. It widens the 32 numbers to 64, bends them, and narrows them back.
Attention and the MLP hold almost the same number of the model's 20,864 numbers: 8,448 and 8,384.
Reading the stream on the way up
We can stop the stream on any landing and ask it to guess the next word early, using the model's own last two steps: its final layer norm and its output table.
Which floor will matter most for copying a word from far back? (the film pauses here)
Straight from the table, the guess has a held-out loss of 2.33. After the first floor, 0.86. After the second, 0.69.
Copying a word from far back is the slow part: right 43 percent of the time after the first floor, and 99 percent at the top.
Switch one room off
Now switch rooms off, one at a time, and score the model again. With everything on, it copies right 99 percent of the time.
Silence the second head on the top floor, and copying falls to 40 percent. Silence the first head there, and nothing changes: 99 percent. On the top floor, one head does most of the copying, and its neighbour is close to idle. Copying needs a head on the first floor too: silence it, and copying falls to 38 percent.
Switch off the first floor's MLP, and the loss jumps from 0.69 to 5.96, worse than a blind guess.
On an untrained model, the same switches barely move the loss. The jobs were learned, not built in.
Why add, not replace
What if a room replaced the stream with its answer, instead of adding to it?
If a room replaces the stream instead of adding to it, what happens to the loss? (the film pauses here)
For the top floor's attention, the loss goes to 6.62. The room was only ever adding a little, and the rest of the stream was still needed.
One room is the odd one out. The first floor's MLP adds a vector about 10 times the size of the stream it reads, so replacing barely matters there: 0.69.
Goodnight
A block is two rooms on one stream: attention looks along the sentence, the MLP works on each word, and each adds its answer back on.
Next in the lighthouse: how the whole tower is trained. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's attention model, taken apart for this film. media/school/block/facts.json, written by tools/school/fill_block.mjs from system/school/block.mjs: a hooked forward pass (equal to the model's own forward, bit for bit) that reads the residual stream, applies the final norm and output matrix at each depth, and knocks out heads, attention layers and MLPs by zero, mean or shuffle, or makes a sub-layer replace the stream instead of adding to it; scored on 256 held-out rows (6,144 next words, 506 copy positions). https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/block
Used for: Parameters: 20864 in all; attention 8448, MLP 8384, embeddings 2240, output 1472, norms 320. Readout at each depth (held-out loss): 2.33 from the table, 1.73 after layer 0 attention, 0.86 after layer 0 MLP, 0.75 after layer 1 attention, 0.69 at the top; copy accuracy 0.12, 0.21, 0.43, 0.82, 0.99. Zero knockouts: layer 1 head 1 drops copy accuracy to 0.4032; layer 1 head 0 leaves it at 0.9921; layer 0 MLP raises the loss to 5.9605. On the untrained model knockouts move the loss by under 0.05. Replace instead of add: layer 1 attention loss 6.62, layer 0 attention 4.93, layer 1 MLP 2.85, layer 0 MLP 0.69 (its output is about 10 times the stream it reads).
Does not prove: That large models divide work the same way; a knockout shows what the model needs here, not what a part means.
Vaswani and others 2017, Attention Is All You Need. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 30. arXiv:1706.03762. https://arxiv.org/abs/1706.03762
Used for: The transformer layer has two sub-layers, multi-head self-attention and a position-wise feed-forward network, each with a residual connection and layer normalisation (section 3.1); the paper applies the norm after the add.
Does not prove: Anything about this model.
Radford and others 2019, Language Models are Unsupervised Multitask Learners (GPT-2). Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Used for: Layer normalisation was moved to the input of each sub-block (section 2.3).
Does not prove: Anything about this model.
nostalgebraist 2020, interpreting GPT: the logit lens (blog post). nostalgebraist (2020). interpreting GPT: the logit lens. LessWrong, 31 August 2020. A blog post, not peer reviewed. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
Used for: Applying the output step to the stream at intermediate layers gives guesses that improve with depth.
Does not prove: Anything about this model.
What this lesson leaves out
Two layers and two heads on a 46-word grammar: large models spread the same jobs over many more parts, and a knockout here shows what this model needs, not what a part means.
Reading the stream early uses the model's own final norm and output table; other readouts give other numbers, and facts.json gives both.
Before this: Attention (L10). Next: The training loop (L12).