The film
About 4:40, hosted by Pylon. It asks you two questions and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.
Nothing loads until you press play.
The lab: the training dock
Train the attention model from scratch in this page with your own step size, batch, clipping and weight decay, and watch its held-out loss and copy rate against the film's run. Rerun from its seeds, the loop gives back the lesson's model exactly: 29 of 29 weight tables.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
The training loop
The training loop. Five steps, two thousand times.
Every lesson so far has a piece of training in it. Tonight they go round together, and we watch the whole loop teach the attention model, from the first step to the last.
One lap
One lap goes like this. Take a batch of 16 rows from the training text. Run the model forward and score every next word: that is the loss.
Run backward to get the slope for every one of the model's 20,864 numbers. Then the optimiser takes one step downhill. That is one lap.
On one core of our test machine, with this lesson's own code, a lap takes about 21 milliseconds, and most of that is the backward pass: 14 of them.
The settings
The optimiser is Adam, with a little weight decay that shrinks most of the numbers slightly each step, as in AdamW. The layer norms' gains and the biases are left out.
The step size starts tiny and grows over the first 30 laps to 0.01. That is warmup.
And if the whole slope, all the numbers taken together, is ever longer than 1, it is shortened to 1 before the step. That is clipping. It fired on 76 of the 2,000 laps, 62 of them in the first 250.
Two thousand laps
Now run it. At lap zero, the held-out loss is 3.84, a blind guess.
By lap 100 it is 0.97: the easy patterns, which word tends to follow which, come first.
Will copying a word from far back be learned early, late, or bit by bit? (the film pauses here)
Copying a word from far back comes late, and then fast. At lap 500, copying is right 43 percent of the time. By lap 750, 97 percent.
We ran the loop again from the same seeds, and got the attention lesson's model back exactly: 29 of 29 weight tables, number for number.
Change one thing
Change one setting at a time, train for 1,000 laps, and compare. First, the noise: the same recipe on two other sets of training rows ends at 0.70 and 0.70, and the original at 0.70, all three within a few thousandths. A difference smaller than that cannot be told apart from luck.
A step size ten times smaller ends at 0.78, and copies right only 54 percent of the time. Ten times larger ends worse, at 0.95, copying 22 percent.
Make the step size ten times larger. Faster, or worse? (the film pauses here)
Without weight decay: 0.70, inside the noise. On this small model, at this length, it makes no difference we can see.
Batches of 4 rows for the same number of laps see a quarter of the text, and end at 0.87. Given the same text over four times the laps, they still end at 0.76.
A loop that cannot learn
Two broken loops, as checks. With a step size of zero, nothing moves: the loss stays at 3.84.
Turn every slope around, so each step goes uphill, and the loss climbs to 68. The loop only works because the slope points the right way.
Goodnight
Batch, forward, loss, backward, step. Two thousand times, and a blind guess becomes a model that copies.
Next on the dock: what goes into the batches. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's training loop, rerun for this film. media/school/trainloop/facts.json, written by tools/school/fill_trainloop.mjs from system/school/trainloop.mjs: the attention model retrained from its seeds (2,000 steps) and compared with model.json; one-change variants of 1,000 steps each; gradient norms and clipping; timing in Node; two broken-loop controls. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/trainloop
Used for: The retrained model equals model.json: 29 of 29 weight tensors byte for byte, 81 of 81 curve points. Clipping at norm 1 fired on 76 of 2000 steps, 62 of them in steps 1 to 250; warmup 30 steps to learning rate 0.01. Held-out loss 3.84 at step 0, 0.97 at 100, 0.82 at 500, 0.72 at 750; copy accuracy 0.43 at 500 and 0.97 at 750. Variants at 1,000 steps: reference 0.6981; other data seeds 0.700 and 0.702; learning rate 0.001 0.7835 (copy 0.5395); 0.1 0.9506 (copy 0.2174); no weight decay 0.6995; batch 4 at the same steps 0.8702. Controls: learning rate 0 stays at 3.8414; the gradient sign flipped climbs to 68.5. About 21 ms per step in Node on one core: forward 6.3 ms, backward 14.0 ms, update 0.2 ms.
Does not prove: That these settings are best for larger models; timings are for one machine.
Kingma and Ba 2015, Adam: A Method for Stochastic Optimization. Kingma, D. P., Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. arXiv:1412.6980. https://arxiv.org/abs/1412.6980
Used for: Adam keeps running averages of each gradient and its square and steps by the first divided by the square root of the second (Algorithm 1); this model uses beta2 0.99 where the paper suggests 0.999.
Does not prove: Anything about this model.
Loshchilov and Hutter 2019, Decoupled Weight Decay Regularization (AdamW). Loshchilov, I., Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019. arXiv:1711.05101. https://arxiv.org/abs/1711.05101
Used for: Weight decay applied directly to the weights, separate from the gradient step (AdamW, Algorithm 2); here the decay is scaled by the full learning rate, as common libraries do.
Does not prove: Anything about this model.
What this lesson leaves out
Each changed setting is one run with one seed; only differences larger than the spread of the noise runs mean anything.
Timings are from one test machine; the lab's runs in your browser will be slower or faster.
Before this: The transformer block (L11). Next: Data (L13).