The film
About 5:50, hosted by Pylon. It asks you one question and waits for your answer before it shows you. The narrated cut runs 4:30, voiced by Pylon in a synthesized voice made from the author's own voice; the author approved these cuts for publishing. Captions are on by default. The live version under it draws the same film in your browser, waits for your answer at each question, and runs a little longer; the full words are also below.
Nothing loads until you press play.
The lab: Pylon's race
Race careful steps, the heavy ball and Adam on three hills, each with the step you give it, from wherever you click. On the long narrow valley, at the steps the film chose, the heavy ball got down to stay in 285 updates, careful steps in 6,217 and Adam in 16,790. Then train the spiral network from the bucket chain with any walker and step, and compare the runs.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
Optimisers
Optimisers. Better ways to step.
Last time I walked downhill one careful step at a time. Tonight the walker gets weight, and then a pair of shoes.
The trouble with careful steps
Remember the long, narrow valley. Steep across, nearly flat along.
Careful steps have to stay small for the steep way, so along the floor they crawl. At the best step we tried, it takes 6,217 steps to get the loss under a ten-thousandth and keep it there.
Give the walker weight
Now make the walker heavy. Each step keeps 90 percent of the last step's motion and adds the new slope on top.
Across the valley, the slopes keep flipping, so they cancel. Along the floor they all point the same way, so they add up. The walker picks up speed exactly where it was crawling.
Boris Polyak described it in 1964. It's known as the heavy ball method.
Shoes for every direction
The other idea is shoes that adjust to the ground. Adam keeps a running average of each slope, and a running average of each slope squared.
It divides the first by the square root of the second. So a direction with a steady slope gets a step of about the same size, however steep or gentle it is: steep directions are slowed down, gentle ones sped up. A direction whose slope keeps flipping gets a short step.
Both averages start at zero, so at first they're too small. Adam scales them back up, so its very first step is the step size you chose, to within a hair, in every direction that has any slope at all.
The race
Same valley, same start. Each walker gets the best step from a few we tried.
Which walker gets down this valley first? (the film pauses here)
The heavy ball, by a mile: 285 steps. Careful steps take 6,217. Adam dips under the line at step 10,346, keeps wandering around it, and only stays under after 16,790.
And on the gentle bowl from last time, careful steps win: 33 steps, against 146 for the other two. No walker is best on every hill.
Past the edge
Remember the edge on the bowl: past a step of 0.5, careful steps fly off. Try 0.6.
Careful steps fly off, as before. The heavy ball's edge here is 0.95, so it gets down. Adam's steps don't grow with the slope, so it doesn't fly off, and here it gets down too.
Changing the step as you go
The step size doesn't have to stay fixed. Many models start with a warmup, small steps while everything is new, then let the step size fall away along a smooth curve.
And AdamW adds one more thing: almost every number shrinks by a small fraction at each step, kept separate from Adam's own step. It pulls numbers that aren't earning their keep back towards zero.
On real networks
On the spirals from the bucket chain, with each walker's step chosen on a separate set of points, careful steps first pass 95 percent on new points at step 2,000, the heavy ball at 1,300, and Adam at 900.
But run five different starting networks and batch orders, and Adam's final score ranges from 91.8 to 99.3 percent, while careful steps stay between 99.0 and 99.8. Fast, and on this network, less steady.
The language model from the attention lesson trains with AdamW. For its first 400 steps, with the same start and the same sentences, AdamW reaches a held-out loss of 0.85, the heavy ball 0.89, and careful steps 0.96.
What a race can't tell you
Every number tonight depends on the hill and on the handful of steps we tried for each walker. Try more, or a different hill, and the order can change.
And four hundred steps is the first fifth of the language model's training. It has barely started to copy, so this shows who gets off the line fastest, not who finishes best.
Goodnight
Weight carries you through the zigzags. Shoes give every direction its own stride. Neither wins every hill.
Next along the hill: learning a harbour by heart, and why that's a trap. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's surfaces and optimiser races, computed for this film. media/school/optimisers/facts.json, written by tools/school/fill_optimisers.mjs from system/school/optimisers.mjs and gradients.mjs. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/optimisers
Used for: Rosenbrock from (-1.2, 1), threshold 1e-4 and staying under: plain descent (best of 5 steps, 0.0015) 6,217 updates; heavy ball (beta 0.9, best step 0.003) 285; Adam (best step 0.001) first under at 10,346, stays under from 16,790. Tilted bowl, threshold 1e-6: plain descent (0.4) 33 updates; heavy ball and Adam 146 each. At step 0.6 on the bowl: plain descent diverges; the heavy ball's measured edge is 0.95 (predicted 0.95) and it converges; Adam converges.
Does not prove: Anything general about optimisers; the order depends on the surface and on the step grids tried.
Polyak 1964, the heavy ball method. Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1-17. https://doi.org/10.1016/0041-5553(64)90137-5
Used for: The heavy ball method adds a multiple of the previous step to the gradient step: x(k+1) = x(k) - a g(k) + b (x(k) - x(k-1)); it is widely known as the heavy ball method.
Does not prove: That momentum helps on every problem.
Kingma and Ba 2015, Adam: A Method for Stochastic Optimization. Kingma, D. P., Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. arXiv:1412.6980. https://arxiv.org/abs/1412.6980
Used for: Adam keeps exponential moving averages of the gradient and of its elementwise square, corrects both for their initialisation at zero, and divides the first by the square root of the second (Algorithm 1). With bias correction, the first step is alpha times g / (|g| + eps) in each coordinate: alpha to within eps wherever the gradient is non-zero; with a steady gradient the step is about alpha whatever the gradient's scale (section 2.1).
Does not prove: That Adam generalises as well as other methods on every task.
Loshchilov and Hutter 2017, SGDR: Stochastic Gradient Descent with Warm Restarts. Loshchilov, I., Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. arXiv:1608.03983. https://arxiv.org/abs/1608.03983
Used for: Cosine annealing lowers the learning rate along half a cosine from its maximum to its minimum.
Does not prove: That cosine decay is the best schedule.
Loshchilov and Hutter 2019, Decoupled Weight Decay Regularization. Loshchilov, I., Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019. arXiv:1711.05101. https://arxiv.org/abs/1711.05101
Used for: AdamW applies weight decay directly to the weights, separately from the adaptive gradient step and scaled by the schedule multiplier, instead of adding an L2 term to the gradient. In our model the decay applies to the matrices and embeddings (20,096 of 20,864 numbers), not to gains and biases.
Does not prove: That decoupled decay is better for every model.
Goyal and others 2017, Accurate, Large Minibatch SGD. Goyal, P., et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv:1706.02677. https://arxiv.org/abs/1706.02677
Used for: A gradual warmup of the learning rate over the first 5 epochs made large-minibatch training work at a high learning rate (section 2.2).
Does not prove: That warmup is needed for every model, or how common it is.
Night School's spiral network, retrained with each optimiser. media/school/optimisers/facts.json (spiral section): the F4 network, 4,000 updates, batch 32; steps chosen on a separate validation draw (seed 4), never on the held-out set. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/optimisers
Used for: First update at 95% held-out accuracy: plain descent 2,000, heavy ball 1,300, Adam 900. Over 5 network and batch-order seed pairs, final held-out accuracy ranges: Adam 0.9175 to 0.9925, plain descent 0.99 to 0.9975, heavy ball 0.995 to 0.9975 (facts.json, spiral.seedSpread).
Does not prove: That Adam is less stable in general; one small network, five seeds.
Night School's tiny transformer, retrained for 400 updates with three optimisers. media/school/optimisers/facts.json (transformer section): same initial weights, batches, warmup and clipping as the lab trainer; only the update rule differs; AdamW is the trainer's own and reproduces model.json's curve. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/optimisers
Used for: After 400 updates, held-out loss: AdamW 0.8456, heavy ball 0.8925, plain descent 0.9608 (each at its best step from a 3 or 4 point grid, chosen on 64 validation rows). Besides the update rule, the AdamW arm differs in its weight decay and its second-moment average; at 400 updates copy accuracy is 0.32 (AdamW) against a chance level of about 0.10.
Does not prove: Anything about full training: at 400 updates no arm has learned the copy skill yet.
What this lesson leaves out
Which walker wins depends on the hill and on the handful of steps tried for each; comparisons between optimisers are sensitive to how carefully each one is tuned, and these are small, chosen surfaces.
The language model comparison covers its first 400 updates only, a fifth of its training, before it has properly learned to copy; it shows who gets off the line fastest, not who finishes best.
Before this: Backpropagation (F4). Next: The held-out set (F6).