The film
About 5:18, hosted by Pylon. It asks you one question and waits for your answer before it shows you. The narrated cut runs 4:07, voiced by Pylon in a synthesized voice made from the author's own voice; the author approved these cuts for publishing. Captions are on by default. The live version under it draws the same film in your browser, waits for your answer at each question, and runs a little longer; the full words are also below.
Nothing loads until you press play.
The lab: practice and test
Fit a curve of any flexibility to 16 points, with or without a leash, and see its error on them and on 500 held-out points; draw new training points and watch the best flexibility move. Then train the spiral network on 10 to 400 points. In the film, 40 points gave 78.2% on new points and 400 gave 99.5%.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
The held-out set
The held-out set. Learning, or remembering?
I could learn this harbour by heart, every crate in its place. Then one night they send me to a different harbour.
Sixteen points
Here are 16 points. They come from a smooth curve, plus a little noise, the way real measurements do.
Let's fit a curve through them. A straight line misses the bend. A gentle curve follows it. And a wild one, with enough freedom, passes through every single point.
Which curve will do best on points it has never seen? (the film pauses here)
The wild one gets every point right: its error on them is about one part in a hundred billion. That's zero, apart from rounding in the arithmetic.
Points kept back
So keep some points back. 500 of them, drawn the same way, never used for fitting. That's the held-out set.
On those, the gentle curve's error is 0.33. The wild one's is 3,936. Between the points it learned it swings wildly, and past the outermost ones it shoots off the chart.
Make the curve more flexible, step by step, and the error on the training points only ever goes down. The error on the held-out points goes down, then turns around and climbs, with a few bumps on the way.
A leash
One fix is a leash: keep the wild curve, but charge it for every large number it uses.
A moderate charge brings its held-out error down from 3,936 to 0.43. That's the best of seven charges we tried, picked by looking at these same held-out points, so it flatters the leash a little. Pull too hard and it gets worse again: at the heaviest pull we tried, 0.70.
The spirals, with too few points
The bucket chain from last time learned two spirals from 400 points. Give it only 40.
It still gets every training point right. On new points it's right 78.3 percent of the time, against 99.5 with all 400.
Watch the held-out loss. It's lowest early, at step 300, and then climbs while the training loss keeps falling. Its score on new points still creeps up for a while, to 80 percent by step 1,000, but it grows ever more sure of itself, wrong answers included. After that the score stops improving, and training only tightens its grip on these forty points.
One draw is one draw
And a score from one training set is one draw. Pick five different sets of 40 training points, and the held-out score runs from 75.5 to 86.0 percent.
Even which curve looks best depends on the draw. Over two hundred draws of sixteen points, the gentle curve came out best 113 times, not every time.
The language model
The language model from the attention lesson was trained on a stream of freshly made sentences, a new batch every step. Its little grammar does repeat itself: 507 of its 32,000 training rows were exact copies of earlier ones. But no row came round more than 5 times, so it had little chance to learn its training rows by heart.
Score the finished model on rows it trained on, and its loss is about 0.63. On the next stretch of the same stream, which it didn't train on, 0.67. A small gap, not a chasm.
And its own held-out set turns out to be a little hard: 0.69, higher than five fresh draws of the same size, which range from 0.65 to 0.67.
Keep it held out
One rule makes all of this work. Once you use the held-out points to choose anything, how flexible the curve is, how long to train, they stop being held out.
So careful work keeps a third set locked away until the very end, and looks at it once.
Goodnight
Getting the practice right is not the test. Keep points back, and check on those.
That's the hill done. Next, down at the dock, we cut words into crates. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's curve fits, computed for this film. media/school/heldout/facts.json (poly section), written by tools/school/fill_heldout.mjs from system/school/heldout.mjs: least squares by Householder QR; ridge by QR on the design matrix stacked over sqrt(lambda) times the identity. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/heldout
Used for: 16 training points from sin(pi x) with noise sigma 0.2; 500 held-out points. Degree 3 held-out RMSE 0.330; degree 15 train RMSE 1.4e-11 and held-out RMSE 3,936. Ridge at degree 15: held-out RMSE 3,936 at lambda 0, 0.429 at 0.01 (the best of 7), 0.703 at 100. Over 200 training draws, degree 3 is the best degree on held-out data 113 times. Degree 3 and the penalty 0.01 were picked on the same held-out points they are scored on, which flatters them a little. The wild curve's held-out error is 2.99 on points inside the training range and far larger past the outermost training points.
Does not prove: Anything about real data; a toy curve chosen to show one effect.
Goodfellow, Bengio and Courville 2016, Deep Learning. Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. MIT Press. Chapter 5.2 (capacity, overfitting and underfitting) and 5.3 (hyperparameters and validation sets). https://www.deeplearningbook.org/
Used for: Training error falls as capacity grows, while test error typically falls and then rises (overfitting). Choosing hyperparameters on the test set biases it; a separate validation set is used for choices, and the test set only to estimate generalisation at the end.
Does not prove: Anything about a particular model.
Hoerl and Kennard 1970, Ridge Regression. Hoerl, A. E., Kennard, R. W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics 12(1), 55-67. https://doi.org/10.1080/00401706.1970.10488634
Used for: Ridge regression adds a penalty on the size of the coefficients (lambda times their sum of squares) to least squares.
Does not prove: Which lambda to use.
Night School's spiral network on 40 points against 400. media/school/heldout/facts.json (spiral section): the F4 network, plain descent at 0.3, 10,000 updates, on the first 40 of the 400 training points and, as the control, on all 400. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/heldout
Used for: Both reach training accuracy 1. Held-out accuracy 0.7825 with 40 points, 0.995 with 400. With 40 points the held-out loss is lowest at update 300 (0.648) and ends at 1.487. Over five different 40-point draws, held-out accuracy ranges from 0.755 to 0.86. After the held-out loss minimum at update 300, held-out accuracy still rises, to 0.8025 at update 1,000, while the held-out loss climbs.
Does not prove: Anything general about how many points a task needs.
Night School's tiny transformer, its training and held-out losses. media/school/heldout/facts.json (attention section): model.json's own curves, and the final weights rescored on rows they were trained on, on unseen rows of the same stream, on the lab's 256 held-out rows and on five fresh 256-row draws. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/heldout
Used for: The trainer draws a fresh batch every update; the generator repeats rows by chance: 507 of the 32,000 training rows were exact copies of earlier rows, and no row came round more than 5 times. 8 of the 256 'unseen' rows and 3 of the lab's 256 held-out rows also occur in training (facts.json, attention.repeats). Final weights: loss 0.626 on the last 16 trained batches, 0.668 on the next 16 unseen batches, 0.687 on the lab's held-out rows, 0.650 to 0.672 on five fresh draws.
Does not prove: That the model would not memorise with repeated data; it never had any.
What this lesson leaves out
The curve and the spirals are toy problems with known answers, so overfitting is easy to see; in real work the truth is unknown and the held-out set is the only window onto it.
Some choices in the film (the best degree, the best leash) were picked on the same held-out points they are scored on, which flatters them a little; the film says so, and the closing rule (a third set, looked at once) is the fix.
Before this: Optimisers (F5). Next, at the dock: Tokens (L7).