The film
About 3:45, hosted by Tallow. It asks you one question and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.
Nothing loads until you press play.
The lab: the shelf
Pick a width and a number of floors, train the model in this page, and watch its gap above the floor against the work done, over the film's frontier. The film's twelve models run from 616 to 100,544 numbers outside their word and place tables.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
Scaling
Scaling. Bigger, longer, how much.
Large language models are large because, for years, making them bigger and training them longer kept making them better. Tonight we try that on a shelf of small ones, and we are honest about what a shelf this small can show.
The shelf
We built 12 versions of the attention model: 8 to 64 numbers per word, one to three floors. The smallest has 616 numbers outside its word and place tables; the largest, 100,544.
Each trained for 4,000 steps on the same text, and each is scored by how far its held-out loss sits above the floor, 0.6585, the loss of the rules themselves.
Bigger
The smallest ends 0.15 above the floor and gets 36 percent of look-back words right. Two floors of 16: 0.0161 above the floor, and 99.6 percent.
Over twenty times more numbers than the two-floor model of 16. How much closer to the floor? (the film pauses here)
Past that, for the models with two or three floors, the gap barely moves. The lesson's own recipe, trained twice as long, ends 0.0150 above the floor, and the largest 0.0225.
Three runs of the same model with different seeds spread about as much as that. So on this text, at this length, we cannot see any gain from size above about 4,480 numbers.
Longer, and the frontier
Now count the work instead: the arithmetic each training run has done, in floating-point operations.
At every amount of work, take the lowest loss any model on the shelf had reached. Plotted on a log-log chart, their gaps above the floor fall on a straight line, with a slope of -0.65. More work, a smaller gap, at a steady rate.
And the best model for each amount of work mostly grows as the work grows: small models win the small budgets and bigger ones the big ones, though not at every step.
The real curves
Kaplan and colleagues, in 2020, found curves like this over models from hundreds of numbers to more than a billion, and over a much wider range of work.
Hoffmann and colleagues, in 2022, found that most large models of the time were too big for their training text. Their Chinchilla had 70 billion numbers, a quarter of Gopher's, trained on more than four times the text, and beat it.
Our shelf is far too small to test their numbers. What it can show is the shape: a floor, a gap that shrinks with work, and a best size that grows with the budget.
Goodnight
Bigger and longer help until a model gets close to what its text allows. On our shelf, that comes early.
Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's shelf of small models, trained for this film. media/school/scaling/facts.json, written by tools/school/fill_scaling.mjs from system/school/scaling.mjs: 12 models (width 8 to 64, depth 1 to 3) trained 4,000 steps on the same text, scored by held-out loss above the generator's floor; fits on log-log axes; the frontier over compute; three seeds at two grid points. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/scaling
Used for: Excess over the floor (0.6585): smallest 0.15426, d16 L2 0.016079, d32 L2 0.014979, d64 L3 0.022533. Seed spread of log10 excess: 0.13761 (d16 L2), 0.11772 (d32 L2). Frontier slope of excess against compute: -0.64767 (R2 0.97109); best size against budget slope 0.41372.
Does not prove: The papers' exponents; one learning rate and schedule for every size is a confound.
Kaplan and others 2020, Scaling Laws for Neural Language Models. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
Used for: Loss falls as a power law in non-embedding parameters, data and compute over about 768 to 1.5e9 parameters and several orders of magnitude of compute.
Does not prove: Anything about this model.
Hoffmann and others 2022, Training Compute-Optimal Large Language Models (Chinchilla). Hoffmann, J., Borgeaud, S., Mensch, A., and others (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. https://arxiv.org/abs/2203.15556
Used for: Parameters and training tokens should grow in equal proportion with compute; many large models of the time were undertrained; Chinchilla (70B parameters, 1.4T tokens) outperforms Gopher (280B, 300B tokens) at the same compute.
Does not prove: Anything about this model.
What this lesson leaves out
The shelf is tiny next to the papers' models and one learning rate serves every size; it shows the shape of scaling, not their exponents.
Above about 4,500 numbers, the differences between sizes are within the spread of three seeds.