The film
About 4:23, hosted by Tallow. It asks you one question and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.
Nothing loads until you press play.
The lab: the measuring jugs
Type a number and see the values each format can store around it; run a long sum; or score the model on held-out rows in each mode. In the film, adding 1 stalled at 2,048 in fp16 and 256 in bf16; near 110, fp16's marks are 0.0625 apart. The model's loss was 0.6676 in 32 bits and 0.6679 in bf16 throughout.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
fp16, bf16 and tensor cores
fp16, bf16 and tensor cores.
Graphics chips multiply much faster in 16-bit numbers than in 32-bit ones. Tonight Tallow measures what a 16-bit number can hold, where it goes wrong, and why the sums are often kept in 32 bits.
Measuring jugs
Think of a number format as a measuring jug. The marks are its precision; the height is its range.
fp16 has 11 bits of significand and tops out at 65,504. bf16 keeps the range of 32-bit numbers but has only 8 bits of significand. 32-bit numbers have 24.
Near one hundred and ten
Look at the marks near 110. In fp16 they are 0.0625 apart. In bf16, 0.5. Any number in between is rounded to the nearest mark.
fp16 near 110. How far apart are the marks? (the film pauses here)
So in fp16 a number near 110 can be off by up to one thirty-second, about 0.03. That is about three parts in ten thousand. A job that needs one part in ten thousand needs a significand of 13 bits; fp16 has 11. We will need that in the next lesson.
A sum that stops
Now add 1 to a running total, five thousand times, keeping the total in fp16.
It stops at 2,048. There the marks are 2 apart, and adding 1 rounds back to where it was. In bf16 it stops at 256.
Small inputs, big sums
This is why tensor cores, the units that multiply small tables of numbers in one step, can take 16-bit inputs and keep the running sums in 32 bits. With fp16 inputs they can also sum in fp16. With bf16 inputs, the sums are always 32-bit.
Paulius Micikevicius and colleagues trained large networks this way in 2018 and kept a 32-bit master copy of the weights.
On the second floor's feed-forward table of our model, the error with fp16 inputs and 32-bit sums is about 0.0003 of the answer's size. Keep the sums in fp16 too, and it grows to about 0.001.
The whole model
Run the whole model with every stored number rounded to 16 bits and score it on 1,024 held-out rows.
The loss barely moves. It stays at 0.6676 with fp16 inputs and 32-bit sums, and rises to 0.6679 with bf16 throughout. Its numbers stay far below fp16's limit, so nothing overflows. Its sums are short, at most 64 terms, so the rounding hardly shows.
We only ran this model in 16 bits; we did not train it that way. When large models are trained in 16 bits, the rounding can show. Micikevicius and colleagues needed three fixes to stop it.
Goodnight
Small inputs, big sums: the marks you can afford for the parts, and the marks you need for the total.
Next, at the prism: one glint of light that 16 bits could not hold precisely enough. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's precision measurements, computed for this film. media/school/precision/facts.json, written by tools/school/fill_precision.mjs from system/school/precision*.mjs: exact fp16 and bf16 rounding checked on every bit pattern; spacing and error by magnitude; long sums; the attention model's layers and the whole model run with 16-bit storage and 32-bit or 16-bit accumulation, scored on 1,024 held-out rows with paired intervals. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/precision
Used for: Near 110: fp16 spacing 0.0625, worst rounding 0.03125; 13 significand bits needed for 1e-4, fp16 has 11. Adding 1 stalls at 2,048 in fp16 and 256 in bf16. Whole-model loss changes under 1e-3 nats in every 16-bit mode.
Does not prove: Anything about large models trained in 16 bits.
IEEE 754-2019 binary16. IEEE (2019). IEEE Standard for Floating-Point Arithmetic, IEEE 754-2019 (binary16 parameters, checked against ISO/IEC TS 18661-3). https://standards.ieee.org/ieee/754/6210/
Used for: binary16: 11-bit significand, 5-bit exponent, largest finite 65,504.
Does not prove: Anything about bf16.
Micikevicius and others 2018, Mixed Precision Training. Micikevicius, P., Narang, S., Alben, J., and others (2018). Mixed Precision Training. ICLR 2018. arXiv:1710.03740. https://arxiv.org/abs/1710.03740
Used for: Train in fp16 with an fp32 master copy of weights, loss scaling, and fp32 accumulation of products.
Does not prove: Anything about this model.
NVIDIA CUDA Programming Guide, tensor core element types. NVIDIA (2026). CUDA Programming Guide 13.4.2, section 5.4.11.6, Element Types and Matrix Sizes. https://docs.nvidia.com/cuda/cuda-programming-guide/
Used for: Tensor core operations take fp16 inputs with fp16 or fp32 accumulators, and bf16 inputs with fp32 accumulators.
Does not prove: Anything about other vendors.
raw-native evidence note on the denoiser's tensor-core parity, 15 October 2026. HarperZ9/raw-native, evidence/rr-tc-note-2026-10-15.json. https://github.com/HarperZ9/raw-native/blob/main/evidence/rr-tc-note-2026-10-15.json
Used for: A value near 110 needs about 1e-4 relative precision; fp16 holds about 1e-3 there (order of magnitude).
Does not prove: Anything about this film's arithmetic beyond the stated figures.
What this lesson leaves out
The 16-bit formats are emulated by rounding in the main processor; no graphics chip and no tensor core ran these numbers.
One small model; its scores are small enough that fp16 never overflows here, which large models cannot count on.
Scores on 1,024 held-out rows; differences from 32 bits are paired on the same rows.
Before this: Fused attention (K23). Earlier: Quantisation (S18).