The film
About 4:52, hosted by Tallow. It asks you one question and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.
Nothing loads until you press play.
The lab: the palette
Pick the bits, the scale and the rounding; the page quantises the model, scores it on held-out rows against the full one and repaints one table. In the film, 4 bits with one scale per output raised the loss by 0.0078 by plain rounding and 0.0028 by GPTQ, at 18% of the size.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
Quantisation
Quantisation. Fewer bits per number.
Every number in our model is stored in 32 bits. Big models have billions of numbers, so the memory adds up fast. Tonight Tallow stores the weights in fewer bits and measures what the model loses.
A smaller palette
Think of a painter with fewer paints. With 255 shades a picture looks smooth. With 15, you start to see bands. With 3, you see mostly bands.
Quantising a table of weights is the same move. Pick a scale, divide every number by it, and round to the nearest of a few levels. At 4 bits our scheme has 15 levels; at 2 bits, only 3.
To use the table again, multiply back by the scale. Every number comes back to its nearest level, off by up to half a step.
What it saves
Our model takes 83,456 bytes in 32-bit numbers. At 4 bits, with one scale for each output of each table, it takes 15,376: 18 percent of the size.
What it costs
Score every version on the same 1,024 held-out rows. The full model's loss is 0.6676.
At 8 bits per number, one scale per output: 0.6677. At 6: 0.6685. At 4: 0.6754. At 3: 0.7249. At 2 bits: 3.48, and the model is broken.
Four bits instead of thirty-two. How much worse is the loss? (the film pauses here)
With one scale for each whole table instead of each output, it breaks sooner: 1.59 already at 3 bits.
Where it hurts
One scale has to cover a table's biggest number. If a few numbers are much bigger than the rest, the steps get wide, and every small number is rounded coarsely.
Quantise one table at a time to 4 bits, with one scale for the whole table, and see which one hurts. The readout, the table that turns the model's last list of numbers into word scores, costs the most: 0.0276 nats on its own.
Tim Dettmers and colleagues found in 2022 that in large models, once the activations are stored in 8 bits too, the trouble comes from a few huge values in them. Activations are the numbers flowing between tables. We left those in full precision here, so that part of the story does not show.
Rounding with care
Rounding each number on its own throws away the fact that they work together. GPTQ, from Elias Frantar and colleagues in 2023, rounds the numbers of a table one input column at a time and nudges the ones not yet rounded to make up for the error. It uses a small sample of text to see which errors matter.
At 4 bits with one scale per output, plain rounding raises the loss by 0.0078. GPTQ raises it by 0.0028, roughly a third as much.
It is not magic. At 2 bits with one scale per table, both versions are broken, and GPTQ comes out worse.
Goodnight
Fewer bits per number is how large models fit on smaller machines. Our model has only 83,456 bytes of numbers, and large ones behave differently, so how few bits you can get away with has to be measured on the model you have.
Next, in the cannery: the sums these tables are used for, and how a chip does them fast. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's quantisation runs, computed for this film. media/school/quant/facts.json, written by tools/school/fill_quant.mjs from system/school/quant.mjs and quant-gptq.mjs: the attention model's weights quantised by round-to-nearest and GPTQ at 8, 6, 4, 3 and 2 bits, per tensor, per output channel and per group, scored on 1,024 held-out rows with bootstrap intervals; one tensor at a time; controls. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/quant
Used for: Float loss 0.6676; per channel round-to-nearest 8/6/4/3/2 bits: 0.668, 0.668, 0.675, 0.725, 3.483. 4 bits per channel: 15,376 bytes (18 percent); rise +0.0078 (round-to-nearest) and +0.0028 (GPTQ). Readout alone at 4 bits: +0.0276.
Does not prove: Anything about large models; activations stay in float.
Jacob and others 2018, Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Jacob, B., Kligys, S., Chen, B., and others (2018). CVPR 2018. arXiv:1712.05877. https://arxiv.org/abs/1712.05877
Used for: Real values are represented by an integer and a scale (and zero point): r = S (q - Z).
Does not prove: Anything about this model.
Dettmers and others 2022, LLM.int8(). Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. arXiv:2208.07339. https://arxiv.org/abs/2208.07339
Used for: Large-magnitude outlier features in activations emerge in large transformers and break naive 8-bit quantisation.
Does not prove: Anything about this model.
Frantar and others 2023, GPTQ. Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. arXiv:2210.17323. https://arxiv.org/abs/2210.17323
Used for: Quantises weights column by column, updating the not-yet-quantised weights to compensate, using second-order information from calibration data.
Does not prove: Anything about this model.
What this lesson leaves out
One model of 83,456 bytes; large models behave differently, and their activation outliers do not show here because activations stay in full precision.
Symmetric round-to-nearest without a zero point, and GPTQ with a symmetric grid and calibration text from the lesson's own generator.
Scores on 1,024 held-out rows; differences from the full model are paired on the same rows.
Before this: Threads, workgroups and memory (K21). Next, in the cannery: Matmul tiling (K22).