Error bars. How sure is the number.

Error bars

Error bars. How sure is the number.

Every lesson so far has quoted numbers, a loss here, a share of right answers there. Each came from one set of test rows. Tonight Rerun asks how much those numbers would move if the rows had been different.

One number

Score the attention model on 1,024 held-out rows: a loss of 0.6676 and 73 percent of next words right.

Those rows were drawn at random. Another draw of the same size would give a slightly different loss. The question is how different.

Drawing again from what you have

Bradley Efron's answer, in 1979, was the bootstrap. You cannot draw more rows from the world, so draw from the rows you have: pick as many as you have, at random, allowing repeats, and score that pile. Do it thousands of times.

The scores spread out. We take the middle 95 percent of them as the interval.

With 64 rows the loss is 0.7000, and the interval runs from 0.6723 to 0.7282.

Four times as many rows. How much narrower does the interval get? (the film pauses here)

More rows, narrower bars

Its width is 0.0838 with 16 rows, 0.0559 with 64, 0.0311 with 256 and 0.0161 with 1,024.

From 16 to 64 rows it narrows by a third; after that, by nearly half each time. Averaged over a thousand fresh draws, 0.1014, 0.0523 and 0.0264, each fourfold step halves it, as the square root of the number of rows says it should. Four times the rows buys an interval half as wide.

Does it work?

An interval is only worth something if it catches the truth as often as it claims. Usually nobody knows the truth. Here we do: the rules that write the guides' sentences give an exact average loss for their own guesses, 0.6361.

So draw 1,000 fresh sets of 64 rows from the rules, score the rules' own guesses on each, build an interval, and count how many contain 0.6361. The answer: 93.1 percent. With 256 rows, 95.3. Close to the 95 promised, a little short when rows are few.

What to draw again

One choice matters more than it looks: what counts as one draw. Evan Miller's 2024 paper on error bars for evaluations warns that questions which come in clusters must be counted as clusters when the error is worked out, or the interval comes out too narrow.

Our rows are clusters too, but of a kind that steadies the score. Each holds one word at every position, so a row is a balanced sample of easy and hard places. Resampling single words instead gives intervals that are too wide here: they caught the truth 100 percent of the time.

Now make real clusters: 1,000 test sets of 16 rows, each row copied four times. Treated as 64 separate rows, the interval catches the truth only 66.9 percent of the time. Resampling the 16 distinct rows brings it back to 93.3.

Two models, same rows

In the distillation lesson, one student learned from the text and one from the teacher's odds. Score both on the same 1,024 rows. The student taught by the text has the lower loss, by 0.0045.

Resample the rows once for both models, keeping each row's two scores together, and the interval for the difference runs from -0.0081 to -0.0011: it leaves out zero. Resample each model on its own and the interval is 3.27 times as wide, and it contains zero.

Keeping the pairs cancels the rows that are hard for both. Shuffle the pairing and the advantage disappears.

The spread the rows cannot see

But that difference came from one training run of each student. Train each with five seeds, the run we just scored and four more, and score them all on the same rows.

For one student, the spread of loss across seeds is about 12 times the spread the bootstrap found from rows; for the other, about 10. Some runs never learned to look back, and those runs make most of it. Seed by seed, the difference between the two students was -0.0045, 0.0845, -0.0049, 0.0007 and 0.0064.

The first seed's interval was honest about its rows. It said nothing about the next seed.

Goodnight

A number without its spread is half a number. Ask what was drawn at random, the rows, the seeds or both, and draw again from that. Everything tonight came from one small model and a handful of runs; the method carries over, the numbers do not.

Next under the prism: what happens when training pushes a model toward the measure instead of the goal. Goodnight.