Distillation. A big teacher, a small student.

Distillation

Distillation. A big teacher, a small student.

A big model is slow and costly to run. One way to get a small one is to train it on what the big one says, instead of on the text alone. Tonight the attention model teaches a student a fifth its size, and we find out when that helps.

Two kinds of answer

After "rerun knocked", the rules that write the guides' sentences say "on" half the time and "twice" the other half.

The text itself only ever shows one. A sentence says "on", and that is the answer the model is marked against: a hard label, one word, as if the other had been impossible. Only across many sentences does the split show.

The teacher's own guess keeps the split: 54 percent "twice", 45 percent "on". That is a soft target, the whole wheel instead of one slot.

Open choices like this are common. On 40 percent of the words in our held-out text, the likeliest next word has no more than three quarters of the odds.

Turning up the temperature

Hinton, Vinyals and Dean, in 2015, added one more trick. Before the teacher's scores become shares, divide them by a number they called the temperature.

At a temperature of 1 the wheel is the teacher's own. At 2 and 4 the big slots shrink and the slivers grow, so the student hears more about the words the teacher thinks are unlikely, and how unlikely.

The student matches the softened wheel with its own wheel, softened the same way. The paper multiplies that loss by the temperature squared, so the push stays about the same size whatever the temperature.

Our student has 4,112 numbers, 20 percent of the teacher's 20,864: one floor instead of two, and lists of 16 numbers per word instead of 32.

With plenty of text

First, give the student all the fresh text it wants: 2,000 steps, new sentences every step, three students of each kind with different seeds.

With all the text it wants, does the student learn more from the teacher's wheel than from the text's own words? (the film pauses here)

Trained on the text's own words, the student's held-out loss is 0.72. On the teacher's wheel: 0.70 at temperature 1, 0.74 at 2, 0.72 at 4. None of those differences holds up across the three seeds. Only temperature 8 is clearly different, and it is worse, by 0.04. Some students never learned to look back, whichever way they were taught, and those runs swing the averages.

So with plenty of text, we saw no gain from the teacher that three seeds could confirm. The teacher did make the student more like itself: at temperature 2, students taught by the wheel pick the teacher's top word 92 percent of the time, against 83 for students taught by the text.

With very little text

Hinton and his colleagues made a stronger claim: soft targets carry what the teacher learned to a student that sees very little data.

So give the student only 32 rows of 24 words, and show it the same rows over and over, 1,000 times each.

Now the student gets only 32 rows, over and over. The text's words, or the teacher's wheel? (the film pauses here)

Trained on those rows' own words, the student learns them by heart. Its held-out loss was 1.72 at its best check and 4.80 by the end. On the teacher's wheel for the same rows, its best check was 1.39, and it ended at 1.64. Best against best, every seed agrees.

With 256 rows, the text's words give 0.90 at best and 1.28 by the end, and the teacher's wheel 0.73, close to the teacher's own 0.69.

One caution. The teacher learned from far more text than these rows. What crosses over is what the teacher already knew, carried in its wheel.

Two teachers that fail

A teacher can fail in two ways. Take only the teacher's top word, and train on it as if it were certain. The student copies look-back words well, 95 percent right, but its held-out loss is 3.84, worse than just counting how often each word appears, 3.27. Where the text could go more than one way, it is sure of one word, and pays heavily whenever the text picks another.

Or take a teacher that never trained. Its wheel is close to even, and its student ends at 3.84, worse than counting words too.

And a perfect answer key, the rules' own odds, gives 0.69 with plenty of text, about what the teacher itself scores. It beat the text alone in all three seeds, but by too little for three seeds to settle.

Goodnight

A small model can learn a lot from how a big one hesitates, most of all when there is little else to learn from. For our student, with text that never ran out, the text was enough. With a fixed training set, Hinton's team found a gain even then.

Next in the boatyard: teaching a trained model a new job by fitting a small rudder, instead of rebuilding the hull. Goodnight.