Decoding. Turning guesses into sentences.

Decoding

Decoding. Turning guesses into sentences.

In the lesson on next-word prediction, the model's guess was a spread of belief over every word. A sentence needs one word at a time. Tonight, how the choosing is done, and what each way of choosing does.

Always the widest

The simplest way: always take the widest slot. That is called greedy decoding.

Start the model on pylon and let it run. It writes: pylon found a kettle. Pylon put a cone on the kettle. Pylon found a kettle. And again, until the page runs out.

Why the kettle every time? After found a, the rules choose among 10 things equally, but the model leans a little: kettle gets 14 percent. A small lean, taken every time, becomes the only story it tells.

Spin the wheel

The other way: spin a wheel where each word's slot is as wide as its share, and take whatever comes up.

We spun it 1,500 times from the same start. Those runs wrote 1,482 different texts, and 89 percent of the runs could have been written by the rules.

Our check for that is strict. It parses each run with the rules themselves, including the long-range ones: the cone goes on the thing that was found, and the answer to who names the guide who did it.

Temperature

Before the wheel is cut, the scores can be divided by a number called the temperature. Below 1, the wide slots get wider and the thin ones shrink. Above 1, the shares move toward even.

After tallow found a, the kettle's slot is 20 percent of the wheel at temperature 0.5, and 11 percent at temperature 2, where words that make no sense there get bigger slots too.

Turn the temperature up to 2. What share of runs will still follow the rules? (the film pauses here)

At temperature 0.5, 98 percent of runs follow the rules. At 1, 89. At 1.5, 51. At 2, only 11.

And at temperature close to zero, the widest slot takes the whole wheel. In 200 spins at 0.01, every run was the greedy one.

Top-k: only the widest few

Another fix cuts the wheel down to the k widest slots before spinning. Fan and colleagues used ten for writing stories in 2018.

With k at 3, after tallow found a, the wheel keeps 3 slots: kettle, map and crate. The other things that could be found are never written there at all.

The catch is that k never changes. Where only two words make sense, three slots is one too many. Where ten make sense, three is too few.

Top-p: as many as it takes

Holtzman and colleagues, in 2020, cut the wheel by share instead: keep the widest slots until together they reach p, and drop the rest. Where the model is sure, that is one or two words. Where it is torn, it is many.

With p at 0.9, after tallow found a, the wheel keeps 9 slots. Runs that follow the rules: 94 percent.

Push p down to 0.5 and nearly every run follows the rules, 99.7 percent. But its 1,500 runs wrote only 210 different texts. Safe, and samey.

Nearly every run follows the rules. What has it given up? (the film pauses here)

What we did not find

That paper also found that plain spinning, at temperature 1, drifts into nonsense in big models. Thousands of unlikely words each get a sliver of the wheel, and together the slivers add up to far too much.

Ours shows a small version. At temperature 1, 4 percent of runs contain a word pair it never saw in training. Top-p at 0.9 lets none through, and rule-following rises from 89 to 94 percent.

The paper's other warning sign does not show here: scoring its own runs, the model is torn between 1.87 words per step, and scoring the rules' own runs, 1.92. With only 46 words, the tail is short. The full problem needs a bigger model to see.

Goodnight

Greedy tells one story forever. Spinning tells many. Temperature, top-k and top-p decide how wide the wheel's slots are before it spins.

Next in the lighthouse: how to tell whether a language model is any good. Goodnight.