Zain Dana Harpernight school / S20

Our MMBU Challenge entry. The logbook, misses included.

For the MMBU Challenge we fine-tuned two small open models on 86 disclosed sources and wrote down every bar before the result it judged. Tallow opens the logbook: the runs that missed, the rule that turned away its own champion, a score that flipped sign when the answer key changed, and the gap in our own record.

The film

About 7:15, hosted by Tallow. It asks you one question and waits for your answer before it shows you. The guide's voice is not recorded yet, so for now the film plays with captions; the full words are also below.

Nothing loads until you press play.

The lab: redo the logbook's intervals

Pick a comparison and the lab reruns the scorer's bootstrap on the published per-row scores, in this page, with numpy's own random stream ported to JavaScript. With the scorer's seed, 8 intervals come out to the logbook's printed digits, among them the candidacy bar, +0.0425 [0.0354, 0.0498], and both readings of the held-out context score. Another seed moves the bounds in the third or fourth decimal.

The scores are loaded from this site and resampled on your computer. Nothing is sent anywhere.

Everything the film says

Our MMBU Challenge entry

Case study: our MMBU Challenge entry.

For two weeks this boatyard fine-tuned a small model for a public contest, writing down every bar before the result it would judge.

The contest

MMBU is a biomedical image benchmark from Stanford MARVL, built from 410 public datasets. A model answers each question about an image once, with no tools, and is scored on the answer and on context, such as what kind of image it is.

We trained small LoRA strips on two open Qwen 3.5 models, of 2.3 and 4.7 billion numbers, from 86 sources across all the runs, every one listed in our disclosure. This is benchmark work, not a medical tool, and every score tonight is from our own scorers, not the contest's judge.

The bar comes first

Before each result, we wrote down the comparison that would decide it and the bar it had to clear: usually, new model minus old on about 3,100 practice questions, with a bootstrap interval whose lower end sits above zero.

The fourth run added 20 public datasets, 17 of them ones we believe the benchmark draws on.

Seventeen of the benchmark's likely sources. Did the practice answers go up? (the film pauses here)

A miss, kept

No. The difference was -0.007, interval -0.018 to 0.004. Zero sits inside, so by the rule those rows did not go forward.

The fifth run gave the champion a second pass over its own rows: up 0.001, interval -0.007 to 0.009. Another miss, written down the same way a win is.

Between a guess and a cheat

Then a worrying number. The champion got 21.6 percent of the practice answers right. A guesser who could see the judge's options, which our model never sees, would average 20.7. A cheat giving each kind of question its most common answer, 23.7.

Was it even looking? Given blank pictures, the fourth run fell from 20.9 percent right to 8.2. It uses the picture. And 51 percent of its answers named none of the options at all.

Better at picking than at saying

So, for our eyes only, we showed the fifth run's model each option and measured how likely it found each one. Its favourite option was right 41 percent of the time; its written answer, 20.

Picking from a list is easier than writing, but much of the loss is in the words it writes. The rule chose the next step, training on the labels' own words, and on region questions the sixth run beat the fifth by 0.026, interval 0.008 to 0.044.

The rule that turned away its own champion

The sixth run cleared every answer bar but missed a format bar: valid replies 99.8 percent of the time, against a bar of 99.9. The champion it would replace scored only 99.6. As written, the bar turned away its own champion.

The verdict stood. For later runs we signed a new rule, while the seventh run was at step 20: replace the champion when every answer bar is met and the format is no worse. The seventh run met it, up 0.043 on our size-adjusted score, interval 0.035 to 0.050.

Whose words count

One number went the wrong way. On our held-out sources, the seventh run's context score fell 0.089 below the sixth run's.

It now answered in the benchmark paper's words, X-ray rather than X-ray imaging, while our answer key kept each source's old words. With the key rewritten in the paper's words, the same replies scored 0.152 higher. Neither key is neutral, and this re-score came after we saw the fall: a measurement, not a test.

Keeping the lamp lit

Each run takes a lock on the shared graphics card, saves every 50 steps, and has a watchdog. A shutdown mid-run lost nothing. Code that had never run idled the card for about 10 hours, and two trainers on one run cost 40 steps; now, when a check cannot tell if a trainer is alive, it counts it as alive.

What the logbook can prove

Every bar was fingerprinted into a ledger, and all 19 rule files still match it, 16 by full fingerprint and 3 by the short prefix it kept. But the ledger was meant to be pushed to GitHub, which stamps the time, before each result. Only entries up to 27 were, the last on 4 October.

Entries 28 to 89, every run tonight, were first pushed on 18 October, after the results; the ledger is now public. So the claim that each bar came before its score rests on our word. A fingerprint proves the bytes, not the hour.

Goodnight

The seventh run is the candidate; nothing has been submitted. The eighth stopped at step 500 of 1,918 when the author paused the work. The contest runs to 31 December 2026.

In the lab, redo eight of the logbook's intervals, digit for digit. Goodnight.

Sources

Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json, published.json, dev_per_row.json, heldout_context_per_row.json, heldout_task_per_row.json.

What this lesson leaves out

Every score here comes from our own local scorers. The contest's judge was never called, and dev scores will not match the final leaderboard. Nothing has been submitted.

This is benchmark work. The images are benchmark data, and nothing here is a medical tool or a clinical claim.

The ledger's fingerprints still match the rule files, but only entries up to 27 were pushed to GitHub before the results, the last on 2026-10-04; entries 28 to 89, which cover every run in the film, were first pushed on 2026-10-18, after the results, and the ledger is now public. So the claim that each bar came before its result has no outside timestamp before that date for those runs.

The held-out vocabulary re-score (FT7 minus FT6: -0.08946 under the sources' words, +0.15225 under the paper's) was made after the fall was seen, so for these runs it is a measurement, not a test. The eighth run's reading was preregistered, but that run was paused before it was scored.

MMBU work is paused: the eighth run stopped at step 500 of 1,918 when the author paused it on 16 October 2026. The contest runs to 31 December 2026.

The lab re-derives the arithmetic from the published per-row scores. It does not check that those scores are right; they come from our scorers.

Before this: LoRA and fine-tuning (S19), Evaluating a language model (L16) and When the check can be fooled (E24). The same lesson about fingerprints, from the other side: A score that carries its receipt (E25).