What the six suites scored
Five suites scored 100 percent on small test sets, where the true rate could be as low as 61 percent. Flywheel Local Coder 14B scored 3.05 points below its base model.
- 95% interval: the range likely to hold the true score, with its low end named
Score on the suite's own tests
Accountability
Does an unaccountable system get a low score here? 8 dimensions, mean score
100%
Governed agent
Does a workflow refuse an action above its allowed level? 6 scenarios
100%
Agent recovery
After an injected fault, does the agent recover without failing quietly? 6 scenarios
100%
Provider swap
Does saved state survive a switch to another model provider? 10 checks
100%
Source-mined checks
Do checks drawn from source datasets still hold against those datasets? 26 cases
100%
The one comparison between two models
Change in pass rate, in points
Flywheel Local Coder 14B against its base model
Did further training change how well the model completes code? 164 tasks
3.05 points lower
A gap this small can be chance: of 164 tasks, 9 passed only on Flywheel Local Coder 14B and 14 passed only on the base model.
How we know: the chart
- Does not prove
- This record does not rank Flywheel against any other coding harness. No other harness was run to produce a single entry in the capability list, and the entries for the other tools are a dated reading of public documentation. The one comparison the record does contain is negative and sits inside its own noise.
- Limits
- The five 100 percent scores come from small test sets of 6 to 26 items each, and the capped lines show how wide that leaves each interval. Each suite runs offline, so none of these scores measures capability against a frontier model.
- Method
- Every score and count is read out of the sealed record. The capped line is a Wilson 95 percent interval, computed for this page from each suite's pass count and denominator. Accountability reports a mean score across its dimensions, so it gets no interval. The comparison sits on its own signed axis, because a drop drawn on a 0 to 100 track would look like a partial success. The model comparison re-reads per-task results the two models produced earlier. Its exact McNemar (binomial) test gives p = 0.40, and the continuity-corrected test on the model comparison page gives 0.404: 9 tasks passed only after continued pretraining and 14 passed only on the base model. The record holds no interval for that difference, so none is drawn.
- Tally
Each suite with its size, result and interval. Suite Size Result 95% interval Accountability 8 dimensions 100% None, because this is a mean score Governed agent 6 scenarios 100% 61 to 100% Agent recovery 6 scenarios 100% 61 to 100% Provider swap 10 checks 100% 72 to 100% Source-mined checks 26 cases 100% 87 to 100% Flywheel Local Coder 14B against its base model 164 tasks 3.05 points lower None in the record - What each suite asks
- Accountability. Does an unaccountable system get a low score here? This suite scores accountability only. Pair it with a capability benchmark.
- Governed agent. Does a workflow refuse an action above its allowed level?
- Agent recovery. After an injected fault, does the agent recover without failing quietly?
- Provider swap. Does saved state survive a switch to another model provider?
- Source-mined checks. Do checks drawn from source datasets still hold against those datasets?
- Flywheel Local Coder 14B against its base model. Did further training change how well the model completes code? Continued pretraining on the workspace corpus brought no improvement in general code completion and no significant harm. The point estimate is a small drop inside the noise. The model names in these results no longer point to any model on a current roster, so the exact weights behind each arm cannot be fetched again from the name alone. The per-task outcomes are committed, so anyone can redo the arithmetic, but the answers cannot be generated again. Each arm ran one greedy sample per task. One sample shows the greedy output only, so it does not measure the model's range of answers. This compares base weights with continued-pretrained weights on general code completion. It does not measure Flywheel's checking harness, which is what this repository builds.
Record dated 3 September 2026