How many of the 164 tasks each model passed on its first try
Base Qwen 14B passed 141 of the 164 tasks and Flywheel 14B passed 136, a gap of 3.05 points that is not statistically significant.
Where the two models split
Each square is one task. Its left half is filled when Base Qwen 14B passed it, its right half when Flywheel 14B passed it.
- 127 both models passed
- 14 only Base Qwen 14B passed
- 9 only Flywheel 14B passed
- 14 both models failed
A gap this small can be chance: the two models split on 23 of the 164 tasks, 14 for Base Qwen 14B and 9 for Flywheel 14B. It covers one 164-task code-completion suite and does not measure agentic tool use or general coding quality.
How we know: the comparison
- Does not prove
- This result does not show a statistically significant improvement, general coding superiority, agentic reliability, production readiness, or independent validation.
- Limits
- The base artifact digest is not pinned in this result file. The result covers one 164-task code-completion suite and does not measure agentic tool use. Hardware, runtime version, latency, cost, and resource use are not recorded in this result file.
- Method
- Same 164 code-completion tasks, same harness, pass@1, greedy decoding, temperature 0. Each bar is the share of the 164 tasks a model passed on its first try, on a scale that starts at zero.
- Test
- Flywheel 14B was 3.05 percentage points lower. McNemar test on the paired outcomes: continuity-corrected χ²=0.696, p=0.404, significant at 0.05: no.
- Paired outcomes
- 9 gains; 14 regressions; 127 both pass; 14 both fail
- Tally
Text equivalent for both model results. Model role Model reference Passed / n Pass@1 Base Qwen 14B ollama:qwen2.5-coder:14b-instruct-q4_K_M 141/164 85.98% Flywheel 14B ollama:flywheel-local-coder-14b 136/164 82.93% - Source
Source SHA-256:
587da8ea4a04d5e520cc5ed4a83efb6c070fab6c587e0e58b00053cfdee5ed9d
Measured result, captured 28 August 2026