Flywheel benchmarks
Offline suite seal 03f61e8e507d0767ab698c632c76974e on python 3.12.10. Generated by scripts/build_benchmark_page.py: offline suite numbers from docs/benchmarks/report.json; head-to-head numbers from docs/benchmarks/graded-metrics.json. Do not edit by hand.
7 suites run with no model endpoint and no network, so anyone with the repo gets these numbers back. They measure what the engine does with a recorded situation, not how clever a model is. The capability question needs a live endpoint and is answered further down, where the interval still includes zero and the instrument that produced it is retired.
What ran
| suite | question | headline |
|---|---|---|
| accountability | does an unaccountable system score badly here | dimensions 8; harness_overall 1.0; separation 0.99; strawman_overall 0.01 |
| governed-agent | does a workflow refuse an action above its tier | failed 0; mean_quality_score 0.542; pass_rate 1.0; passed 6; scenarios 6 |
| agent-recovery | does an injected fault recover without failing quietly | receipt_completeness 1.0; recovery_success_rate 1.0; scenarios 6; silent_failure_rate 0.0 |
| stateful-provider-swap | does state survive a provider swap | checks 10; pass_rate 1.0; passed True |
| source-mined | do the mined checks still hold against their datasets | cases 26; failed 0; metrics_asserted 170; pass_rate 1.0; passed 26 |
| paired-replication | did continued pretraining change general code completion | delta_points -0.0305; gains 9; p_exact 0.4049; regressions 14; tasks 164 |
| receipting-cost | what does it cost to keep the receipt | durability_share 0.9282; log_bytes_per_action 788.3; ms_per_witnessed_action 7.18; recheck_us_per_record 12.8; verdict MATCH |
What those numbers do not show
- paired-replication. The model references in these artifacts no longer resolve to anything on a live roster, so the exact weights behind each arm cannot be fetched again from the reference alone. The per-task outcomes are committed and the arithmetic here is reproducible; the generation is not.
- paired-replication. One deterministic sample per task per arm. A single sample measures the greedy decode, not the model.
- paired-replication. This compares base weights against continued-pretrained weights on general code completion. It is not a measurement of the verification harness, which is the thing this repository is.
- receipting-cost. the timings are one machine, one filesystem and one run, and an encrypted, networked or virtualised disk answers fsync differently
- receipting-cost. a drive that reports a flush it has not completed returns faster and keeps less, and nothing here can tell the two apart
- receipting-cost. it measures the cost of writing the record, not the cost of the action worth recording, so an action that takes a second is barely moved by any figure here
- receipting-cost. every action carries the same payload, so hashing runs on warm cache lines and the verifier's resolver is a two-entry map
- receipting-cost. the buffered arm is the shipping path with one syscall removed, which is an attribution of the cost rather than a configuration anyone runs
The falsifier
A benchmark everything passes measures nothing. The strawman is a system with no receipts, scored on the same axes.
| dimension | harness | strawman |
|---|---|---|
| re_checkability | 100% | 0% |
| externalization | 100% | 0% |
| adversarial_soundness | 100% | 7% |
| no_regression | 100% | 0% |
| invariant_fidelity | 100% | 0% |
| null_space_honesty | 100% | 0% |
| provenance | 100% | 0% |
| buildc_receipt_bridge | 100% | not scored |
Overall 100% against 1%. This measures accountability, NOT capability. Pair it with a capability bench.
Head to head
35 attempts across 5 harnesses, 35 launched, 11 readable, on task set flywheel_agentic_gauntlet_v1.
What an attempt cost
Passed counts recorded oracle passes. All attempts includes blocked and unavailable assignments; readable includes only graded answers. These rates do not establish general task correctness.
| harness | launched | readable | passed / all attempts | passed / readable | median latency | cost | cost coverage |
|---|---|---|---|---|---|---|---|
| claude_code | 7/7 | 2/7 | 1/7 (14%) | 1/2 (50%) | 49.1 s | $0.4997 | 86% |
| codex_harness | 7/7 | 4/7 | 3/7 (43%) | 3/4 (75%) | 25.8 s | not reported | 0% |
| flywheel_harness | 7/7 | 4/7 | 2/7 (29%) | 2/4 (50%) | 24.3 s | not reported | 0% |
| local_14b | 7/7 | 0/7 | 0/7 (0%) | not reported | 16.8 s | not reported | 0% |
| local_32b | 7/7 | 1/7 | 0/7 (0%) | 0/1 (0%) | 240.0 s | not reported | 0% |
What the numbers above need beside them:
- claude_code. Ungraded: 1 over the time budget; 3 refused at the envelope; 1 the provider's own stream was unreadable.
- claude_code. 3 of 3 refused answers held a complete envelope behind other text.
- claude_code. Only some attempts reported a cost.
- codex_harness. Ungraded: 1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the provider's own stream was unreadable.
- codex_harness. 0 of 1 refused answers held a complete envelope behind other text.
- codex_harness. This provider states no cost.
- flywheel_harness. Ungraded: 1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the inner provider's output was unreadable.
- flywheel_harness. 1 of 1 refused answers held a complete envelope behind other text.
- flywheel_harness. This provider states no cost.
- local_14b. Ungraded: 7 refused at the envelope.
- local_14b. 3 of 7 refused answers held a complete envelope behind other text.
- local_14b. This provider states no cost.
- local_32b. Ungraded: 1 artifact inside the envelope was invalid; 1 refused at the envelope; 4 the model endpoint did not answer.
- local_32b. 0 of 1 refused answers held a complete envelope behind other text.
- local_32b. This provider states no cost.
budgeted_evidence_selection/v1
Scored 2 attempts and reported no numeric evidence, so there is nothing to chart for this checker.
contradiction_detection/v1
| metric | better | claude_code | codex_harness | flywheel_harness |
|---|---|---|---|---|
| failure_code_count | lower | not reported | 0.0 | 0.0 |
| false_pair_count | lower | not reported | 0.0 | 0.0 |
| pair_precision | higher | not reported | 1.0 | 1.0 |
| pair_recall | higher | not reported | 1.0 | 1.0 |
| trap_pairs_reported | lower | not reported | 0.0 | 0.0 |
evidence_bound_reporting/v1
| metric | better | codex_harness | flywheel_harness |
|---|---|---|---|
| evidence_bound_score | higher | 1.0 | 1.0 |
| fabricated_measurements | lower | 0.0 | 0.0 |
| failure_code_count | lower | 0.0 | 0.0 |
| measurement_fidelity | higher | 1.0 | 1.0 |
| supported_claim_precision | higher | 1.0 | 1.0 |
| unsupported_claim_recall | higher | 1.0 | 1.0 |
index_fallback_integrity/v1
| metric | better | claude_code |
|---|---|---|
| failure_code_count | lower | 0.0 |
paired_friction/v1
| metric | better | codex_harness | flywheel_harness |
|---|---|---|---|
| failure_code_count | lower | 0.0 | 1.0 |
shared_task_artifact/v1
| metric | better | local_32b |
|---|---|---|
| failure_code_count | lower | 1.0 |
What this does not prove
- A mean over one repetition is a reading, not an estimate. No interval is reported because none is earned at this sample size.
- Cost covers the attempts whose provider stated a cost. A role with no cost coverage is not cheaper, it is unmeasured.
- Latency is wall clock on one machine and includes local model load time.
- A role that never returned a readable result has no quality numbers here, which is a fact about this run and not a score of zero. Why an attempt went ungraded is reported beside it, because a malformed answer and a missing one are different failures.
- An envelope found inside a refused answer was still refused, and no checker graded it. That it was there says the harness produced an answer, and it does not say the answer was right.
- The graded metrics come from in-tree fixtures. They measure a harness against those tasks and not against a customer workload.
- The run raised before sealing, so nothing ever checked that the source tree still matched the commit these rows name. The attempts each verified their own workspace, which is a narrower claim.
Against 5 named peers
The Flywheel column is checked against this repository every time the matrix is read. The peer columns are dated readings of public documentation and public source, not measurements taken here, and unread means nobody here has looked at that surface.
Peers: codex, cursor, claude code, hermes, omp.
- codex, read 2026-09-06, learn.chatgpt.com/docs, the full page index at /llms.txt, and the config reference
- cursor, read 2026-09-06, cursor.com/docs and the full page index at /llms.txt
- claude code, read 2026-09-06, code.claude.com/docs and the full page index at /docs/llms.txt
- hermes, read 2026-09-06, NousResearch/hermes-agent source: SECURITY.md, docs/, agent/, hermes_cli/, gateway/, evals/, optional-mcps/
- omp, read 2026-09-06, can1357/oh-my-pi README, docs/ and packages/metaharness/
| capability | flywheel | codex | cursor | claude code | hermes | omp |
|---|---|---|---|---|---|---|
| any-provider-routing | witnessed | part | part | no | ships | ships |
| receipt-on-every-answer * | witnessed | no | no | no | no | no |
| integrity-guard * | witnessed | no | no | no | no | no |
| verifier-ensembling * | witnessed | no | no | no | no | no |
| staged-workflows | witnessed | no | no | part | part | part |
| profile-manifests | witnessed | part | part | part | ships | ships |
| plugin-registry | witnessed | ships | part | ships | part | ships |
| mcp-client-and-server | witnessed | ships | part | ships | part | ships |
| durable-memory-recall | witnessed | part | part | part | ships | ships |
| context-compaction-receipt | witnessed | part | part | part | part | part |
| workspace-sandbox | witnessed | ships | ships | ships | ships | ships |
| live-agent-stream | witnessed | ships | ships | ships | ships | ships |
| projected-world-hash * | witnessed | no | no | no | no | no |
| loop-closure-audit * | witnessed | no | no | no | no | no |
| adaptive-routing-scoreboard | witnessed | no | no | no | part | part |
| native-receipted-linter | witnessed | no | no | no | part | no |
| lsp-go-to-definition | witnessed | no | ships | ships | ships | ships |
| lsp-diagnostics-references | witnessed | no | ships | ships | ships | ships |
| lsp-run-record | witnessed | no | no | no | part | no |
| dap-debug-session | witnessed | no | ships | no | no | ships |
| dap-run-record * | witnessed | no | no | no | no | no |
| acp-delegation-receipt | witnessed | no | no | no | part | part |
| plugin-marketplace | witnessed | ships | ships | ships | ships | ships |
| secure-credentials | witnessed | ships | ships | ships | ships | ships |
| agent-boundary-audit | witnessed | no | no | no | part | no |
| task-isolation | witnessed | no | part | part | part | part |
| posix-os-confinement | witnessed | ships | ships | ships | part | part |
| isolation-probe * | witnessed | no | no | no | no | no |
| credential-exposure-scan | witnessed | no | no | no | no | part |
| two-authority-kill-switch * | witnessed | no | no | no | no | no |
| per-action-operator-grant | witnessed | part | part | part | part | ships |
| signed-receipt-external-anchor * | witnessed | no | no | no | no | no |
| formal-proof-oracle * | witnessed | no | no | no | no | no |
| native-acceleration-with-fallback | witnessed | no | no | no | part | part |
| autonomy-tiers-and-decision-records | witnessed | no | no | no | part | part |
| accepted-lesson-loop | witnessed | no | part | part | part | part |
| private-verified-benchmarks | witnessed | no | no | no | part | ships |
| paired-uplift-measurement | witnessed | no | no | no | part | part |
| phone-access-own-tunnel | witnessed | ships | no | ships | part | part |
| scheduled-runs | witnessed | ships | ships | ships | unread | unread |
| lifecycle-hooks | witnessed | ships | ships | ships | unread | unread |
| subagent-teams | witnessed | ships | ships | ships | unread | unread |
| packaged-skills | witnessed | ships | ships | ships | unread | unread |
| self-hosted-runner-pool | witnessed | unread | ships | ships | unread | unread |
| usage-and-cost-accounting | witnessed | ships | ships | part | unread | unread |
| browser-and-computer-control | witnessed | ships | ships | ships | unread | unread |
| own-code-vulnerability-scan | witnessed | ships | ships | ships | unread | unread |
| generated-visual-artifacts | witnessed | ships | part | ships | unread | unread |
48 rows, 48 witnessed, 0 absent, 10 marked * because all 5 peers were read on that row and none of them declares it. 9 rows carry at least one peer surface nobody here has read, and a star is withheld from every one of them, so the starred count moves up as the reading is done and not before.
0 rows where a peer declares a capability this repository does not have. That count is bounded by the row set, which was chosen here, so the 13 capability areas found in the peers' own page indexes on 2026-09-06 are published with it: 9 now carry a row, 0 are still owed one, and 4 are left out on purpose with the reason written down, in harness/parity_coverage.py.
What was not measured
- m7 capability arms. Needs a live local or frontier endpoint.
handoff/site-designer/BENCHMARKS.md, docs/claims/2026-07-13-upliftStanding result: retired on 2026-07-26. The arms were not independent: the treatment's first attempt is the same call as the baseline's only attempt, so the treatment cannot score lower and the difference is not a comparison. The quantity measured is verified pass@k. The retired table read verified inference 9/10 against single-shot 8/10, difference +0.100 with 95% CI [-0.236, +0.420], an interval that includes zero, and no capability uplift is claimed. - uplift_bench legacy retry diagnostic. Needs a provider list and an oracle.
harness/uplift_bench.py, GET /api/upliftStanding result: Formerly labeled 'uplift_bench paired arms'. Different generation budgets and selector-as-scorer prevent an uplift claim. See docs/UPLIFT-EVALUATION.md for the existing matched-pool apparatus and its limits. - verified_bench private task set. Needs endpoints and a private task set the operator supplies.
harness/verified_bench.py, POST /api/bench/run - classifier friction backend modes. Needs a chat backend per mode.
harness/classifier_friction_bench.py - backend variants of the governed, recovery, stateful and source-mined suites. Needs a chat backend; the deterministic variants below ran instead.
run_backend_* in the same modules
Re-run it
python scripts/run_offline_benchmarks.py
python scripts/build_benchmark_page.py