Cross-harness run across 5 harness roles

5 harness roles ran the same 7 tasks from the same task set. On every one of those tasks the prompt bytes and the runtime-context bytes each role received were byte-identical, so what differs between the rows is the harness and the model behind it. 11 of 35 attempts produced something a checker could read, and why the rest did not is reported beside each row rather than left inside the rate.

Current cross-harness run4 September 2026

What happened to each role's 7 attempts

11 of 35 attempts across 5 harness roles reached a grader and 6 passed, and all 35 receipts verified on re-check.

  • Passed
  • Reached a grader and failed
  • Never reached a grader

Claude Code

Claude Sonnet 5, median 49.1 s

2 of 7 reached a grader, 1 passed

Not graded: 3 refused at the envelope; 1 over the time budget; 1 the provider's own stream was unreadable.

Codex harness

GPT-5.3 Codex Spark, median 25.8 s

4 of 7 reached a grader, 3 passed

Not graded: 1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the provider's own stream was unreadable.

Flywheel harness

GPT-5.3 Codex Spark, median 24.3 s

4 of 7 reached a grader, 2 passed

Not graded: 1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the inner provider's output was unreadable.

Local 14B model

Flywheel Local Coder 14B, median 16.8 s

0 of 7 reached a grader, 0 passed

Not graded: 7 refused at the envelope.

Local 32B model

Flywheel Local Coder 32B, median 240 s

1 of 7 reached a grader, 0 passed

Not graded: 4 the model endpoint did not answer; 1 artifact inside the envelope was invalid; 1 refused at the envelope.

This is one repetition per task on one machine. Times are wall-clock medians and not a speed ranking. It is not market performance and not a quality ranking.

How we know: the run
What this does not prove
The run is one repetition per task on one machine. Nothing here is a market performance claim, a quality ranking, or a statement about a customer workload.
Limits
  • A mean over one repetition is a reading, not an estimate. No interval is reported because none is earned at this sample size.
  • Cost covers the attempts whose provider stated a cost. A role with no cost coverage is not cheaper, it is unmeasured.
  • Latency is wall clock on one machine and includes local model load time.
  • A role that never returned a readable result has no quality numbers here, which is a fact about this run and not a score of zero. Why an attempt went ungraded is reported beside it, because a malformed answer and a missing one are different failures.
  • An envelope found inside a refused answer was still refused, and no checker graded it. That it was there says the harness produced an answer, and it does not say the answer was right.
  • The graded metrics come from in-tree fixtures. They measure a harness against those tasks and not against a customer workload.
  • The run raised before sealing, so nothing ever checked that the source tree still matched the commit these rows name. The attempts each verified their own workspace, which is a narrower claim.
Method
5 harness roles ran the same 7 tasks. On every task the prompt bytes and the runtime-context bytes each role received were byte-identical. Each square in the chart is one attempt.
Outcome and cost

Scroll sideways for more columns.

Outcome and cost for each harness role, with its own denominator.
Role and observed modelLaunchedReached a graderPassedMedian latencyp90 latencyCost
claude_code
claude-sonnet-5		7/72/7149.1 s113.6 s$0.4997 (86% of attempts reported a cost)
codex_harness
gpt-5.3-codex-spark		7/74/7325.8 s47.8 sUnknown: provider_cost_unavailable
flywheel_harness
gpt-5.3-codex-spark		7/74/7224.3 s65.7 sUnknown: provider_cost_unavailable
local_14b
flywheel-local-coder-14b		7/70/7016.8 s33.8 sUnknown: provider_cost_unavailable
local_32b
flywheel-local-coder-32b		7/71/70240 s240 sUnknown: provider_cost_unavailable
Why an attempt went ungraded

A malformed answer and a missing one are different failures, so each is named. An attempt that never returned a readable result has no quality numbers here, which is a fact about this run and not a score of zero.

Ungraded attempts by reason, and whether a refused answer still held one.
RoleReasonsEnvelope recovery
claude_code3 refused at the envelope; 1 over the time budget; 1 the provider's own stream was unreadable.3 of 3 refused answers held a complete envelope behind other text.
codex_harness1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the provider's own stream was unreadable.0 of 1 refused answers held a complete envelope behind other text.
flywheel_harness1 artifact inside the envelope was invalid; 1 refused at the envelope; 1 the inner provider's output was unreadable.1 of 1 refused answers held a complete envelope behind other text.
local_14b7 refused at the envelope.3 of 7 refused answers held a complete envelope behind other text.
local_32b4 the model endpoint did not answer; 1 artifact inside the envelope was invalid; 1 refused at the envelope.0 of 1 refused answers held a complete envelope behind other text.
Graded quality metrics

Each checker graded the attempts that reached it. A mean over one repetition is a reading and not an estimate, so no interval is reported.

budgeted_evidence_selection/v1

agt-017-budgeted-evidence-selection · 2 scored attempts

This checker reported no numeric evidence for the attempts it graded.

contradiction_detection/v1

agt-016-source-contradiction-detection · 3 scored attempts

Checker means for contradiction_detection/v1.
Metricclaude_codecodex_harnessflywheel_harness
failure_code_count
lower is better		Unknown: this role reported no value0 n=10 n=1
false_pair_count
lower is better		Unknown: this role reported no value0 n=10 n=1
pair_precision
higher is better		Unknown: this role reported no value1 n=11 n=1
pair_recall
higher is better		Unknown: this role reported no value1 n=11 n=1
trap_pairs_reported
lower is better		Unknown: this role reported no value0 n=10 n=1

evidence_bound_reporting/v1

agt-015-evidence-bound-reporting · 2 scored attempts

Checker means for evidence_bound_reporting/v1.
Metriccodex_harnessflywheel_harness
evidence_bound_score
higher is better		1 n=11 n=1
fabricated_measurements
lower is better		0 n=10 n=1
failure_code_count
lower is better		0 n=10 n=1
measurement_fidelity
higher is better		1 n=11 n=1
supported_claim_precision
higher is better		1 n=11 n=1
unsupported_claim_recall
higher is better		1 n=11 n=1

index_fallback_integrity/v1

agt-001-index-fallback-integrity · 1 scored attempt

Checker means for index_fallback_integrity/v1.
Metricclaude_code
failure_code_count
lower is better		0 n=1

paired_friction/v1

agt-009-receipts-vs-guardrails-friction · 2 scored attempts

Checker means for paired_friction/v1.
Metriccodex_harnessflywheel_harness
failure_code_count
lower is better		0 n=11 n=1

shared_task_artifact/v1

agt-003-codex-flywheel-shared-task · 1 scored attempt

Checker means for shared_task_artifact/v1.
Metriclocal_32b
failure_code_count
lower is better		1 n=1
Prompt and context parity
Prompt and runtime-context hashes for each task, across all roles.
TaskPrompt SHA-256Runtime context SHA-256Equal across roles
agt-001-index-fallback-integrityfd05c2171e23b1bd2c4c31262feb9bc817459d2ed1f9f867c8a502ef444ce78551835760e28b51f3ccf7769937d6b1fca27569e49ecd1d55fe54d41a4e92cc1fyes, across 5 roles
agt-003-codex-flywheel-shared-task494cc2bbbc2321e2ccfec612d11333e162641718b6a14c1908f35b74f670ffca150a520f6e455a424df6088b33a29aa8a087d03e2e73617e32773ce8ed03f9cbyes, across 5 roles
agt-009-receipts-vs-guardrails-friction79e42fe7d3449f198244a944a611c27b3696af96321654087dadcea92521cdc1eb71123f88d3d0c2603a9d1bcdfc6334ff36dbe5627e2f5e42baf9bb9e83b35ayes, across 5 roles
agt-010-documentation-schematic-maintenance8278c08be170a0b1567573187bd091f360576064a69b6a5389dbb74b53a76ca7b1b8ce7b42775a612368dfe64760a4d5fd6327e6e4837b3a60bba25e3803a33dyes, across 5 roles
agt-015-evidence-bound-reporting6ab23a5ed8e25f80836a230e9f7af20c133ff9f41d3aa8d44e6253b96138b669d6845d06fe795c4bcdb6e4bb9f8fbe302d4111b010a0b0620d081cd7aa0f6278yes, across 5 roles
agt-016-source-contradiction-detection3350fb9664a65a8e0a9c07e577ccd2a534f6a7e1531c4dd3d8f184db5f43a27d6a872a42dd24ff7e6ff795c6d426b207896b0d4d28b318c12fcb51294ccfdf88yes, across 5 roles
agt-017-budgeted-evidence-selection3d297ea31419c4011a2eac4053cbb432256d9d031844cd67d9315e96cb51c16793d85fc4bf151f22aa1071225a5151e0793bc1d91cdd4acb37b1c45bb1abdebbyes, across 5 roles
Benchmark code
66c8498108b8e08a920c717cda7d320409524080
Run identifier
hh3-graded-20260904-050752
Task set
flywheel_agentic_gauntlet_v1
Public result source
Sanitized source record · e8e1191892719af8d47bc915e071bb67ae8969ea27981529f2cae7eeb146a503
Evidence availability
operator-local-hash-only; raw attempt outputs remain private and are represented here by hashes and sanitized receipt records.
Source tree state
The run recorded its source tree as unsealed, so nothing confirmed the tree still matched the commit these rows name. Each attempt verified its own workspace, which is a narrower claim.
Resource use
Unknown: CPU, memory, GPU, and energy observations unavailable
Sanitized receipt records

Every attempt carries a receipt whose subject hash names the artifacts it produced. 35 of 35 verified on re-check; any other state is printed as it was recorded.

Public receipt and receipt-subject identities for all 35 attempts.
Role and taskStateReceipt SHA-256Subject SHA-256
claude_code
agt-001-index-fallback-integrity		verifiedfda4fdc25c762c0a5194474f731f61f650819b23a4e75d779466d22372f018c3d37060295dd45c9e7c962ef585b073656e3df86307fffdcbca91a136e44525db
claude_code
agt-003-codex-flywheel-shared-task		verified7283faae78e8de5cc0c4edc2dbcac6f081f2fefc2019c5b8d3c764b83fbe32e2099e2468e8fe99464bff0ed951fa874fc8a04c6f816e31cd40270a50eed59c87
claude_code
agt-009-receipts-vs-guardrails-friction		verified81e9b15e0579c7a92d6872eea15e988f3b6d2696ebcaf7add4d3b2726798c7949fc59bd193ccd0659884a44b17ec519fe10338699c8e7e05d1b2721932778889
claude_code
agt-010-documentation-schematic-maintenance		verified4826ffbec5ed27fae32709949d95e0891ceb0c18d5c9a054c05acd3ee4cd41322fa2c7806b32969757b3703c5177d39d408d3892ea643254f633f933ba7d8f2b
claude_code
agt-015-evidence-bound-reporting		verified54ed260d639495629f4436bbf47c281988fc2bf5f3ef3855e1b226f4105ba0eee66ed407481d51c4d5101d8291605e895dd8ae4c6b9b63bf19a7bb8fd74ac700
claude_code
agt-016-source-contradiction-detection		verifieded37c92ef2446d82572fc7ffd5afd0701d294b35ec7ff3d95f20e4a93142054e030c008993ea7be1a0f3b0efba3ff9b9832c5be254a5dd179f9d23672effc9f3
claude_code
agt-017-budgeted-evidence-selection		verifiedf03285ff038bd39e91740ec906157e63a69ef192a46eeec56baf084be6f1a81c0623b2e5fff114ae0b0bacb2f133541fa087003f22e9b4648c4c564ca3693942
codex_harness
agt-001-index-fallback-integrity		verifiedfeff70ae48ef03ba13dee0a811ba00628bf3a2079f9028a1a4abb1201da53c71db4a5c1d29c09106310614f19590a0faa13ec6bd7d63a66816c9f11d4ad3a42c
codex_harness
agt-003-codex-flywheel-shared-task		verified7b8551d819c4efe8c824f56faba849fb1a832f379824a5c2753ec17808041485dfd0396ac633eeb4ef9b4863640a71b04ded600df8f8c4b151962702b6f175f4
codex_harness
agt-009-receipts-vs-guardrails-friction		verified822d47d64a6bd9a4c33b254402103597c8faa007dc6d33993f97211b63339de4c45b9812b4cebf10baa77a2b59d8abd2aad7d4eb4b9405614c9c4580df9d7433
codex_harness
agt-010-documentation-schematic-maintenance		verified88af55ded2c19ce654563e6eca5c6a450c007f44644a78226426d14bdc319d69fd27c73197cf3d2e1ca657d52a162c98b69f3a0e528ebc714fa563d11384fca1
codex_harness
agt-015-evidence-bound-reporting		verifiedd56cef8af0354144b912bfa9443a107064971e55c2b2f85ce48e18c628cbc53b43b5292bcb7bb4aad08384769bd8343de2c99a88f8e907d2e3381e0b8a4b4bcb
codex_harness
agt-016-source-contradiction-detection		verifiedea1759931d79c078bf2159ac64ee72287b709375f500533fd33544a3e1bf6018a99a6e00e5306684b48a1d96930660e3bf6beef9727c511844f12b9f532a02ff
codex_harness
agt-017-budgeted-evidence-selection		verified73bd3e67bcc716990d4b74cca6bf17bee107809f1e3aae29166c83897417912443ec6d6fbcc1007671ceb823e79f52bfa9bdf4442d70c0a0d43b5cbe3463799f
flywheel_harness
agt-001-index-fallback-integrity		verified4b470c57eb05bce6559ab080d93330433f55add3b4f8c3d1d392be7d788f3acbc14e1d41837dba2e8af8404fe894a0aa00e1b8433e87910150a9f0b800df95d3
flywheel_harness
agt-003-codex-flywheel-shared-task		verifiedfda74b14c1b135c662e3f337beec15909937351a76a051503aa868ecfebfbf457cf540d6214ae5c63fb4bfd75b0f6fb9007b91ef0993f1c742742f4fe5d9b480
flywheel_harness
agt-009-receipts-vs-guardrails-friction		verifiedcde1bfa30a6176c2cdc5b22411cc9dacbc2e8918dc20ed2776e995b74d5299fe7cb7d9e2cf7757c8f94efb288b4f564ba75955a9e0c0c3de2a7f3945c0afb956
flywheel_harness
agt-010-documentation-schematic-maintenance		verified6f8ca396dfc8aaf799758c138a2c402ff507aa8b51f364192d5d311def21d55bac7c3c909c12a3b256c9efb4b9aa05abb5183fa3e238d2293d6a31ac960f61bf
flywheel_harness
agt-015-evidence-bound-reporting		verifiedbbb30cdf4e68d67cd955f80199d7db486dfa7d71811ef6d2125c02fd02508e5457f4a3c14af8fd860729c168b1dcfaa86b1b354e6ee9d3415b37e5cce352fbfe
flywheel_harness
agt-016-source-contradiction-detection		verifieda0af0182a5da65aea325def36405001ba7fce0cd9aeef714941be58018e3a7c25ab01b5feca57b7f270f6790e510b2641e9a31c0039c28f536b313475faac2f8
flywheel_harness
agt-017-budgeted-evidence-selection		verified04cf10c401178131232cd95057cfa35f84f6a11f8c54177c277495535dfc7d01273e79a5b4e104c0eb48b04d889ad23f772e03edbb21cf5b381bd194f7e1373c
local_14b
agt-001-index-fallback-integrity		verified5d91952788c73262ffcfb05e85adf3fe215b705e20748f8c94539f8a67446ae30c48fd3bde45ef14066551f186c6985a3748a818c6a815dcfd37239062fac5e1
local_14b
agt-003-codex-flywheel-shared-task		verified0c58c20b34d3d0cad087a6c50175d6ea05c27df39627b578c2e77cf52e6927045675caa86a53d8328b674e9ba7564f711aeffa9ddc0008cfab2b81f7613e798a
local_14b
agt-009-receipts-vs-guardrails-friction		verified8596730a9b922c960f877c7476e530d7d8fbfe0689018d41374a54f5ceaa6dc65a8dca52ddde76c4dca3f3d7f66a09e2ce9104c234bcea128b9bc033bc03e598
local_14b
agt-010-documentation-schematic-maintenance		verified83bc17d7051e9e9847705cd7e1bfefcd3afa988d1bb7ef6248a57ebc39839c7a7e3a6e365cb426a26daf57a84e5954fd051e3bae1375aeda01fc9ec54411a344
local_14b
agt-015-evidence-bound-reporting		verified1ea77f3c5dc9d50a3c3c1157bcb788766232ae79aebd021d14f1edb5401a042e6f63c3d312cd8bc7594b3b46aa3783165fd925ab703fcd26db7969ea8c3c7839
local_14b
agt-016-source-contradiction-detection		verified2b88e8bc047321b748917933b34c59b387ad70a20bea0e77abab11ec7d8f8637595f450d0582fcb58563ee7832c9a91505b96dcc3872296b769ba19fa844f706
local_14b
agt-017-budgeted-evidence-selection		verified3723f067796c3c0bdab03c7fbdb003b4810fd713fdf530cef7b606e1eaf19949bfe57c6e2365cd4e6d1c2c3d64e86c9ae378c876751e6cbedc87a791ccb25cfc
local_32b
agt-001-index-fallback-integrity		verified8d88ec1807eb8231ade944d8d93ec9adb2d185ae5a2b37053de51df23d7aa2318124e1ed3ad6b33801d2dcaac35998291b02244684ae8e78390d41cc8cb88424
local_32b
agt-003-codex-flywheel-shared-task		verifieda923196af4d9f494b47df73b57d7222060e2fb343bc639143754a0239f633cb60c9aa25e8cdf519dca6958338385e3d9bef44e59d26a2f05dff3985c5931f407
local_32b
agt-009-receipts-vs-guardrails-friction		verified7531fe03aad98486597e6aa5d6193862f5c58cfee69418e07a8ed35794b3028721f5d4ac1917aa637773cb4e2fa3074490fcb6a744533ac367c0194d783fa283
local_32b
agt-010-documentation-schematic-maintenance		verified500c0153b5151641d57f2b9041e70031c08dd383071b508d7a9fb95d6f085871054883ac0db8fa8ac1cd964c59ec8d3655abdd2421d97db16ce5547fc4907cf0
local_32b
agt-015-evidence-bound-reporting		verifiedbded5251bcb101e73cf91368d7a954e0896e2b1c2480dbfe8cdf6a1816b369f4966181c1415bfe2ce6919cc6237a95bbacbed0bf823faf331ffcda82a89c854a
local_32b
agt-016-source-contradiction-detection		verified5ff8e1497a4575b79559e35c92da3a21ca7bde61fdf2e0448165bcc93f3f675fd7da9118d4e9e3f4938aafa47a9a2f32fef888367a0963baeff7d4b62ab38740
local_32b
agt-017-budgeted-evidence-selection		verified16a9cbf23838439c654d650555b6ab1a1c96c216901bdb8351c52f0fd79291bbc73f6f88065c5c0bc209f01a95acdd4bd97cb5a2fcf83923fc9cce248eaf4b0e
Artifact hashes
SHA-256 identities for the operator-local run artifacts.
ArtifactSHA-256
comparison-input.jsona131530fb1bb4c1509a558fb7248dd913ff3627e0bddd636bdb0fc81ee353a07
gate.json97fc3a6992d2803c2c109aa2788f4440727aab40ab48c69125846986bcb29026
manifest.jsonf18d1e1f397cb85a7e70cfa708dba00e688f1ac2c336ef875a4d389c16bbf36e
matrix.jsonefdbd5ec0fee4fa9cf03aad841fbaab31ab0bd6b7b4c219f98175ce360c68056
profiles.json0b7a04c7ef2e8323666fbbb68af5ca4a85c33b4237a28883e384ebab8c8f830e

Run captured 4 September 2026