Where the benchmark evidence stands
No same-task comparison has passed the scorecard gate yet, so none of the 6 named baseline tools has a result; the 2 measured comparisons are published separately with their own limits.
0
same-task comparison groups passed the scorecard gate
2
measured comparisons published on their own page
6
named baseline tools not measured yet
Aider, Claude Code, Cline, OpenAI Codex, OpenCode and OpenHands
112
benchmark-like records kept out of any comparison
Read the measured results: The seven-case stack matrix · The 164-task model comparison
A tool that has not been measured has no score here, which is not a score of zero.
How we know: the evidence status
- Does not prove
- A same-task-set comparison does not establish general model quality, production reliability, causal harness advantage, security, adoption, or independent validation. Results remain scoped to the recorded tasks, prompts, policy, models, harnesses, commit, and metric schema.
- Scorecard gate
- Generic scorecard gate: unknown · no_valid_comparable_scorecards_found. 0 same-task-set comparison groups passed that gate. 2 separately scoped, source-hashed actual comparisons are published with their own limitations. A target marked NOT_MEASURED is a planned baseline, not a result and not a zero.
- Named baseline targets
Market-adjacent and focal tools named by the site-owned benchmark plan. Tool Model Status Reason Plan provenance Aider captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source Claude Code captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source Cline captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source OpenAI Codex captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source OpenCode captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source OpenHands captured at execution NOT_MEASURED No same-task, same-environment executed scorecard is present in the public analytics capture. official · source - Excluded benchmark candidates
Tracked benchmark-like JSON kept out of generic comparative figures. Project Tracked path Schema Exclusion reason flywheel artifacts/agent_recovery_benchmark/20260708_225412/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_225850/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230039/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230336/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230513/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230634/agent_recovery_benchmark.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_231022/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_231022/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_231654/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_231654/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232208/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232208/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232512/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232512/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/forum_benchmark_cases.json forum.benchmark-case-set/v1 unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/loop/cache/e7d325c9cd42556a.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/loop/env/example_pass-96a48c08fee968f6.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/m7_flywheel_local_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232931/m7_frontier_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/agent_recovery_benchmark/20260708_232931/report.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_fail_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_fail_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_pass_a/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_pass_b/task.json unknown unsupported_benchmark_result_schema flywheel artifacts/flywheel-local-coder-14b-benchmark-ci.json unknown unsupported_benchmark_result_schema flywheel artifacts/flywheel-local-coder-14b-benchmark-m7-hard-scorecard.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts/flywheel-local-coder-14b-benchmark-m7-scorecard.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_dry.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_dry_easy_1task.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_dry_easy_full.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_dry_full_attempt.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_frontier_dry_easy_full.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_frontier_dry_hard_full.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_harness_dry_easy.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_codex_live.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_local_ollama_easy_full.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_local_ollama_hard_full.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel artifacts_local_run_2.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel benchmarks/agentic-task-set-adapter-v1.json harness.agentic-task-set-adapter/v1 planning_or_contract_artifact_not_results flywheel benchmarks/agentic-task-set-implementation-plan-v1.json harness.agentic-task-set-implementation-plan/v1 planning_or_contract_artifact_not_results flywheel benchmarks/agentic-task-set-v1.json harness.agentic-task-set/v1 unsupported_benchmark_result_schema flywheel benchmarks/cross-harness-adapter-contract-v1.json harness.cross-harness-adapter-contract/v1 planning_or_contract_artifact_not_results flywheel benchmarks/cross-harness-adapter-contract-v2.json harness.cross-harness-adapter-contract/v2 planning_or_contract_artifact_not_results flywheel benchmarks/dry-run-preflight-command-deck-v1.json harness.dry-run-preflight-command-deck/v1 unsupported_benchmark_result_schema flywheel benchmarks/embodied-realtime-multimodal-v1.json harness.benchmark-contract/v1 planning_or_contract_artifact_not_results flywheel benchmarks/fixtures/cross-harness/documentation-topology-v1.json harness.cross-harness-fixture.documentation-topology/v1 unsupported_benchmark_result_schema flywheel benchmarks/fixtures/cross-harness/index-events-v1.json harness.cross-harness-fixture.index-events/v1 unsupported_benchmark_result_schema flywheel benchmarks/fixtures/cross-harness/paired-friction-observations-v1.json harness.cross-harness-fixture.paired-friction-observations/v1 unsupported_benchmark_result_schema flywheel benchmarks/fixtures/cross-harness/shared-task-facts-v1.json harness.cross-harness-fixture.shared-task-facts/v1 unsupported_benchmark_result_schema flywheel benchmarks/fixtures/evidence-journey/durable-restart-v2.json flywheel.evidence-journey-acceptance-fixture/v2 unsupported_benchmark_result_schema flywheel benchmarks/forum-ledger-deep-verify-scaling-v1.json harness.benchmark-contract/v1 planning_or_contract_artifact_not_results flywheel handoff/site-designer/evidence/benchmark-ci.json unknown unsupported_benchmark_result_schema flywheel m7_scorecard.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel project-docs/releases/14B/shipped-page/benchmarks/easy_set_8task_scorecard.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel project-docs/releases/14B/shipped-page/benchmarks/hard_set_scorecard_QUARANTINED.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel project-docs/releases/14B/shipped-page/benchmarks/he_base_comparison.json flywheel.he-base-comparison/v1 unsupported_benchmark_result_schema flywheel project-docs/releases/14B/shipped-page/benchmarks/humaneval_base_qwen14b.json unknown unsupported_benchmark_result_schema flywheel project-docs/releases/14B/shipped-page/benchmarks/humaneval_flywheel14b.json unknown unsupported_benchmark_result_schema flywheel tasks/research/encoding_sweep_scorecard_20260706.json unknown unsupported_benchmark_result_schema flywheel tasks/research/flexible_marker_scorecard_20260706.json unknown unsupported_benchmark_result_schema flywheel tasks/research/m7_hard_scorecard_20260706.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel tasks/research/m7_scorecard_20260706.json m7-scorecard/v1 legacy_summary_missing_task_set_and_execution_identity flywheel tasks/research/perception_scorecard_20260706.json unknown unsupported_benchmark_result_schema forum benchmarks/otlp_metrics_sample.json unknown unsupported_benchmark_result_schema forum benchmarks/otlp_trace_sample.json unknown unsupported_benchmark_result_schema forum benchmarks/storage_scaling.json forum.storage-scaling-benchmark/v1 unsupported_benchmark_result_schema telos docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark-2026-07-02.json project-telos.hyphal-context-benchmark/v1 unsupported_benchmark_result_schema telos docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark-correction-2026-07-18.json project-telos.hyphal-context-benchmark-correction/v1 unsupported_benchmark_result_schema telos docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark.learn-packet.json unknown unsupported_benchmark_result_schema telos docs/outreach/receipts/twenty-second-wave/learn-hyphal-context/tutor/twenty-second-wave-hyphal-context-benchmark.prooflesson.json unknown unsupported_benchmark_result_schema
Status captured 28 August 2026