Benchmark evidence · 2026-08-28
Benchmark evidence status
Generic scorecard gate: unknown · no_valid_comparable_scorecards_found. 0 same-task-set comparison groups passed that gate. 2 separately scoped, source-hashed actual comparisons are published with their own limitations. A target marked NOT_MEASURED is a planned baseline, not a result and not a zero.
Named baseline targets
| Tool | Model | Status | Reason | Plan provenance |
|---|---|---|---|---|
| Aider | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
| Claude Code | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
| Cline | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
| OpenAI Codex | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
| OpenCode | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
| OpenHands | captured at execution | NOT_MEASURED | No same-task, same-environment executed scorecard is present in the public analytics capture. | official · source |
Excluded benchmark candidates
| Project | Tracked path | Schema | Exclusion reason |
|---|---|---|---|
| flywheel | artifacts/agent_recovery_benchmark/20260708_225412/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_225850/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230039/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230336/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230513/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230634/agent_recovery_benchmark.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/artifacts/agent_recovery_benchmark/20260708_230923/artifacts/agent_recovery_benchmark/20260708_230923/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_230923/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231022/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_231654/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232208/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232512/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/forum_benchmark_cases.json | forum.benchmark-case-set/v1 | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/loop/cache/e7d325c9cd42556a.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/loop/env/example_pass-96a48c08fee968f6.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/m7_flywheel_local_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/m7_frontier_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/report.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_fail_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_fail_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_pass_a/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/agent_recovery_benchmark/20260708_232931/spin/fw_pass_b/task.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/flywheel-local-coder-14b-benchmark-ci.json | unknown | unsupported_benchmark_result_schema |
| flywheel | artifacts/flywheel-local-coder-14b-benchmark-m7-hard-scorecard.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts/flywheel-local-coder-14b-benchmark-m7-scorecard.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_dry.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_dry_easy_1task.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_dry_easy_full.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_dry_full_attempt.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_frontier_dry_easy_full.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_frontier_dry_hard_full.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_harness_dry_easy.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_codex_live.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_local_ollama_easy_full.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_local_ollama_hard_full.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | artifacts_local_run_2.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | benchmarks/agentic-task-set-adapter-v1.json | harness.agentic-task-set-adapter/v1 | planning_or_contract_artifact_not_results |
| flywheel | benchmarks/agentic-task-set-implementation-plan-v1.json | harness.agentic-task-set-implementation-plan/v1 | planning_or_contract_artifact_not_results |
| flywheel | benchmarks/agentic-task-set-v1.json | harness.agentic-task-set/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/cross-harness-adapter-contract-v1.json | harness.cross-harness-adapter-contract/v1 | planning_or_contract_artifact_not_results |
| flywheel | benchmarks/cross-harness-adapter-contract-v2.json | harness.cross-harness-adapter-contract/v2 | planning_or_contract_artifact_not_results |
| flywheel | benchmarks/dry-run-preflight-command-deck-v1.json | harness.dry-run-preflight-command-deck/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/embodied-realtime-multimodal-v1.json | harness.benchmark-contract/v1 | planning_or_contract_artifact_not_results |
| flywheel | benchmarks/fixtures/cross-harness/documentation-topology-v1.json | harness.cross-harness-fixture.documentation-topology/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/fixtures/cross-harness/index-events-v1.json | harness.cross-harness-fixture.index-events/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/fixtures/cross-harness/paired-friction-observations-v1.json | harness.cross-harness-fixture.paired-friction-observations/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/fixtures/cross-harness/shared-task-facts-v1.json | harness.cross-harness-fixture.shared-task-facts/v1 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/fixtures/evidence-journey/durable-restart-v2.json | flywheel.evidence-journey-acceptance-fixture/v2 | unsupported_benchmark_result_schema |
| flywheel | benchmarks/forum-ledger-deep-verify-scaling-v1.json | harness.benchmark-contract/v1 | planning_or_contract_artifact_not_results |
| flywheel | handoff/site-designer/evidence/benchmark-ci.json | unknown | unsupported_benchmark_result_schema |
| flywheel | m7_scorecard.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | project-docs/releases/14B/shipped-page/benchmarks/easy_set_8task_scorecard.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | project-docs/releases/14B/shipped-page/benchmarks/hard_set_scorecard_QUARANTINED.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | project-docs/releases/14B/shipped-page/benchmarks/he_base_comparison.json | flywheel.he-base-comparison/v1 | unsupported_benchmark_result_schema |
| flywheel | project-docs/releases/14B/shipped-page/benchmarks/humaneval_base_qwen14b.json | unknown | unsupported_benchmark_result_schema |
| flywheel | project-docs/releases/14B/shipped-page/benchmarks/humaneval_flywheel14b.json | unknown | unsupported_benchmark_result_schema |
| flywheel | tasks/research/encoding_sweep_scorecard_20260706.json | unknown | unsupported_benchmark_result_schema |
| flywheel | tasks/research/flexible_marker_scorecard_20260706.json | unknown | unsupported_benchmark_result_schema |
| flywheel | tasks/research/m7_hard_scorecard_20260706.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | tasks/research/m7_scorecard_20260706.json | m7-scorecard/v1 | legacy_summary_missing_task_set_and_execution_identity |
| flywheel | tasks/research/perception_scorecard_20260706.json | unknown | unsupported_benchmark_result_schema |
| forum | benchmarks/otlp_metrics_sample.json | unknown | unsupported_benchmark_result_schema |
| forum | benchmarks/otlp_trace_sample.json | unknown | unsupported_benchmark_result_schema |
| forum | benchmarks/storage_scaling.json | forum.storage-scaling-benchmark/v1 | unsupported_benchmark_result_schema |
| telos | docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark-2026-07-02.json | project-telos.hyphal-context-benchmark/v1 | unsupported_benchmark_result_schema |
| telos | docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark-correction-2026-07-18.json | project-telos.hyphal-context-benchmark-correction/v1 | unsupported_benchmark_result_schema |
| telos | docs/outreach/receipts/twenty-second-wave/hyphal-context-benchmark.learn-packet.json | unknown | unsupported_benchmark_result_schema |
| telos | docs/outreach/receipts/twenty-second-wave/learn-hyphal-context/tutor/twenty-second-wave-hyphal-context-benchmark.prooflesson.json | unknown | unsupported_benchmark_result_schema |
A same-task-set comparison does not establish general model quality, production reliability, causal harness advantage, security, adoption, or independent validation. Results remain scoped to the recorded tasks, prompts, policy, models, harnesses, commit, and metric schema.