Zain Dana Harpercrucible · claim testing

Test a thesis, one claim at a time.

Crucible is test infrastructure for claims about AI systems. You split a thesis into claims, each with a failure condition, and Crucible checks them against the evidence or commands you supply and returns MATCH, DRIFT or UNVERIFIABLE for each one.

See it work, step by step: an animated walkthrough with the real commands and output from the repository.

crucible · v1.3.0 on PyPI (crucible-bench) · no core dependencies
Plate 01 Coral growth that sticks only where it touches something solid.

Try it

$ pip install crucible-bench
$ crucible run examples/thesis-binary-search.json     --measurements examples/measurements-binary-search.json     --registry .crucible --bundle reports/binary-search-run
MATCH 1  DRIFT 1  UNVERIFIABLE 1

A thesis file lists claims, each with the condition that would make it false. A measurement file gives the observed deviation for each claim. crucible run returns one verdict per claim and writes a report you can hand to a reviewer. Watch the recorded workflow: a three-claim thesis goes from 1 MATCH and 2 DRIFT to 3 MATCH over two rounds.

Commands

registerstore a thesis as claims, each with its failure condition
steelmanpropose the strongest refutation of a claim and the test that would settle it
measurerecord how far an observation sits from what the claim predicts
assessreturn MATCH, DRIFT or UNVERIFIABLE per claim
refinefind the weakest claim and run another round
reviewcheck that a packet for an outside verifier holds only the spec and the artifact

Also available: registry, recheck, drift, report, batch, export, measurement-gate and verdicts. The same commands run from a clone with python -m crucible and over MCP.

When a claim comes back UNVERIFIABLE

A claim with no measurement, a mismatched one, or one outside a positive tolerance comes back UNVERIFIABLE or DRIFT. crucible assess then lists what is missing for each such claim and what to add next. Run with --strict to fail on ill-posed measurement rows before assessment.

$ crucible registry stats .crucible --require-witnessed-match
match_provenance: witnessed 4  asserted 2  asserted_zero 1
refused: a latest MATCH rests on an asserted measurement with no replayable recheck (exit 1)

Add --require-witnessed-match in CI to fail when a MATCH rests on a typed-in number with no measurement behind it Crucible can rerun.

Install and release

Install from PyPI as crucible-bench (pip install crucible-bench). Use 1.3.0 or later: advisory GHSA-49qx-cj4f-wfqv covers 1.2.0 and earlier, where a measurement could widen its own tolerance.

Source → github.com/HarperZ9/crucible  ·  PyPI → crucible-bench  ·  Version → 1.3.0 release  ·  Recorded demo → watch the workflow  ·  Cleanroom demo → verdict packet  ·  ← all flagships  ·  the catalog