Findings F204 Method and statistics, Computing and mathematics, Instruments and control found 2026-10-31
Backward and contrary checks caught 39 of 47 quiet faults in eight small simulated instruments
In a mutation simulation on eight small instrument models, running a model backward and against its contrary caught 39 of 47 faults that had passed a calibration point, where a wide forward test caught 29. Half the faults came from one instrument, and a model that is wrong in a self-consistent way passed everything.
- Tier
- B, signal
- Outcome
- mixed
- Rung
- hypothesis
- Kill rule
- not fired
- Corrections
- 4, latest 2026-11-14
- Version
- v2, 2026-11-14
- Redo it
- Reproduce
Download PDF 532 KB BuildTeX source F204.bld
What this means for a reader
If you maintain code that models a physical instrument, a test that runs the model forward and then backward costs little and needs no trusted answer, and in this simulation it found faults that many forward reference points missed. It has a blind spot that matters: a model that is wrong in a self-consistent way inverts itself perfectly and passes. So keep at least one absolute reference, and never calibrate with every scale factor set to 1. The numbers come from eight small models and planted faults, and a test on a real code base with real bugs is what would confirm or sink them.
Objection
This is a simulation on planted faults. Mutants are arithmetic slips made by rule, and no study here shows that they resemble the faults people make in instrument code.
What would change the call. The same test on a real code base with real faults, checked by someone outside the project.
From the paper's first stated limitation; our register records no separate objection.
Abstract
A numerical instrument model is usually tested forward: feed it a case with a known answer and compare. We asked how much two checks that need no known answer add: running the model backward against itself, and running it against its contrary. The study is a simulation. Code planted one fault at a time at every arithmetic site of eight small instrument models we wrote, and scored which tests noticed. Among 47 single faults that passed one calibration point, the backward and contrary checks together caught 39 (83 percent) and a wide forward test caught 29 (62 percent); sign-flip p 0.003, corrected to 0.013 within the study and 0.009 across the 84 tests of the nine-study set. The advantage sits in three of the eight instruments, and one of them, the Radon projector, supplies 23 of the 47 faults; without it the counts are 16 and 12 of 24. Seven faults passed every test: four divide by a grid spacing that every test set to 1, and three leave a wrong wave propagator that still inverts itself exactly. On pairs of faults the order reversed with no clear difference: the wide test caught 27 of 30 and the backward and contrary checks 24 of 30, p 0.12. Nothing here was measured on hardware or on code written by anyone else.
c1 observed, evidence for claim 1
- The numbers in claim 1
- 47; 39; 83; 29; 62; 0.003; 0.013; 0.009; 84 matches the result file, 2026-11-14
- Chart 1
- How faults were planted and which tests were asked to catch them.
- Basis
- observed: measured in this study, with a result file
c2 observed, evidence for claim 2
- The numbers in claim 2
- 23; 47; 16; 12; 24 matches the result file, 2026-11-14
- Chart 2
- Quiet faults caught by each test, instrument by instrument.
- Basis
- observed: measured in this study, with a result file
c3 observed, evidence for claim 3
- The numbers in claim 3
- 1 matches the result file, 2026-11-14
- Chart 1
- How faults were planted and which tests were asked to catch them.
- Basis
- observed: measured in this study, with a result file
c4 observed, evidence for claim 4
- The numbers in claim 4
- 27; 30; 24; 0.12 matches the result file, 2026-11-14
- Basis
- observed: measured in this study, with a result file
Introduction
Scientific code often has no cheap source of right answers. Software testing calls this the oracle problem, and metamorphic testingDefinition: metamorphic testing software engineering (testing) Testing a program by checking a relation that must hold between the outputs of several related runs, used when no trusted expected output is available. Excludes a test that compares one output with a stored expected value, and any proof that an output is correct. full definition is the standard response: check a relation that must hold between several runs of the program. The term comes from software engineering and covers any test that compares outputs of related inputs with each other; a test that compares one output with a stored expected value is outside it. Segura and colleagues surveyed 119 papers on the method [1], and a later article by the same first author warns that a satisfied relation does not confirm that an output is right [2].
Physics supplies such relations for free. A lossless film transmits the same from either side. A wave run forward and then backward in time returns to its start. A projector and its adjointDefinition: adjoint linear algebra and inverse problems The partner of a linear map that carries vectors from the output space back to the input space so that the inner product of an output with a vector equals the inner product of the input with the partner's result. Excludes the inverse, which undoes the map; an adjoint exists for maps that cannot be undone. full definition satisfy an inner-product identity. The adjoint, in linear algebra and inverse problems, is the partner map that moves a vector from the output space back to the input space so that the two inner products agree; it is a different thing from the inverse, which undoes the map. Claerbout's dot-product test, which compares two scalars computed with random vectors, has been a routine check of such pairs in geophysical imaging for decades, and his texts say a failure condemns the code while a pass leaves correctness unproven [3]. A recent tutorial on adjoint gradients names the same blind spot we met here: the test checks mutual consistency, so a wrong derivative with a matching wrong transpose passes [4].
None of this is new, and we claim no novelty. The question was a practical one for our own work: how much weight a reference-free check should get beside a forward test when a new numerical kernel is written. The decision it informed was whether the check enters our catalogue of default checks. We priced it with mutantsDefinition: mutant software engineering (mutation testing) A copy of a program with one small fault planted by rule, such as a plus swapped for a minus, used to measure whether a test notices. Excludes a bug written by a person; mutants stand in for such bugs and may not resemble them. full definition. A mutant, in mutation testing, is a copy of a program with one small fault planted by rule; it is a stand-in for a bug and may not resemble the bugs people write. The survey above describes the share of mutants a test detects as a common measure of test strength [1].
Preregistration and design
The study ran twice on 31 October 2026, each time under a preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition: a plan with forecasts and rules, hash-locked before the run.
The first run compared four test sets on every mutant of every function. It gave the backward and contrary checks 79 of 89 false passes (89 percent) against 29 of 89 (33 percent) for the wide forward test. Reading the 51 faults that only the reference-free checks caught showed that most sat in functions the forward tests never called, or in an output the forward film test never read. The result was partly a gap in our own forward tests. That run is kept as it ran, and its headline is not claimed here.
The second preregistration was written after the first result was seen, and says so in its text. It scores only faults in functions that the forward tests call, makes the forward film tests read both outputs, reports faults in inverse functions apart, and adds pairs of faults. Its forecasts were that the wide forward test would catch more than the reference-free checks, on single faults and on pairs. Its kill conditionDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition, the outcome fixed in advance that would end the claim, was a catch rate under 30 percent for the reference-free checks on single faults. Each forecast became a rule that code reads off the result as a verdictDefinition: verdict research method (our usage) The pass or fail of one preregistered rule set against the run's numbers, read off by code. Excludes an interpretation written after reading the numbers. full definition.
The first preregistration also covered a re-reading of two earlier studies of ours with the same three-way idea. That part was post hoc and is not reported in this paper.
Data and provenance
There is no external dataset. The material is eight instrument models written for this study in plain Python with NumPy, each a few lines long so that every arithmetic site can be changed: a normal-incidence thin film by characteristic matrices; an angular-spectrum wave propagator; a leapfrog wave equation in one dimension; a blur with its adjoint and a regularised deblur; a pixel-driven Radon projector with its adjoint; an explicit heat-diffusion step; ray transfer matrices; and a Stokes analyzer row with a least-squares inversion. The faults were generated by rule from that source. The forward reference values are closed-form results from textbook physics, coded by us. All of it is our own work, written the same day as the tests.
Methods
Planting faults
A program walked the syntax tree of each model and listed every site of seven kinds: a plus or minus, a times or divide, a unary minus, a complex conjugate, a numeric constant (twice: add 1, and double), and an inequality. Each mutant changes one site: plus and minus swap, times and divide swap, the unary minus or the conjugate is dropped, the constant changes, the inequality flips. We did not choose which faults to plant.
A mutant can leave the program's behaviour unchanged; mutation testing calls that an equivalent mutantDefinition: equivalent mutant software engineering (mutation testing) A mutant whose change leaves the program's behaviour the same on every input, so that no test can or should fail on it. Excludes a fault that a weak test happens to miss. full definition, and it is no fault at all. A differential oracleDefinition: differential oracle software engineering (testing) A judge of behaviour that runs two versions of a program on the same inputs and reports whether their outputs differ. Here it marks a mutant as a fault when it differs from the original on random inputs. Excludes any statement that the original program is right. full definition separated the two cases: the mutant and the original ran on random inputs wider than any test uses, and any difference in output beyond a relative tolerance of 1e-9, or a change of array shape, marks a fault. This is a comparison of two programs and says nothing about whether the original is right.
Four test sets
Each instrument has four tests, written before any mutant was scored and adjusted until the original code passed all of them.
- The calibration pointDefinition: calibration point our usage, from instrument practice A single forward test case with a known closed-form answer, standing for the one check a new instrument model often gets. Excludes a calibration of hardware against a physical standard. full definition: one forward case with a closed-form answer, such as the reflectance of a quarter-wave film. It stands for the single check an instrument often gets.
- The wide forward testDefinition: wide forward test our usage A set of 4 to 15 forward test cases with closed-form answers at random parameter values, all of which must pass. Excludes any check that runs the model backward or against a complement, and any reference taken from measurement. full definition: 4 to 15 forward cases with closed-form answers at random parameter values.
- The backward checkDefinition: backward check our usage, from physics symmetries A test that runs an instrument model in reverse against itself, by time reversal, reciprocity, an adjoint identity or an inverse, and needs no reference value. Excludes a comparison with a known answer. full definition: the model against itself in reverse. Transmission through a film stack from the other side; a wave propagated out and back; a wave run on with its last two states swapped; blur then deblur; the inner-product identity for the projector and its adjoint; ray matrices for the elements in reverse order; analyzer rows inverted to recover a known Stokes vector. Diffusion has no time reversal, so one small step with the diffusivity negated stood in for it.
- The contrary checkDefinition: contrary check our usage, from physics symmetries A test that runs an instrument model against its opposite or complement, such as an aperture and its complementary screen or a lens and its negative, and needs no reference value. Excludes a comparison with a known answer, and the backward check, which reverses the same case. full definition: the model against its opposite or complement. A half-wave absentee layer that must leave reflectance unchanged, with energy conserved; an aperture and its complement summing to the open field (Babinet's principle); source and receiver swapped; a lens followed by its negative; crossed polarizers reading zero.
The last two use no reference value. The three-way testDefinition: three-way test our usage The backward and contrary checks taken together after a forward calibration. A fault is caught three-way when either check fails. Excludes the wide forward test, which is scored apart. full definition is our name for the backward and contrary checks taken together after a forward calibration: a fault is caught three-way when either one fails.
The score
A false passDefinition: false pass our usage A planted fault, confirmed by the differential oracle, that still passes the calibration point. Excludes equivalent mutants, and mutants that crash a test. full definition is a mutant that the oracle marks as a fault and that still passes the calibration point. These are the quiet faults. For a test set that fails on of the false passes, the catch rate is
Mutants that crashed any test left the count, since a crash is noticed without a test.
The two rates were compared on the same faults. For each false pass let be 1 when only the three-way test caught it, minus 1 when only the wide test caught it, and 0 otherwise. A sign-flip testDefinition: sign-flip test statistics (permutation tests) A paired test that gives each difference a random sign many times and counts how often the mean of the re-signed differences reaches the real mean. Excludes a test that assumes a normal distribution, and any comparison of unpaired groups. full definition drew 4,000 random sign patterns for the and counted how many gave a mean at or above the real one:
with that count and = 4,000. The four p values of the run (each direction, on singles and on pairs) were corrected with the Benjamini-Hochberg correctionDefinition: Benjamini-Hochberg correction statistics (multiple testing) An adjustment of a family of p values that limits the expected share of false discoveries among the tests called significant. Excludes control of the chance of any single false discovery, which is the stricter Bonferroni aim. full definition [5], which limits the expected share of false discoveries among the tests called significant, and again across all 84 tests of the nine simulation studies run together that day.
Pairs
Two errors can cancel at a calibration point. The run drew 400 random pairs of sitesDefinition: compensating pair our usage Two planted faults in one function, drawn at random, kept in the score when together they still pass the calibration point. Excludes pairs chosen by hand to cancel. full definition, each pair inside one forward function, from a fixed seed, planted both faults and scored the result the same way. Draws from a function with a single site were skipped, which left 344 pairs.
False alarms
A reference-free check that fails on a correct program would be useless. Two things bound that: every test passes on the original code, and in the first run neither test failed on any equivalent mutant. With 0 failures in = 9 equivalents, the one-sided 95 percent upper bound on the false-alarm rate is
which is 0.28.
Controls and planted checks
The controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition permuted the fault labels among the mutants of each instrument 200 times and recomputed the three-way catch rate. It gave a mean of 0.726 with a 95th percentile of 0.76, below the real 0.830 (p 0.005). The preregistration calls this a weak check, and it is: only 9 of the 148 scored mutants are equivalent, so permuting labels changes little.
- planted checkPlanted: 200 single faultsOne arithmetic slip at a time at every arithmetic site of eight small instrument models. Everything here is simulated.
- data47 quiet faultsFaults that change the output and still pass one calibration point with a known answer.
- measurementThree-way: backward and contrary checksRun the model backward against itself and against its contrary. Neither needs a known answer.
- controlComparison: a wide forward testMany forward reference points with known answers, written before any fault was scored.
- forecast failedForecast: the wide test wins. It caught 29, the checks 3911 faults fell only to the backward and contrary checks and 1 only to the wide test. A model that is wrong in a self-consistent way passes both.
Read from result.json (second run), run 20261031T110819Z. SHA-256 of the run's file 2aca513516dbd12215b2e93776406f8916982172d2ba044997877a1a46000387 and of the published copy d183ef6401c9447bb096a716afe98e132988030976ab087bac3a6cbbd8046c60. In the copy, file paths are cut to the file name and the layout (indentation, line endings) may differ; every value is the same.
The first run carried a planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition: with the conjugate dropped in the blur adjoint, the forward tests had to pass and the contrary check had to fail. Both happened.
All four tests passed on the original code of all eight instruments in both runs.
Results
The second run planted 200 single faults in the eight forward functions. Of these, 52 crashed a test and left the count, 148 were scored, 9 were equivalent, and 47 were faults that passed the calibration point.
Among 47 single faults that passed one calibration point, the backward and contrary checks together caught 39 (83 percent) and a wide forward test caught 29 (62 percent); sign-flip p 0.003, corrected to 0.013 within the study and 0.009 across the 84 tests of the nine-study set. Taken alone, the backward check caught 33 and the contrary check 28. Eleven faults fell only to the three-way test and 1 only to the wide test. All three test sets together caught 40. Figure 1 gives the rates.
The total hides how uneven the instruments are. The advantage sits in three of the eight instruments, and one of them, the Radon projector, supplies 23 of the 47 faults; without it the counts are 16 and 12 of 24. The split without the projector was computed after the result was seen. Table 1 and Figure 2 give every instrument. In three instruments the three-way test caught nothing.
Seven faults passed every test: four divide by a grid spacing that every test set to 1, and three leave a wrong wave propagator that still inverts itself exactly. The four are the single false pass in the wave code and all three in the heat code; each swaps a division by the grid spacing for a multiplication, which changes nothing when the spacing is 1. The three in the angular-spectrum code are two sign changes inside the term under the square root and one division by the transfer function where the code should multiply. Each of those leaves a propagator that conserves power, obeys Babinet's principle and undoes itself when run backward, so the reference-free checks pass, and the forward tests read only a beam width that the fault does not change. The one-step undo that stood in for time reversal in the heat code caught none of its 3 false passes.
On pairs of faults the order reversed with no clear difference: the wide test caught 27 of 30 and the backward and contrary checks 24 of 30, p 0.12. Of the 344 pairs, 155 crashed, and 30 of the remaining 189 were faults that passed the calibration point. Three fell only to the wide test, none only to the three-way test, and 3 passed everything.
Faults in the inverse and adjoint functions were scored apart. All 34 scored mutants there were false passes, the three-way test caught 32 of them and the wide forward test caught 0. The zero could not have come out otherwise, because no forward test calls those functions. The 32 show only that a pairing test reaches code a forward test does not.
Table 2 sets the forecasts against the outcomes. The forecast that the wide test would win on single faults was wrong.
Discussion
The two kinds of test fail in different places. A forward test sees only the outputs it reads at the parameter values it visits; the thin-film and projector faults it missed were visible to a symmetry. A symmetry test sees only departures from the symmetry; a model that is wrong and still symmetric is invisible to it. The seven survivors are the useful part of the result, because each names a habit to change. Four of them existed only because every test used a grid spacing of 1, where multiplying and dividing agree. Three existed because a propagator with the wrong dispersion is still unitary.
That gives two working rules, which we adopted: calibrate at values other than 1, and keep one absolute reference beside any reference-free check. The catalogue entry for our own kernels now asks for both kinds of test.
The size of the advantage should not be carried anywhere. It depends on which instruments are in the sample, and one instrument holds half the faults. On pairs of faults, where two errors cancel at the calibration point, the wide test did at least as well.
Limitations
- This is a simulation on planted faults. Mutants are arithmetic slips made by rule, and no study here shows that they resemble the faults people make in instrument code.
- The eight instruments are small models we wrote ourselves, and we wrote the tests too, on the same day. A stronger forward test or a weaker symmetry test would move both rates.
- One instrument, the Radon projector, supplies 23 of the 47 faults. The three-way test caught 0 of the 7 faults in the angular-spectrum, wave and heat code.
- 52 of 200 single mutants and 155 of 344 pairs crashed a test and left the count, so the rates describe quiet faults only.
- The false-alarm rate is bounded loosely: 0 failures on 9 equivalent mutants allows a rate up to 0.28.
- The second design was written after the first result was seen. It was made to remove an advantage the first design gave the three-way test, and it is still a design chosen with knowledge of the data.
- The split without the Radon projector is post hoc.
- The label-permutation control is weak, as the preregistration says.
- Scoring was by our own code and was not blind. Nobody outside the project has checked the code or the result.
Evidence
Mixed. The forecast that a wide forward test would win on single faults failed (29 against 39 of 47). On 30 pairs of faults the wide test caught 27 and the backward and contrary checks 24 (p 0.12). The backward and contrary checks caught 0 of the 7 false passes in the angular-spectrum, wave and heat code, and 7 faults passed every test. False alarms are bounded only below 0.28 (0 of 9).
Figures
These are the paper's own figures, numbered as in the PDF, and each "Figure" reference in the text points here. The charts on this page redraw the same results from the result files and carry their own numbers.
Tables
| Instrument | False passes | Wide forward | Three-way |
|---|---|---|---|
| Thin film | 6 | 4 | 6 |
| Angular spectrum | 3 | 0 | 0 |
| Wave, leapfrog | 1 | 0 | 0 |
| Blur | 0 | 0 | 0 |
| Radon projector | 23 | 17 | 23 |
| Heat diffusion | 3 | 0 | 0 |
| Ray matrices | 4 | 4 | 4 |
| Stokes analyzer | 7 | 4 | 6 |
| All eight | 47 | 29 | 39 |
| Quantity | Forecast | Outcome |
|---|---|---|
| Single faults passing the calibration point | 25 | 47 |
| Wide forward catch rate, singles | 60 percent | 62 percent |
| Three-way catch rate, singles | 55 percent | 83 percent |
| Singles caught only by the wide test | at least 3 | 1 |
| Singles caught only by the three-way test | at least 3 | 11 |
| Pairs passing the calibration point | 50 | 30 |
| Wide forward catch rate, pairs | 75 percent | 90 percent |
| Three-way catch rate, pairs | 65 percent | 80 percent |
Appendix A. The preregistered verdicts
Each verdict is a rule written before the second run. Values are read from that run's result file. Five rules hold and two fail. The pairs rule holds on the point values alone; the difference behind it has p 0.12.
| Verdict | Deciding rule | Value | Holds |
|---|---|---|---|
| Kill condition not met | three-way catch rate on singles at least 0.3 | 0.83 | yes |
| Original code clean | every test passes on the original code | all pass | yes |
| Wide test has its own catches | singles caught only by the wide test, at least 3 | 1 | no |
| Three-way test has its own catches | singles caught only three-way, at least 3 | 11 | yes |
| Wide wins on singles | wide rate above three-way rate | 0.62 against 0.83 | no |
| Wide wins on pairs | wide rate above three-way rate | 0.90 against 0.80 | yes |
| Beats the label control | three-way rate above the control's 95th percentile | 0.83 against 0.76 | yes |
Numbers recorded in our publication register
| What | Value |
|---|---|
| Sample | 47 single faults that passed a calibration point, of 148 scored; 30 such pairs of 189 |
| Main estimate and interval | no interval; backward and contrary checks 39 of 47 against the wide forward test 29 of 47, sign-flip p 0.003 |
| Forecast against outcome | Did not hold: the forecast that a wide forward test would win on single faults (29 against 39 of 47). On pairs the wide test caught 27 of 30 and the backward and contrary checks 24 |
| Known-answer check | every test passes on the original code of all eight instruments |
As recorded in the publication register, the summary row each finding gets; the paper's own numbers above were re-read from the result files.
What lowers it
- blinding Not blind. The second design was written after the first result was seen; the models and the tests were written by the same project. from our publication register, on blinding
- depth of reading The cited literature was read at the level of search snippets; full texts were not read. from the Verified section
What holds it up
- scoring Scored by code against rules fixed in advance. from our publication register, on who scored it
- preregistration Preregistered and hash-locked before the run (sha256 d82bfea0). from the Reproduce section
- control Controls: 9 equivalent mutants raised no false alarm, which bounds the rate below 0.28; the label-permutation control is weak, as the preregistration says. from our publication register, on controls
Where the claim stands
- lead
- hypothesis
- thesis
- conjecture
- theorem
- law candidate
- killed
A preregistration with pass-or-fail rules is on file, which this rung needs. The next rung needs a named rival reading beaten by a decisive observation, and none is on file. Tier B: A real signal in simulation and a useful blind spot. The next step is the same scoring on real instrument code with a reader outside the project.
Trail
Versions
- v2 2026-11-14, paper version 1.0, first published PDF sha256 620e36d626e7a7ac
- v1 2026-10-31, entered in the findings ledger
Corrections
2026-11-14 scope found by the publication pass
The re-reading of two earlier studies, which the ledger entry lists, was left out because it was post hoc and needs those studies to be read.
2026-11-14 wording found by the publication pass
The per-instrument split was added beside the headline count, because the three-way test caught none of the false passes in three instruments.
2026-11-14 number found by the publication pass
The ledger entry gives one corrected value, 0.009; this paper prints both the within-study value, 0.013, and the across-studies value, 0.009, and says which is which.
2026-10-31 wording found by the study itself, before publication
The first run's result, 89 percent against 33, was traced to forward tests that did not reach the inverse functions or the film's transmittance. now The second run was preregistered to remove that, and its numbers replace the first run's.
Each correction is a dated note; the findings ledger is never rewritten, and a note governs where it differs.
Ledger and version dates up to publication are local dates in US Pacific time. Clock times in the text are UTC, seven or eight hours ahead, so a lock or a run stamped just after midnight UTC belongs to the previous local day. Corrections from 14 November 2026 on are dated in UTC.
References
- Segura, S., Fraser, G., Sanchez, A. B. and Ruiz-Cortes, A. (2016). A survey on metamorphic testing. IEEE Transactions on Software Engineering. abstract or summary, 2026-11-14: search snippets; full text not read used in Introduction, Introduction
- Segura, S. and colleagues (2020). Metamorphic testing: testing the untestable. IEEE Software. abstract or summary, 2026-11-14: search snippets; full text not read used in Introduction
- Claerbout, J. F. (2010). Basic Earth Imaging. Stanford Exploration Project. Section "Dot product test". abstract or summary, 2026-11-14: search snippets; full text not read used in Introduction
- From inverse problems to adjoint gradients: a discrete, implementation-oriented tutorial on PDE-constrained optimization. arXiv:2610.04789. abstract or summary, 2026-11-14: search snippets; full text not read used in Introduction
- Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57, 289-300. abstract or summary, 2026-11-14: search snippets; full text not read used in Methods
Cited by later findings
- No later record in the findings ledger treats this finding.
Reproduce
Preregistration
sha256 d82bfea0f79dd4708e1fe0c06639da6091c977f6e3d10ab07bc11bb7c2803940
plan be230e2b62c03e607781e72f210fc6023865d5557bd5680f8f2ce02901409918
Steps
- Data: no external dataset. The inputs are the eight instrument models and four test sets per model described in the Methods section, written for the study in Python with NumPy; they are not yet in a public repository.
- Plant every single-site fault of the seven kinds in each forward function. Mark faults with a differential oracle on random inputs. Drop mutants that crash a test.
- Keep the faults that pass the calibration point and compute eq. 1 for the wide forward test, the backward check, the contrary check, and the last two together.
- Compare the wide and three-way tests with eq. 2 on 4,000 sign patterns, and correct the p values by Benjamini-Hochberg.
- Draw 400 pairs of sites within one forward function from seed 20261106 and score them the same way. Score the inverse and adjoint functions apart.
- Preregistration SHA-256: d82bfea0f79dd4708e1fe0c06639da6091c977f6e3d10ab07bc11bb7c2803940. Plan SHA-256: be230e2b62c03e607781e72f210fc6023865d5557bd5680f8f2ce02901409918. Run 20261031T110819Z.
- First run, kept as run: preregistration SHA-256 480be6fe4423efd197e7b83994d0cf41696b39fa474565c972d9a4f54bf745e9, run 20261031T110646Z.
Cost. The analysis took about 3 s on one desktop CPU. The study was not re-run for this paper.
Downloads
- Whitepaper, PDF F204.pdf, 532 KB
- BuildTeX source F204.bld, 34 KB
- Build receipt F204.buildtex.json, 1 KB
- BuildLang doc module doc.bld, 5 KB, imported by the source
The PDF is typeset by BuildTeX from F204.bld, which is written from the same source as this page. The receipt records the source's SHA-256 and the PDF's (620e36d626e7), so anyone can rebuild it and compare. The copy here has its hidden author field blanked, as the site does for every file it publishes; blank the same field in a rebuild before comparing. The name stays on the first page.
Verified
Verified on 2026-11-14. On 14 November 2026 the counts of mutants, crashes, equivalents and false passes, every catch rate for single faults, pairs and inverse functions, the per-instrument rows, the four p values with their corrected values, the control's mean, percentile and p, the seven verdicts and the list of seven surviving sites were re-read from the second run's result file; the first run's 89 false passes, its two rates, the 51 faults and the 0 of 9 false alarms were re-read from the first run's result file; and the corrected value across 84 tests was re-read from the study set's correction file. They match the findings ledgerDefinition: findings ledger our records The append-only, dated list of every result we have recorded, numbered F1, F2 and on. Corrections are added as new dated lines and old entries are never rewritten. Excludes a curated list of successes. full definition, whose q of 0.009 is the across-studies value. The counts printed as "39 of 47" and the like are the file's rates times its denominators. The seven surviving sites were located in the instrument source by listing its mutation sites with the study's own site finder, and the grid spacing of 1 was read in the test source. The preregistration's SHA-256 matches the hash in the result file, the lock file and the sidecar file. Both figures were drawn by a script that reads only the second run's result file. The Benjamini and Hochberg paper is cited for the method only.
Not checked. The study was not re-run for this paper. The literature was found by two searches on 14 November 2026 and read at the level of search snippets; the full texts were not read.
Terms on this page
Each term is defined where it is first used on this page. Each name below opens its entry in the findings glossary, which lists every page that uses it.
- metamorphic testing software engineering (testing) first use
- adjoint linear algebra and inverse problems first use
- mutant software engineering (mutation testing) first use
- preregistration research method (metascience) first use
- kill condition research method (our usage, after Popper's falsification) first use
- verdict research method (our usage) first use
- equivalent mutant software engineering (mutation testing) first use
- differential oracle software engineering (testing) first use
- calibration point our usage, from instrument practice first use
- wide forward test our usage first use
- backward check our usage, from physics symmetries first use
- contrary check our usage, from physics symmetries first use
- three-way test our usage first use
- false pass our usage first use
- sign-flip test statistics (permutation tests) first use
- Benjamini-Hochberg correction statistics (multiple testing) first use
- compensating pair our usage first use
- control statistics and experimental design first use
- planted check research method; called a positive control in experimental science first use
- findings ledger our records first use
To report an error, name the finding, the claim number and the source on the Work page, which gives the address. Accepted corrections join the Trail with their date.