Published 1 sigma error bars hold near nominal; a 95 percent interval needs 3 to 5 sigma

A stated uncertainty is a forecast of where later, better measurements will put a value. We scored that forecast on four reference sources: particle masses and widths in the Particle Data Group tables (2010 to 2025), atomic masses in the Atomic Mass Evaluation (2003 to 2020), the CODATA recommended constants (2002 to 2022) and Huchra's compilation of published Hubble constant values. For each quantity we measured the shift between the first and last edition in units of the first edition's stated sigma. The earlier 1 sigma interval covered 0.68 of later PDG values that moved (n 227) and 0.66 of later AME values (n 2,213, nearly all of which moved), close to the nominal 0.6827.c1 The tails are far heavier than a normal curve. Covering 95.45 percent of later values took 3.8 times the stated sigma for moved PDG values, 4.0 for AME and 4.9 for CODATA.c2 Student t fits to the nonzero shifts gave nu of 1.67 for PDG, 1.22 for AME and 2.17 for CODATA, close to the Cauchy shape.c3 Published Hubble constant values before 1990 covered a proxy truth of 70 km/s/Mpc within 1 sigma in 0.35 of 115 cases.c4 Later tables reuse inputs from earlier ones, which flatters every coverage figure here.

A stated 1 sigma uncertaintyDefinition: stated 1 sigma uncertainty metrology (the standard uncertainty of the GUM) and physics data tables The plus-or-minus a table prints beside a value, meant as one standard deviation of the value's error. For a normal error the interval of one sigma either side holds the truth 68.27 percent of the time and two sigma 95.45 percent. Excludes an expanded uncertainty already multiplied by a coverage factor, and any bound meant to hold always. full definition is the plus-or-minus a table prints beside a value, meant as one standard deviation of its error. For a normal error, one sigma either side should hold the truth 68.27 percent of the time and two sigma 95.45 percent. Later editions of the same table give a better estimate of the true value, so the share of later values that fall inside an earlier interval, its coverageDefinition: coverage statistics and metrology The share of cases in which an interval drawn round an earlier value contains the value that later replaced it. Here the interval is one or two of the earlier value's own stated sigma either side. Excludes the share of intervals that contain the true value, which no later table can supply. full definition, can be counted.

Henrion and Fischhoff [5] examined the history of the physical constants, including the speed of light, and found that reported uncertainties were consistently too small, with recommended values often moving by more than their stated errors. Chen, Gott and Ratra [6] built the error distribution of 461 Hubble constant measurements in Huchra's compilation against a central value of 71 km/s/Mpc. They found it non-Gaussian, with 95.4 percent limits at 7.0 times the quoted errors and a shape described by a widened Student t with two degrees of freedom. Bailey [7] analysed about 41,000 measurements of 3,200 quantities from medicine, nuclear and particle physics and interlaboratory comparisons, and found the normalised differences between measurements of the same quantity close to Student t distributions, often almost Cauchy. Its reading is that researchers estimate the size of typical errors well and underestimate the chance of large ones.

This paper asks the same question of edition-to-edition changes in four large tables at once, with a planted check, a normal-world control, a test that removes unchanged values, and forecasts fixed in advance. Bailey's data include nuclear and particle physics. Our searches on 9 November 2026 did not return a study that scores the PDG mass-width files edition against edition; that is what the searches returned, and it does not show that none exists.

The design was written as a preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition dated 31 October 2026 and hash-locked on 1 November 2026 at 01:54 UTC. The forecasts were copied from an earlier forecast file written and hashed before any data were read. Before the lock, the file formats of the PDG, AME and Huchra tables were read; no coverage was computed. The decision it informs was stated in the file: whether stated uncertainties, and our own forecast bands, get a correction factor.

The locked file names the tables and editions, the coverage count, a moved-values armDefinition: moved-values arm research method, our usage The same coverage count repeated after dropping every quantity whose value did not change between the two editions, since an unchanged value counts as covered whatever its stated uncertainty. Excludes a test of whether the unchanged values were right. full definition that drops every quantity whose value did not change (an unchanged value counts as covered whatever its sigma), the proxy truthDefinition: proxy truth research method, our usage A value fixed in advance and used in place of the unknown true value where no later edition settles it. Here three values of the Hubble constant, 67.4, 70 and 73 km/s/Mpc, each reported on its own. Excludes a claim that any of the three is the true value. full definition values for the Hubble constant, a planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition (data built so the right answer is known, run through the same analysis), a controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition (a version of the data with the structure under test removed, scored the same way), the assumptions to be changed one at a time, the forecasts and a kill conditionDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition, a rule that decides against the claim with no room for judgement after the run.

Kill condition, without discretion: if the planted check fails, no coverage claim counts.

The same run also scored our own numeric forecast bands from earlier studies. That arm is reported as a separate finding (F366) and appears here only in the appendix.

All four are public tables of measured values. The files were read as data. If the PDG 2016 file had been identical to 2014 it would have been dropped; it was not.

For quantity $i$ with value $a_i$ and stated sigma $s_i$ in the first edition and value $b_i$ in the last, the shift in units of the earlier sigmaDefinition: shift in units of the earlier sigma statistics, our usage; called a normalised residual in metrology The size of the change in a quantity's value between an earlier and a later edition of a table, divided by the uncertainty stated in the earlier edition. Excludes the sign of the change, and any uncertainty in the later value. full definition is

Only quantities present in both editions with $s_i>0$ enter.

Coverage at $m$ sigma is the share of quantities with $z_i\le m$:

where $N(z\le m)$ counts the quantities whose shift is at most $m$ and $n$ is the number of quantities. The needed multipleDefinition: needed multiple metrology, our usage; a coverage factor fitted after the fact The multiple of the stated sigma that would have covered 68.27 or 95.45 percent of the later values. It is the matching percentile of the shifts in units of the earlier sigma, written k68 and k95. Excludes a factor known before the later values were published. full definition $k_{68}$ is the 68.27th percentile of the shifts and $k_{95}$ the 95.45th percentile: the multiples of the stated sigma that would have covered those shares. For honest normal errors both coverages sit at 0.6827 and 0.9545 and the two multiples at 1 and 2.

A folded Student t distributionDefinition: Student t distribution and its tail index nu statistics A bell-shaped family with heavier tails than the normal curve. Its degrees of freedom nu set the tail weight, so small nu means large shifts are common, nu of 1 is the Cauchy distribution and nu without limit is the normal curve. Excludes skewed or bounded shapes. full definition with a scale and degrees of freedom nu was fitted by maximum likelihood to the nonzero shifts. Small nu means heavy tails; nu of 1 is the Cauchy distributionDefinition: Cauchy distribution probability The Student t distribution with nu equal to 1, whose tails are so heavy that its mean and variance do not exist. Excludes any distribution with a finite variance. full definition, and a normal curve has nu without limit.

The interval on each 1 sigma coverage comes from a bootstrapDefinition: bootstrap statistics, resampling Redrawing the quantities of a table with replacement, here 300 times, and recomputing the coverage each time. The middle 95 percent of the recomputed values is the interval. Excludes uncertainty from the choice of editions or from shared inputs, which resampling quantities does not touch. full definition: quantities are redrawn with replacement 300 times and the coverage recomputed; the 95% intervalDefinition: 95% interval statistics The range that holds the middle 95 percent of the bootstrap or control draws for a number. We report it beside the number. Excludes a statement that the true value lies inside with 95 percent probability. full definition is the middle 95 percent of the redrawn values.

Huchra's table has no later edition of each entry. Each published value was scored against a proxy truth fixed in advance, 67.4, 70 or 73 km/s/Mpc, reported separately, and the values published before 1990 were scored against 70.

Each was run apart: the moved-values arm; the top 1 percent of shifts removed; the last pair of editions alone; and, for CODATA, one row per distinct shift rounded to two decimals, since many constants are derived from the same inputs. A shrink test asked, for each quantity in three or more editions, whether its stated error fell faster than its edition-to-edition steps. An out-of-sample test fitted the needed multiples on the earlier half of the editions and applied them to the later half.

The planted check built 4,000 shifts in a world where the stated sigma is half the true error and the errors follow a Student t with three degrees of freedom. The true $k_{68}$ is

where $q_3$ is the 0.84135 quantile of that t distribution. On a planted world with stated errors set to half the true error, the method recovered the needed multiple as 2.405 against a true 2.394.c5 The fitted nu was 3.002 against a true 3. The kill condition did not fire.

The honest normal worldDefinition: honest normal world statistics, our control Simulated shifts drawn from a standard normal distribution at the PDG sample size, so that every stated sigma is exactly right, scored the same way as the real shifts. Excludes heavy tails and shared inputs. full definition draws standard normal shifts at the PDG sample size (347) and scores the same shortfall

Over 500 draws the shortfall had mean 0.0005 and 95th percentile 0.040, the top of its null distributionDefinition: null distribution statistics The spread of scores a control gives over many random draws. Its 95th percentile is the bar a real score must clear. Excludes a theoretical distribution assumed rather than drawn from the data. full definition. The real PDG shortfall over all values was -0.110: the PDG tables over-cover, so the preregistered hypothesis that their shortfall exceeds chance failed (p from drawsDefinition: p value (from draws) statistics The share of control draws scoring at or above the real score, with one added to the count and to the number of draws, so it is never zero. Excludes a p value from a textbook formula that assumes a distribution. full definition 1.0).

Table 1 gives every field and Figure 2 draws the same numbers. The earlier 1 sigma interval covered 0.68 of later PDG values that moved (n 227) and 0.66 of later AME values (n 2,213, nearly all of which moved), close to the nominal 0.6827.c1 The 95% intervals on those two coverages are 0.63 to 0.74 for PDG and 0.65 to 0.68 for AME; only 0.3 percent of the AME values did not move. At 2 sigma both fall short of 0.9545, at 0.85 and 0.82.

Covering 95.45 percent of later values took 3.8 times the stated sigma for moved PDG values, 4.0 for AME and 4.9 for CODATA.c2 Student t fits to the nonzero shifts gave nu of 1.67 for PDG, 1.22 for AME and 2.17 for CODATA, close to the Cauchy shape.c3

Over all PDG values, including the 0.35 that never moved, the tables over-cover at 1 sigma (0.79). In the last PDG step, 2024 to 2025, 0.96 of 421 values did not move. Coverage that counts those values says more about which values were carried forward than about how honest the stated errors are.

The CODATA step from 2018 to 2022 covered only 0.47 at 1 sigma. A later reading of this run, made after the lock and recorded as a separate finding, traced most of that step to one input: 99 of the 273 rows moved by 4.2 to 4.7 of their 2018 sigma when the fine-structure constant moved by 4.55 of its own. The CODATA 2022 adjustment reports that move; we did not re-read the report for this paper. The rows are not independent: kept to one row per distinct shift, CODATA has 76 rows, with 1 sigma coverage 0.63 and $k_{95}$ 5.1.

Published Hubble constant values before 1990 covered a proxy truth of 70 km/s/Mpc within 1 sigma in 0.35 of 115 cases.c4 Over all 544 values the needed $k_{95}$ was 5.4 to 7.7 depending on the proxy, the worst field in the set at every proxy. Ten published speed-of-light values from before 1983, scored against the defined value, covered 0.5 at 1 sigma; six of those came from fetched sources and the rest were transcribed, so we give that row no weight.

Removing the top 1 percent of shifts changed 1 sigma coverage by about 0.01 (PDG 0.80, AME 0.67, CODATA 0.86) and lowered $k_{95}$ to 2.8 for PDG, 3.6 for AME and 3.2 for CODATA. The largest single shift, the alpha particle molar mass in CODATA at 74 sigma, comes from the 2019 redefinition of the SI changing the molar mass rows. The largest PDG shift (12.7) and the largest AME shift (silicon-32, 64) were not traced.

Needed multiples did not carry forward in time where values moved. Fitted on CODATA 2002 to 2014 and applied to 2014 to 2022, $k_{68}$ of 0.65 lowered 1 sigma coverage from 0.40 to 0.32, and $k_{95}$ of 1.46 reached 0.79 where 0.9545 was the aim. In AME, 2016 to 2020 hardly moved, and coverage stayed at 0.94.

The shrink test found stated errors falling faster than the steps between editions in 0.66 of 287 CODATA quantities, 0.52 of 104 PDG quantities and 0.25 of 2,227 AME quantities. No general rule that uncertainty shrinks faster than values settle follows from that.

Table 2 sets the forecasts against the outcomes. Of the ten forecast verdicts in the result file, six held.

The pattern matches Bailey [7]: the core of a stated uncertainty is about right and the tails are not. Our fitted nu values, 1.2 to 2.2, sit at or below the heavy end of what that paper describes, and nearer the Cauchy shape. Our Hubble constant result agrees with Chen, Gott and Ratra [6] in kind: they put 95.4 percent limits at 7.0 times the quoted errors against a central value of 71, and we find 6.5 against 70 on a larger version of the same compilation.

In these tables the fault sits in the tails. Bailey draws the same distinction: underestimating every sigma by one factor widens a normal curve without giving it heavy tails. In our tables, the 1 sigma multiple sits near 1 while the 95.45 percent multiple sits near 4. A single scaling of every error bar cannot fix both.

The CODATA step from 2018 to 2022 shows how the tails arise in a table built from shared inputs: one input moves and carries many derived values with it. Henrion and Fischhoff [5] found the reported uncertainties of the constants too small across their history. A correction factor fitted on quiet years will miss the next one, which is what the out-of-sample test found.