More variables raise prediction skill, yet most targets stall far below their noise ceiling

A common hope says that measuring enough things about a system lets you predict it. We tested that hope on public data against the best score each target's own measurement noise allows. Variables were added in random orders and the model refitted and scored on held-out cases, for six targets with a measurable ceiling: tree growth, next-beep mood, music valence and arousal ratings, forecaster accuracy and a similarity task. Held-out skill rose with added variables in every target and beat a shuffled-target control in every target.c1 Only the similarity task, whose variables were learned from the same task, reached its ceiling (0.985 of the way); mood, valence, arousal and forecaster accuracy stopped at 0.65, 0.54, 0.46 and 0.18, and the tree ceiling estimator failed.c2 The largest gains came from a different kind of variable: past accuracy lifted forecasters from 0.131 to 0.360 and prior growth lifted trees from 0.396 to 0.639.c3 Both preregistered forecasts for the claim failed. Two toys with every variable known set hard limits: chaos returns forecast time only logarithmically in precision, and a published forecast that people react to can defeat itself.

How predictable an outcome is, and how much of that predictability a model captures, are two separate questions. Salganik and colleagues [1] gave 160 teams a rich birth-cohort dataset and asked them to predict six life outcomes; the best predictions were only slightly better than a simple benchmark, and error depended more on which family was predicted than on the method. That study measured how far models got. It did not set the result against a ceiling from the outcome's own noise.

In neuroscience the noise ceilingDefinition: noise ceiling measurement theory and computational neuroscience The best held-out score any model could reach on a target, given how far repeated measurements of that same target disagree. It is estimated from the data's own repeats. Excludes the best score reached so far by any model, and a bound set by missing variables. full definition is routine: the best held-out score any model could reach on a target, given how far repeated measurements of that same target disagree. Lage-Castellanos and colleagues [2] compare ways to estimate it for brain responses. Hebart and colleagues [3] reported that their similarity model explained most of the explainable variance in human odd-one-out choices, against a ceiling set by repeated triplets.

For forecasters, Himmelstein, Atanasov and Budescu [5] found that psychometric traits predict future accuracy when little track record exists and that past accuracy takes over as it accumulates. That points to one kind of variable that may close a gap others leave open.

We asked whether held-out skill reaches the noise ceiling when enough measured variables are added, across unrelated domains in one design. Searches run on 9 November 2026 found the work above, and a preprint on the PISA 2022 school assessment [10] that reads predictive R2 against a ceiling set by measurement noise inside one domain. A search for a cross-domain comparison of skill curves against noise ceilings returned nothing closer; that is what the search returned, and it does not show that no such study exists. The decision this informs is where effort goes when a prediction stalls: more of the same variables, a different kind of variable, or acceptance of a hard limit.

The preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition, a dated plan hash-locked before any result is seen, was written on 29 October 2026 before any model in the study was fitted to real data. Data caches were built first and printed row counts only. Several of the datasets had been used in earlier studies of ours, which the plan states so that it is not mistaken for blindness.

The plan fixed the targets, variables, splits, models, ceilings and the rule for each verdictDefinition: verdict research method (our usage) The pass or fail of one preregistered rule set against the run's numbers, read off by code. Excludes an interpretation written after reading the numbers. full definition, the pass or fail of one preregistered rule set read off by code. The predictions were:

The plan also held an identification test on a private image collection. That test, and the seventh P2 target built on the same collection, are left out of this paper; the P2 count below includes that target because the verdict was computed over seven.

The first run, on 29 October 2026, gave forecaster accuracy a held-out R2 of -1.686. The cause was a pipeline fault: 84 of the 1,349 forecaster variables were nearly constant among the training forecasters, so their scaled values in held-out forecasters became enormous. A second preregistration file, hashed after the first run had been read, made one change: every scaled variable is clipped to the range -5 to 5. Domains, splits, seed, models and verdict rules stayed the same. The second run is the primary result. It is a corrected rerun and not a blind one. Outside forecasters, the two runs agree within 0.01.

Five public sources supply six targets with a ceiling and one without:

For each target the variables were added in 8 random orders and one informed order (by absolute training-set correlation with the target), at about 14 counts from one variable to all of them. At each count the model was refitted on training cases and scored on held-out cases. Held-out skillDefinition: held-out skill statistics and machine learning A model's score on cases kept apart from fitting: R2 for numbers, the area under the ROC curve for yes-or-no outcomes, and accuracy for triplet choices. Excludes the score on the data used to fit the model. full definition is R2 for numeric targets, AUCDefinition: AUC statistics (signal detection) The area under the ROC curve: the chance that a randomly chosen positive case is ranked above a randomly chosen negative one, with 0.5 for guessing. Excludes accuracy at any one threshold. full definition (the chance a random positive case is ranked above a random negative one) for tree death, and accuracy for triplets.

Models were ridge regressionDefinition: ridge regression statistics Least squares with a penalty on the size of the coefficients, chosen by leave-one-out error, so that many variables can be fitted without chasing noise. Excludes non-linear models such as boosted trees or neural networks. full definition with a leave-one-out penalty for numeric targets, L2 logistic regression for tree death and a conditional logit for triplets. Medians, scaling and clipping came from the training cases only. Held-out cases were whole plots for trees, each person's last 30 percent of beeps for mood, 30 percent of artists for music, 30 percent of forecasters, and between-subject repeated triplets for the similarity task.

Each ceiling comes from the target's own repeats:

The number of raters was not stated in the files we had; $n$ = 10 was assumed, with $n$ = 5 reported beside it.

The share of ceilingDefinition: share of ceiling our usage Held-out skill above chance divided by the noise ceiling above chance. 1 means the model scores as well as the measurements allow. Excludes the raw skill, which is not comparable across targets. full definition puts targets on one scale, with $s$ the held-out skill, $s_0$ its chance level and $c$ the ceiling:

For the four numeric targets with a valid ceiling, one minus R2 splits into measurement noise, $1-c$, and reliable variance the variables do not reach, $c-s$.

A saturating curveDefinition: saturating curve curve fitting The curve s(k) = a k / (k + h) fitted to skill above chance against the number of variables k, where a is the level it approaches. Excludes a forecast of skill with variables not in the dataset. full definition was fitted to skill above chance against the number of variables $k$:

It was fitted to the mean random-order curve, and to each of the 8 orders separately; the spread of $a$ over the orders gives the plateau's interval. The fit bound was 2.0, and for curves that had not flattened the upper end of the interval reached it.

Each target had a strong variableDefinition: strong variable our usage in this study A variable of a different kind from the base set, named for each target before the run: past behaviour of the same unit, or a human rating of a related quantity. Excludes one more variable of the kind already in the base set. full definition named before the run, of a different kind from the base set: prior diameter growth for trees, the person's mean training happiness for mood, the other dimension's human rating for music, and accuracy on the other half of the questions for forecasters.

The Lorenz-63 system was run with an exact model from 100 states on its attractor, with the starting state measured with error $\delta$ per coordinate from ${10}^{-1}$ to ${10}^{-12}$. The forecast horizonDefinition: forecast horizon dynamical systems (chaos) The time until a forecast from a slightly wrong starting state differs from the truth by more than a set tolerance. In a chaotic system it grows with the logarithm of the starting precision. Excludes errors from a wrong model, which the toy does not have. full definition is the first time the error norm exceeds 1. Theory gives the slope per tenfold gain in precision as

with $\lambda$ = 0.906 the largest Lyapunov exponent.

The crowd toy tests a reflexive forecastDefinition: reflexive forecast economics and social science A published forecast that the forecast population reads and reacts to, so that publishing it changes the outcome it predicts. Excludes a forecast kept private from the people it describes. full definition, one that the forecast population reads and reacts to. Each of $N$ members attends when $u_i+\beta (c-F)>0$, where $u_i$ is the member's own taste, $c$ a fixed threshold and $F$ the published forecast of the attending share; $\beta$ sets how strongly members react. The forecaster measures $k$ members' tastes exactly. An unpublished forecast is never seen; a naive one is the unpublished forecast published anyway; a fixed-point forecast solves for the share that would attend given the forecast itself.

Every target was refitted on shuffled training labels 20 times and scored on the real held-out labels. This controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition removes the link between variables and target. The 95th percentile of those 20 draws, the top of the null distributionDefinition: null distribution statistics The spread of scores a control gives over many random draws. Its 95th percentile is the bar a real score must clear. Excludes a theoretical distribution assumed rather than drawn from the data. full definition, is the bar each real score must clear. The similarity control starts its weights at 1, so a fit that stops early keeps the raw embedding similarity, and its bar sits above chance for that reason.

The Lorenz toy is a planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition for the chaos limit, since its answer is known from theory. A deliberate overfit forms a second check: 150 forecasters fitted by plain least squares with up to all 1,349 variables, which should fit the training data almost perfectly and fail held out.

The tree-growth ceiling estimator failed. It returned -3.96, because a few gross shrinkages dominate the mean of squared negative increments. It is kept as preregistered, and the tree target counts as not at ceiling.

Held-out skill rose with added variables in every target and beat a shuffled-target control in every target.c1 The highest control bar among the numeric targets was 0.013 R2; tree death's was 0.531 AUC against 0.686, and the similarity task's was 0.439 accuracy against 0.642 on its validation triplets.

Table 1 and Figure 1 give the curves against their ceilings. Shares follow eq. 2; the music ceilings assume 10 raters; the last column is the shuffled control's 95th percentile. Only the similarity task, whose variables were learned from the same task, reached its ceiling (0.985 of the way); mood, valence, arousal and forecaster accuracy stopped at 0.65, 0.54, 0.46 and 0.18, and the tree ceiling estimator failed.c2

For mood, the first informed variable, current happiness, gives 0.501 of the 0.521. For forecasters the curve has flattened: the plateau fit gives 0.137 with interval 0.118 to 0.155, far below the ceiling of 0.714. With 5 raters assumed instead of 10, the music ceilings fall to 0.642 for valence and 0.725 for arousal, so the music shares depend on a count we do not have.

Figure 2 shows where the variance left after the full model sits. Outside the similarity task, the larger part is reliable signal that the measured variables do not reach, not noise.

The largest gains came from a different kind of variable: past accuracy lifted forecasters from 0.131 to 0.360 and prior growth lifted trees from 0.396 to 0.639.c3 The other dimension's rating added 0.082 to valence and 0.092 to arousal; mood's training mean added 0.032 and tree death's prior growth 0.015 AUC. Only forecasters moved by the P2 rule. For trees the gain was 0.243, but the plateau interval for tree growth ran to the fit bound of 2.0, so the rule could not pass; the preregistration's addendum records that defect and keeps the rule.

One of six targets reached 0.8 of its ceiling, against a forecast of at least four, and one of seven plateaus moved, against a forecast of at least three.c4 Both are honest nullsDefinition: honest null research method (our usage) A working test that found no effect beyond a stated bound, a forecast that failed, or an error of our own found and fixed, reported with the same weight as a positive result. Excludes a void run, and a result left unpublished because it was negative. full definition of the claim's forecasts, reported with the same weight as the passes.

In the Lorenz toy the forecast horizon grew 2.51 time units per tenfold gain in precision, against 2.54 from theory, with linear R2 0.99994; twelve levels of precision, from one decimal digit to twelve, moved the horizon from 1.2 to 28.7 time units.c5 The gain per decade over the last three decades was 1.02 times that over the first three. Figure 3 shows the line.

In the crowd toy with steep reaction ($\beta$ = 20) and 1,000 members, an unpublished forecast with every member measured scored R2 1.00, the same forecast published naively scored -1.55, and a fixed-point forecast that solves for the reaction scored 0.9999. Measuring 100 of the 1,000 members cut the fixed-point forecast to 0.40 while the unpublished one scored 0.98. With identical members there is no fixed point, and the best published forecast scored -0.02.

In the overfitting check, plain least squares with 150 variables on 150 forecasters fitted the training data at R2 0.96 and scored -135,817 held out. Ridge on shuffled targets stayed at or below -0.005 at every count, and ridge on real targets reached 0.067 with all variables. P5 failed on its own wording, because it asked for a training R2 of at least 0.99 at 150 variables and got 0.96; every other rule in it held.

The direction of the hope holds: skill grows with measured variables in every target and always beats the shuffled control. The strong form fails. In four of the five targets with a valid ceiling, the model stops between 0.18 and 0.65 of the way there, and the remainder is mostly reliable signal, not noise.

The exception teaches the most. The similarity variables were learned from the same triplet corpus, so they were built for the target, and they reach the ceiling. Our 0.660 sits close to the ceiling near 0.67 that Hebart and colleagues reported for an earlier version of the task [3], which we read at the level of abstracts. Variables chosen because they were available, not because they were fitted to the target, stall.

The gaps closed most when a different kind of variable arrived. For forecasters this agrees with Himmelstein, Atanasov and Budescu [5]: past accuracy beats the trait battery. For trees, prior growth beats size, site and stand. A target's own history carries signal its present description misses.

Salganik and colleagues [1] found that many methods on rich data barely beat a simple benchmark. Our ceilings add a reading of that kind of result: a stall well below the noise ceiling says the missing information is in the world and not in the measurement error, so a different measurement can still help.

The toys bound the hope from the other side. With an exact model, chaos turns each tenfold gain in precision into a fixed slice of forecast time. A published forecast that people react to can be right only when the forecaster can solve for the reaction, and that solution degrades fast when most of the crowd is unmeasured.

The tierDefinition: tier our publication register How ready a finding is to publish: A candidate, B signal worth expanding (B* strong), C internal, D honest null worth reporting, E not reliable as it stands. Excludes a measure of how large or how true an effect is. full definition B* places this as a strong signal worth expanding. A rerun with non-linear learners and a preregistered tree ceiling on fresh splits would raise it.