A selection model shrinks the overstatement in published effects but cannot pick which replicate

Published effects that cleared a significance threshold tend to be overstated, and replications of them succeed less often than their own statistics suggest. We asked whether one model of that filter, fitted only to the significant originals of other fields, can predict which effects replicate in a field it has not seen. On 1,222 significant originals in 307 replication projects, pooled from the Reproducibility Project Psychology and the FORRT Replication Database, each of eight fields was held out in turn and forecasts were scored by Brier score. The selection model beat taking the originals at face value by 0.142 in Brier score (95% interval 0.108 to 0.186) and did not beat the base rate, at -0.004 (-0.039 to 0.027).c1 A small logistic model on the original z value and the two sample sizes beat the selection model by 0.037 (0.011 to 0.069).c2 The preregistered kill rule fired, and the claim that the model tells which effects replicate is withdrawn. The reading that significant originals are inflated survives on these data.

The Open Science Collaboration [1] repeated 100 studies from three psychology journals and found that far fewer replications than originals reached significance. Since then, replication results have been gathered into databases. The FORRT Replication Database, FReD, joins earlier collections and pairs original findings with their replications at the level of single effects [4].

One explanation of the gap needs no misconduct. When results are published only if significant, the significance filterDefinition: significance filter (winner's curse) metascience and statistics The selection of published results by a significance threshold, which lets lucky overestimates through more often than underestimates, so the published effects of a set of studies are on average larger than the true ones. Excludes deliberate misconduct, which the filter neither requires nor detects. full definition lets through estimates that happened to land high. Gelman and Carlin [5] named the two resulting errors: the sign of a significant estimate can be wrong (Type S), and its size can be exaggerated (Type M), and both grow as power falls. A replication of the same size then succeeds less often than the original's estimate predicts.

Two families of tools forecast replication. Statistical models of selection, such as z-curve [6], fit the distribution of significant z values and estimate how often exact replications should succeed. Learned models predict from features of the original; Altmejd and colleagues [7] trained such a model on replications in economics and psychology and found, among other things, that interaction effects replicate worse than main effects.

This paper puts a selection modelDefinition: selection model meta-analysis and statistics A model of observed results that writes down both a distribution of true effects and the rule by which results get seen, here the rule that only originals with z above 1.96 in absolute value are seen, and fits both together. Excludes a model that takes the observed results as an unselected sample. full definition in a held-out test against three baselines, with all parameters fitted away from the field being forecast. The question is narrow: does a model of the filter pick out which effects replicate, or does it only set the level? Our search did not find a test of this exact design. That is what the search returned, and it does not show that none exists.

The design was written as a preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition and hash-locked on 31 October 2026 at 08:21 UTC, before any replication outcome was scored. Seen before the lock: column names, counts by category, the number of original statistics that could be converted, and outcomes simulated by the code. The published replication rate of the Reproducibility Project, 36 percent, was known, so the forecast for that field was not blind to its headline.

The locked file names the data, the conversions, the model, the held-out scheme, the baselines, the score, the interval and test methods, the planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition, two data checks, each controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition, the forecasts and the kill conditionsDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition. Nothing was added after the run.

Kill conditions, without discretion:

Two public sets were read through the Open Science Framework's interface: the Reproducibility Project Psychology data [1] [2], released under CC0, and FReD [3], released under CC BY 4.0. Rows of the Reproducibility Project were removed from FReD so that no effect enters twice. Only numeric columns were read from the Reproducibility Project file; names and emails in it were never loaded. Outputs stay at field and project level, and no paper or author is scored or named.

Effects were converted to a correlation from r, d, g, h, standardised beta, odds ratios, eta squared, r squared, f squared, and parsed t, F with one numerator degree of freedom, Z and chi squared with one degree of freedom. Other effect types were dropped. Of 2,164 FReD rows, 1,344 converted. Only originals significant by our conversion entered, since the model conditions on that filter: 201 FReD rows and 18 Reproducibility Project rows were dropped as not significant. That left 1,222 effects, 79 from the Reproducibility Project and 1,143 from FReD, in 307 project clusters, the largest with 156 rows.

Fields are coarse buckets of our own: seven groups of FReD rows (cognitive, economics, management, marketing, other psychology, social and other) and the Reproducibility Project as an eighth. Which effects got replicated was itself selected by the people who chose to replicate them. The results describe this set of replicated effects, not published work at large.

Each effect is put on the Fisher zDefinition: Fisher z statistics The transform of a correlation r to half the natural log of (1 + r) over (1 - r), on which the sampling error is close to normal with standard error 1 over the square root of n minus 3. Excludes the z statistic of a test, which is an effect divided by its standard error. full definition scale, with a standard error from the sample size $n$:

An original enters only if $y/s$ exceeds 1.96 in absolute value. A replication succeeds if its own $y/s$ exceeds 1.96 in the original's direction. The same rule is applied to every row, whatever the database's own label says.

The true effect $\theta$ of an original is either exactly zero, with weight ${\pi }_0$, or drawn from a normal distribution with mean zero and spread $\tau$:

where $\delta$ is a point mass at zero and $\varphi (x;m,v)$ the normal density with mean $m$ and variance $v$. The estimate is normal around the true effect, so an original's density before selection is $f\left(y\right)=\int g\left(\theta \right)\varphi (y;\theta ,s^2)d\theta$. Because only significant originals are seen, each original's likelihood is divided by the chance $w\left(s\right)$ that an estimate with its standard error would be significant:

The parameters ${\pi }_0$ and $\tau$ are fitted by maximum likelihood on the training fields. The weight ${\pi }_0$ is a mixture weight under an assumed prior. It is not the share of false hypotheses, and it measures no misconduct or intent.

For original $i$, let $h_i\left(\theta \right)$ be the density of the true effect given the original's estimate, and ${\psi }_i\left(\theta \right)$ a replication's statistical power at its own sample size, meaning the chance that it succeeds by the rule above when the true effect is $\theta$. The forecast is

Five forecasts compete with it, each built only from training rows. The face-value forecastDefinition: face-value forecast forecasting, our usage A forecast that takes each published original effect as the true effect and predicts replication from the power a replication of the given size would have against it. Excludes any correction for how the original was selected. full definition sets $\theta$ equal to the original estimate. The base-rate forecastDefinition: base-rate forecast forecasting A forecast that gives every effect in a held-out field the same chance of replicating, the share that replicated in the training fields. Excludes any use of an effect's own statistics. full definition gives every row the training success rate. A logistic regressionDefinition: logistic regression statistics A model in which the log odds of success are a weighted sum of predictors, here the original z value, the log of the original sample size and the log of the replication sample size, fitted by maximum likelihood on the training fields. Excludes any model of how originals were selected. full definition uses the original z value and the logs of the two sample sizes. A normal prior with no spike drops the point mass. A transport term lets the replication's true effect differ from the original's by a normal amount whose variance $h$ is fitted on training replications.

Forecasts are scored by the Brier scoreDefinition: Brier score forecast verification The mean squared difference between a forecast probability and the outcome coded 1 or 0, from 0 for a perfect forecast to 1 for a confident wrong one. 0.25 is what a forecast of one half always earns. Excludes a ranking measure such as AUC, which ignores whether the probabilities are at the right level. full definition over $N$ held-out rows with outcomes $o_i$ of 1 or 0:

In a leave-one-field-out testDefinition: leave-one-field-out test statistics, cross-validation A test in which each field in turn is held out, every parameter is fitted on the other fields, and forecasts are scored only on the held-out field. Excludes scoring forecasts on the rows that were used to fit them. full definition each of the eight fields is forecast with every parameter fitted on the other seven, and the held-out scores are pooled. Skill over the base rate is $B_{\mathrm{base}}-B_{\mathrm{sel}}$, so a positive value favours the selection model.

Each comparison is a difference in pooled Brier score, the competitor's minus the selection model's. Its 95% intervalDefinition: 95% interval statistics The range that holds the middle 95 percent of the bootstrap or control draws for a number. We report it beside the number. Excludes a statement that the true value lies inside with 95 percent probability. full definition comes from a project-cluster bootstrapDefinition: project-cluster bootstrap statistics An interval made by recomputing a statistic many times, here 2,000, on samples of whole replication projects drawn with replacement, so rows from one project stay together. Excludes resampling single rows as if they were independent. full definition with 2,000 draws. Its p value comes from a sign-flip permutation testDefinition: sign-flip permutation test statistics A test that sums the per-project differences between two forecasts' scores, then flips the sign of each project's difference at random many times, here 4,000, to see how often a sum as large arises with no true difference. Excludes a test that treats rows inside one project as independent. full definition over projects with 4,000 draws, and the four comparisons are adjusted together with the Benjamini-Hochberg correctionDefinition: Benjamini-Hochberg correction statistics, multiple testing A rule that adjusts a set of p values so that, among the tests called significant, the expected share of false calls stays at or below a chosen rate q, here 0.05. The adjusted values are called q values. Excludes the stricter Bonferroni rule, which bounds the chance of any false call. full definition. The prior's shape was compared by BICDefinition: BIC statistics, model comparison The Bayesian information criterion, minus twice the maximised log likelihood plus the number of parameters times the log of the number of rows. Lower is better, and a difference of 10 is the usual bar for strong preference. Excludes a test of whether either model is right. full definition on all originals against a single normal and a two-normal mixture.

The planted check used the real sample-size pairs with outcomes simulated from the model at three known settings of ${\pi }_0$ and $\tau$: 0.5 and 0.25, 0.05 and 0.35, 0.8 and 0.3, five datasets each. Across the 15 planted datasets the fit recovered ${\pi }_0$ with mean absolute error 0.051 and the success rate with mean error 0.009, and the selection model beat face value in every one.c3 The rules were an error below 0.1, a rate error below 0.03 and a win in at least 90 percent. The fit was weakest at ${\pi }_0$ of 0.05, where two of five datasets gave estimates near 0.18.

Two data checks tested the conversions. The Reproducibility Project rows we kept carried 34 reported successes against the 35 published, within the rule of 4. FReD rows labelled successful by the database had a success rate by our rule of 0.86, and rows labelled failed 0.15, a gap of 0.71 against a rule of at least 0.3. By our own uniform rule, 27 of the 79 kept Reproducibility Project rows succeeded.

Two controls give the null distributionDefinition: null distribution statistics The spread of scores a control gives over many random draws. Its 95th percentile is the bar a real score must clear. Excludes a theoretical distribution assumed rather than drawn from the data. full definition of skill over the base rate. Shuffling replication outcomes across rows (500 draws) gave a 95th percentile of -0.110. Shuffling original statistics across rows while keeping each replication's size (200 draws) gave -0.092 and measures how much skill comes from replication size alone. Both sit far below zero, because a confident forecast paired with the wrong outcome is punished hard. Beating them shows that the real pairing carries signal. The base-rate comparison is the test that decides.

Table 1 and Figure 1 give the pooled held-out Brier scores. The selection model at 0.260 sits next to the base rate at 0.256, far ahead of face value at 0.402 and behind the logistic model at 0.223.

Table 2 gives the four preregistered comparisons. The selection model beat taking the originals at face value by 0.142 in Brier score (95% interval 0.108 to 0.186) and did not beat the base rate, at -0.004 (-0.039 to 0.027).c1 A small logistic model on the original z value and the two sample sizes beat the selection model by 0.037 (0.011 to 0.069).c2

By field (Table 3, Figure 2), skill over the base rate was positive in three of the six fields the run counted as large (cognitive, marketing, other, other psychology, the Reproducibility Project and social; economics with 16 rows and management with 82 were left out), short of the preregistered target of 75 percent. The project counts by field sum to 342, more than the 307 clusters, because a cluster with rows in two fields is counted in each. The model's predicted rate sat above the observed rate in seven of the eight fields and below face value in all eight. For the Reproducibility Project the model predicted a success rate of 0.50 against 0.34 observed, where face value predicted 0.83; in social psychology it predicted 0.65 against 0.41, where face value predicted 0.89.c4

The forecasts were also miscalibrated at the top, as Figure 3 shows. In five bands of equal count, calibrationDefinition: calibration forecast verification Agreement between forecast probabilities and observed frequencies: among effects given a chance near 0.7, about 70 percent should replicate. Checked here in five bands of equal count. Excludes discrimination, the separate question of whether higher forecasts go with more successes. full definition was close in the lowest two and overconfident above: effects given a mean chance of 0.97 replicated at 0.57, and effects given 1.00 replicated at 0.83.

The prior's shape did not hold up. The spike-and-normal prior beat a single normal by 192 in BIC (4,946.8 against 5,138.4), passing its rule of 10. A two-normal mixture beat the spike by 289 (4,657.5). On all originals the fitted ${\pi }_0$ was 0.82 and $\tau$ 0.51, and that description is a poor one. The transport variance $h$ hit the top of its grid, 1.0 on the Fisher z scale, in every fold.

On the 701 rows whose effect type carries a sign, the selection model scored 0.273, face value 0.402 and the base rate 0.256. The ranking of these three is the same as in the full set. With each project weighted equally, a preregistered sensitivity analysis with no decision rule attached, the selection model scored 0.244 against the base rate's 0.256 (Table 1), so the point comparison with the base rate changes sign when projects are weighted equally. No interval was computed for that version, and the preregistered verdict uses the row-weighted score.

Real skill over the base rate was -0.004, above the 95th percentiles of both shuffle controls (-0.110 and -0.092; p 0.002 and 0.005). Table 4 sets the forecasts written before the run against the outcomes.

The result splits the selection reading in two. The part about level holds: forecasts that ignore the filter are badly overconfident, and modelling the filter brings the Brier score down from 0.402 to 0.260. This is the pattern Gelman and Carlin [5] describe, measured here as forecast error on held-out fields.

The part about picking effects fails. A forecast that ignores every effect's statistics and uses the training success rate does as well. On Brier score, the model's spread of forecasts across effects adds nothing over the base rate, and its top bands are overconfident. The logistic model, which uses the same original statistics without a model of selection, does better. A selection model with a rigid prior shape spends its freedom on the level and leaves the ranking to the prior, and the prior fits poorly by BIC.

The transport term points the same way. Allowing the replication's true effect to drift from the original's improved the score to 0.232, and its variance ran to the top of the grid in every fold. The gap between originals and replications on these data is larger than the selection model with a fixed true effect accounts for. Hidden moderators, differences in fidelity, real change over time and design differences ignored by the Fisher z standard error could all add to it, and the original statistics cannot tell them apart.

For practice, this narrows what a selection model is good for. It can set a sober expected replication rate for a body of work. Altmejd and colleagues [7] and z-curve [6] aim at related targets by other routes; this paper does not compare against them.