The replication crisis and Open Science: reproducibility, preregistration, and the reform of psychological research

Intuition Beginner

In 2015 a team of 270 researchers tried to repeat 100 published psychology experiments. Only 36 came back the same. Effect sizes roughly halved; several vanished entirely. This was the replication crisis — the discovery that many celebrated findings in the published record would not reproduce under careful re-testing.

The crisis had clear causes. Analyzing the same data many ways and reporting only the comparison that "worked" — p-hacking — inflates false positives. Writing the hypothesis after seeing the results — HARKing — dresses an exploratory finding as a confirmation. Journals preferring clean, novel, surprising results created a file drawer of unpublished null findings, distorting the literature toward false effects.

The response was Open Science. Researchers now preregister their hypotheses and analyses before collecting data, share raw data and materials publicly, and publish through registered reports, in which journals accept based on method, not outcome. Large multi-lab collaborations like ManyLabs pool thousands of participants. The crisis surfaced first in psychology but, as Ioannidis showed, applies to all of science.

Visual Beginner

The figure is the signature scatter plot from the Open Science Collaboration (2015): each point is one of the 100 replicated studies, with the original effect size on the horizontal axis and the replication effect size on the vertical axis. Points above the diagonal line fell at least as large as the original; points below shrank. Most land below the line, and about two-thirds fall below the shaded significance threshold entirely.

The visual punchline is the asymmetry. If the literature reflected reality, the points would scatter symmetrically around the diagonal with modest noise. Instead they cluster systematically below it, demonstrating that the original effect sizes were inflated — partly by sampling error, partly by the questionable research practices this unit analyses.

Worked example Beginner

The ego-depletion replication failure. In 1998 Roy Baumeister and colleagues proposed that willpower is a limited resource: after resisting fresh cookies, people gave up faster on an impossible puzzle. The finding became one of the most cited in social psychology — thousands of papers built on it, and it entered popular science as fact.

Step 1. In 2016 Martin Hagger and 22 other laboratories ran a preregistered, multi-site replication with a combined sample of $2141$ participants — roughly ten times the median original study. The analysis plan, hypotheses, and exclusion rules were all posted publicly before any data were collected.

Step 2. The result: an effect size of $d=0.04$ with a $95\%$ confidence interval from $-0.07$ to $0.15$. The interval straddles zero. By every conventional standard, the effect was not statistically distinguishable from no effect at all. The original meta-analytic estimate had been around $d=0.62$.

Step 3. The original authors disputed the replication's choice of manipulation and measure. But the largest, most rigorous test ever conducted of the theory found essentially zero willpower depletion.

What this tells us: a finding can become a textbook staple on the strength of a handful of small, flexible-analysed studies, then evaporate when tested at scale under pre-committed methods. The gap between the inflated original ($d\approx 0.62$) and the preregistered replication ($d=0.04$) is exactly the gap that the Open Science reforms are designed to expose.

Check your understanding Beginner

Formal definition Intermediate+

A scientific finding is reproducible when independent investigators, using the original data and code, can reproduce the reported analyses and arrive at the same numbers; it is replicable when an independent team, conducting a new study following the original method, obtains results of comparable magnitude and direction. The two are distinct: reproducibility is a computational standard, replicability an empirical one. The replication crisis is primarily a crisis of replicability, though failures of reproducibility (errors in analysis code, misreported data) contribute.

Definition (direct, conceptual, and large-scale replication). A direct replication repeats the original method as closely as possible — same manipulation, same measure, same population — to test whether the original finding holds. A conceptual replication tests the same underlying hypothesis using a different manipulation or measure, and addresses the question of whether the construct generalises rather than whether the procedure repeats. A multi-site replication (the ManyLabs model of Klein et al. 2014 ^[Klein2014]) runs the same protocol across many laboratories with a combined sample in the thousands, producing an effect-size estimate with tight confidence intervals and high power to detect even small effects.

Definition (questionable research practices, QRPs). Following Simmons, Nelson and Simonsohn (2011) ^{[Simmons2011]}, questionable research practices are undisclosed choices in data collection and analysis that inflate the rate of false-positive findings. The principal categories:

1. P-hacking — running many analyses of the same dataset (different covariates, different exclusion rules, different dependent variables, different sample sizes) and reporting only those that reach significance.
2. HARKing (Hypothesizing After Results are Known) — formulating the hypothesis after inspecting the data and presenting the resulting post-hoc explanation as a pre-hoc prediction.
3. Optional stopping — collecting data in waves and ceasing collection as soon as a test reaches significance, which inflates the Type I error rate far above the nominal $\alpha$.
4. Selective citation of supportive findings and failure to report dependent variables or conditions that "did not work".

Definition (publication bias and the file drawer). Publication bias is the systematic tendency of journals to publish positive (statistically significant) findings over null findings, producing a published record in which the true effect size is inflated. The unpublished null studies constitute the file drawer (Rosenthal 1979). The distortion is quantitative: if 20 studies test a null hypothesis at $\alpha =0.05$, on average one will be significant by chance, and if only that one is published the literature presents a false effect as established.

Definition (Open Science reforms). The reforms introduced from 2011 onward in response to the crisis:

1. Preregistration — a time-stamped, publicly posted statement of hypotheses, sample size, exclusion rules, and analysis plan, deposited before data collection begins.
2. Registered reports (Chambers 2017 ^{[Chambers2017]}) — a publishing format in which journals provisionally accept a paper based on the method, with publication guaranteed regardless of outcome provided the preregistered plan is followed.
3. Open data and open materials — public deposition of raw data, analysis code, and stimulus materials in repositories such as the Open Science Framework, permitting reanalysis and direct replication.
4. Large-scale multi-laboratory collaboration — the ManyLabs model, in which the same protocol runs across dozens of sites, producing high-powered tests of contested effects.

Counterexamples to common slips

• A failure to replicate means the original finding was false. A direct replication can fail for many reasons — the original population differs from the replication population, the manipulation was implemented differently, or genuine boundary conditions exist. A failure to replicate lowers confidence in the finding but does not, by itself, prove the null. Conceptual replications and meta-analytic synthesis are the tools for adjudicating between "false original" and "context-sensitive effect".

• Preregistration prevents all p-hacking. Preregistration fixes the primary analysis, but researchers retain flexibility in exploratory analyses. The discipline of preregistration is to label which analyses were confirmatory and which were exploratory, so that readers can discount the latter appropriately — not to ban exploration.

• Registered reports lower the quality of published work. The evidence runs the other way: registered reports are reviewed for methodological rigour at stage one, when suggestions can still be incorporated, and the published registered reports tend to have larger samples and better designs than conventionally reviewed papers in the same journals.

• The replication crisis is unique to psychology. Ioannidis (2005) ^{[Ioannidis2005]} established the identical PPV (positive predictive value) framework for biomedicine, and subsequent replication projects in cancer biology (Nosek and Errington 2017; Bayer 2011), economics (Camerer et al. 2016, 2018), and preclinical drug development produced similarly low replication rates. Psychology was the first field to confront the crisis openly, not the only field affected by it.

Key result: the Open Science Collaboration 2015 replication rate of 36% Intermediate+

Result (Open Science Collaboration 2015). Of 100 published psychology studies selected for replication, 36 produced a statistically significant effect in the same direction as the original when replicated by independent teams with substantially larger samples. The mean effect size of the replications was approximately half the mean effect size of the originals, and the subjective-rated similarity of replication to original was substantially lower than the authors' own confidence in their effects before replication.

The 2015 result is empirical, but the deeper question — why the rate is so low — is theoretical. The answer is the positive predictive value (PPV) framework of Ioannidis (2005) ^{[Ioannidis2005]}, which models the probability that a published significant finding reflects a true effect. The framework rests on three quantities:

• $R$, the pre-study odds: the ratio of the number of true relationships tested to the number of null relationships tested;
• $\alpha$, the nominal Type I error rate (conventionally $0.05$);
• $1-\beta$, the statistical power (conventionally $0.20$ to $0.80$ in psychology).

Derivation (Ioannidis 2005 Corollary 1: PPV without bias). Suppose a field tests $\mu$ relationships that are genuinely present and $c$ relationships for which no effect exists, so the pre-study odds are $R=\mu /c$. A study of a true relationship yields a significant result with probability $1-\beta$ (the power); a study of a null relationship yields a significant result with probability $\alpha$ (the Type I error rate). The expected number of true positives in the literature is $(1-\beta )\mu$, and the expected number of false positives is $\alpha c$. The probability that a randomly chosen significant finding reflects a true effect is therefore

$$\mathrm{PPV}(\alpha ,\beta ,R)=\frac{(1-\beta )\mu }{(1-\beta )\mu +\alpha c}=\frac{(1-\beta )R}{(1-\beta )R+\alpha }.$$

This is just Bayes' theorem applied to hypothesis testing. Plug in representative numbers for a field with low pre-study odds: let $R=1/20$ (one in twenty tested relationships is true), $\alpha =0.05$, power $1-\beta =0.50$. Then

$$\mathrm{PPV}=\frac{0.50\times 0.05}{0.50\times 0.05+0.05}=\frac{0.025}{0.075}\approx \mathrm{0.33.}$$

Two findings in three are false. Under the Ioannidis framework, a 36% replication rate is not a scandal — it is the quantitative prediction of a low-powered field with permissive testing on rare effects.

Introducing bias changes the arithmetic but worsens the conclusion. Let $u\in [0,1]$ denote the proportion of analyses that would not have reached significance but are presented as significant through QRPs. Then the false-positive count rises from $\alpha c$ to $\alpha c+u(1-\alpha )c$, and the PPV falls to

$$\mathrm{PPV}(\alpha ,\beta ,R,u)=\frac{(1-\beta )R+u\beta R}{(1-\beta )R+u\beta R+\alpha +u(1-\alpha )},$$

which is Ioannidis's Corollary 2. Even modest bias ($u=0.2$) drives PPV below $0.30$ under otherwise-reasonable assumptions. The combined effect of low power, low pre-study odds, and researcher-degrees-of-freedom bias is a literature in which most published findings are false — which is the title claim of Ioannidis 2005, and the theoretical reason the Open Science Collaboration 2015 replication rate of 36% is the empirically observed value rather than an anomaly. $\square$

Bridge. The Ioannidis PPV framework builds toward 29.13.01 psychometrics, where the question of whether a measurement instrument measures what it claims is itself a question of construct validity that only replicable findings can underwrite, and the same Bayesian logic appears again in 29.13.04 intelligence testing, where the Binet-Simon heritability controversies of the twentieth century turn on exactly the conflation of statistical significance with substantive truth that the PPV framework corrects. The foundational reason is that a significance test tells you the probability of the data given the null, not the probability of the hypothesis given the data — and this is exactly the gap that preregistration, registered reports, and large-scale multi-lab replications are designed to close. The central insight generalises: any inferential discipline that reports only significant results, without correcting for selection, produces a literature in which the reported effect sizes are systematically inflated. Putting these together, the Open Science reforms are not a cultural preference but a direct mathematical response to the PPV arithmetic — the bridge is between the Ioannidis (2005) theoretical analysis and the Open Science Collaboration (2015) empirical confirmation.

Exercises Intermediate+

Advanced results Master

Result 1 (Ioannidis 2005: the PPV framework). Ioannidis ^{[Ioannidis2005]} introduced the positive-predictive-value model of scientific findings, in which the probability that a published significant result reflects a true effect is a function of the pre-study odds $R$, the power $1-\beta$, the Type I error rate $\alpha$, and a bias parameter $u$ quantifying the proportion of non-significant analyses that are nevertheless presented as significant. The framework's central corollary is that under the conditions prevailing in many empirical fields — low pre-study odds, low power, conventional alpha, non-zero bias — most published significant findings are false. The paper appeared in PLoS Medicine and is the most-cited paper in the meta-science literature; its title is the literal statement of its main result.

Result 2 (Simmons-Nelson-Simonsohn 2011: false-positive psychology). Simmons, Nelson and Simonsohn ^{[Simmons2011]} demonstrated by simulation that four commonplace, often-undisclosed researcher degrees of freedom — adding observations until significance, adding a covariate, reporting one of two dependent variables, dropping one of two experimental conditions — are sufficient to inflate the nominal $\alpha =0.05$ to an effective false-positive rate above $0.50$. The paper's methodological contribution was the 21-word solution: a statement of the form "We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study" which, if included in every method section, would disclose the flexibility that p-hacking exploits. The paper is the canonical reference for the QRP-inflation argument and the empirical grounding of preregistration.

Result 3 (Open Science Collaboration 2015: the Reproducibility Project). The Reproducibility Project: Psychology ^[OSC2015] coordinated 270 contributing researchers across 36 laboratories to independently replicate 100 published psychology studies drawn from three high-impact journals in 2008. The headline results: $36\%$ of replications produced a statistically significant effect in the same direction as the original, the mean replication effect size was approximately half the mean original effect size, and correlates of replicability included the strength of the original evidence (larger effect, lower $p$-value, larger sample) and the judged quality of the original method. The paper is the empirical foundation of the public understanding of the replication crisis and the single most-cited replication project in any field.

Result 4 (Hagger et al. 2016: ego depletion). Hagger and 22 collaborating laboratories ran a preregistered multi-site replication of the ego-depletion effect (the claim that self-control is a limited resource), with a combined sample of $n=2141$ — roughly ten times the median original study ^[Hagger2016]. The replication produced an effect size of $d=0.04$, $95\%$ CI $[-0.07,0.15]$, statistically indistinguishable from zero. The original meta-analytic estimate had been $d\approx 0.62$. The Hagger replication is the paradigmatic case of a high-profile social-psychological finding failing to survive large-scale preregistered testing, and it catalysed the adoption of registered reports and multi-lab collaboration in social psychology.

Result 5 (Klein et al. ManyLabs 1-5). The ManyLabs programme (Klein et al. 2014 ^[Klein2014], and sequels through ManyLabs 5) developed the methodological template for large-scale multi-site replication: dozens of laboratories each collect data under an identical, preregistered protocol, the results are pooled meta-analytically, and heterogeneity across sites is examined as a signal of context-sensitivity. ManyLabs 1 replicated 16 classic effects and found that some (e.g. anchoring) replicated strongly and others (e.g. symbolic racism effects) did not. The ManyLabs model demonstrated that multi-site collaboration could produce effect-size estimates with tight enough confidence intervals to adjudicate among the competing explanations of "false original", "context-sensitive effect", and "replication failure due to method differences".

Result 6 (Chambers 2017: registered reports). Chambers ^{[Chambers2017]} introduced and championed the registered report publishing format, in which a paper is reviewed and provisionally accepted on the basis of its method before data are collected, with publication guaranteed regardless of outcome provided the preregistered plan is followed. The format severs the link between outcome and publication that drives publication bias. By the early 2020s, over 300 journals had adopted registered reports, and the format had spread from psychology into clinical medicine, neuroimaging, and parts of economics and political science. The Chambers monograph The Seven Deadly Sins of Psychology (2017) is the canonical statement of the cultural-reform case for Open Science.

Result 7 (Munafò et al. 2017: the reproducibility manifesto). Munafò and colleagues ^[Munafo2017] published a manifesto for reproducible science in Nature Human Behaviour, synthesising the reform agenda into a set of concrete interventions: preregistration, open data and code, registered reports, large-scale collaboration, better statistical training, and reform of incentive structures (hiring, promotion, funding) that currently reward novel significant findings over rigorous replications. The manifesto is the most-cited single statement of the reform landscape and the framework within which most funders' open-science mandates are now written.

Result 8 (Ioannidis 2012-2017: the crisis generalises). Subsequent replication projects established that the crisis is not confined to psychology. The Bayer 2011 internal review found that only about $25\%$ of published preclinical oncology findings replicated in-house. The Reproducibility Project: Cancer Biology (Nosek and Errington, 2017-2021) found that of a sample of high-impact cancer-biology papers, fewer than half replicated on the key original claims. Camerer et al. (2016, Nature) replicated 18 laboratory studies in economics and found a significant effect in the same direction in $61\%$; Camerer et al. (2018, Nature Human Behaviour) replicated social-science studies published in Nature and Science and found a significant effect in the same direction in $62\%$, with effect sizes about half the originals. The same PPV arithmetic that Ioannidis developed for biomedicine applies across empirical disciplines.

Synthesis. The foundational reason the replication crisis emerged in psychology first is that psychology had the methodological infrastructure — shared tasks, standardised measures, an undergraduate subject pool — to attempt large-scale replication at all, and this is exactly why the Open Science reforms could take root there before spreading to biomedicine and the social sciences. The central insight is that the PPV framework of Ioannidis (2005), the QRP-inflation argument of Simmons-Nelson-Simonsohn (2011), and the empirical 36% replication rate of the Open Science Collaboration (2015) are three faces of the same phenomenon: a low-PPV literature whose false positives are inflated by researcher-degrees-of-freedom, observed empirically when tested under preregistered methods. Putting these together with the ManyLabs multi-site model of Klein et al. and the registered-reports format of Chambers, the bridge is between the theoretical analysis of why most findings are false and the institutional reforms — preregistration, open data, large-scale collaboration, outcome-independent publication — that directly target the three PPV levers of pre-study odds, power, and bias. The pattern generalises to cancer biology, economics, and medicine, where the same Ioannidis arithmetic explains the same empirical pattern of low replication, and identifies the Open Science reform package as a single coherent mathematical response rather than a collection of unrelated good practices.

Full argument set Master

Proposition (Ioannidis Corollary 1: PPV without bias). Let a field test $\mu$ true relationships and $c$ null relationships, so the pre-study odds are $R=\mu /c$. Let each study have Type I error rate $\alpha$ and power $1-\beta$. Then, in the absence of bias, the probability that a published significant finding reflects a true relationship is

$$\mathrm{PPV}=\frac{(1-\beta )R}{(1-\beta )R+\alpha }.$$

Proof. Under the stated testing model, each of the $\mu$ true relationships yields a significant result with probability $1-\beta$, so the expected count of true positives in the literature is $(1-\beta )\mu$. Each of the $c$ null relationships yields a significant result with probability $\alpha$ (the Type I error), so the expected count of false positives is $\alpha c$. The total expected count of significant findings is $(1-\beta )\mu +\alpha c$. The probability that a significant finding drawn at random from this pool reflects a true relationship is the ratio of true positives to all significant findings:

$$\mathrm{PPV}=\frac{(1-\beta )\mu }{(1-\beta )\mu +\alpha c}.$$

Dividing numerator and denominator by $c$ and substituting $R=\mu /c$:

$$\mathrm{PPV}=\frac{(1-\beta )R}{(1-\beta )R+\alpha }.\square$$

Corollary (most published findings are false when $R$ is small and power is low). For any $\alpha >0$ and $\beta >0$, if $R<\alpha /(1-\beta )$, then $\mathrm{PPV}<1/2$; that is, most published significant findings are false.

Proof. $\mathrm{PPV}<1/2$ iff $(1-\beta )R<\alpha$, iff $R<\alpha /(1-\beta )$. For conventional $\alpha =0.05$ and power $0.50$, the threshold is $R<0.10$: any field in which fewer than one in eleven tested relationships is true, and which runs studies at $50\%$ power, publishes a literature in which most findings are false. For $\alpha =0.05$ and power $0.20$ (a realistic figure for much of the published psychology literature as of the early 2010s), the threshold is $R<0.0625$: most findings are false even if one in seventeen tested hypotheses is true. $\square$

Proposition (Ioannidis Corollary 2: PPV with bias). Let $u\in [0,1]$ denote the proportion of analyses that would not have reached significance but are nevertheless presented as significant through questionable research practices. Then the bias-augmented positive predictive value is

$$\mathrm{PPV}(\alpha ,\beta ,R,u)=\frac{(1-\beta )R+u\beta R}{(1-\beta )R+u\beta R+\alpha +u(1-\alpha )}.$$

Proof. Under bias $u$, a fraction $u$ of the analyses of true relationships that would have failed to reach significance (expected count $\beta \mu$) are nevertheless presented as significant, adding $u\beta \mu$ to the true-positive pool. The biased true-positive count is $(1-\beta )\mu +u\beta \mu$. Similarly, a fraction $u$ of the analyses of null relationships that would have failed to reach significance (expected count $(1-\alpha )c$) are presented as significant, adding $u(1-\alpha )c$ to the false-positive pool. The biased false-positive count is $\alpha c+u(1-\alpha )c$. The total significant-finding pool is

$$(1-\beta )\mu +u\beta \mu +\alpha c+u(1-\alpha )c.$$

The probability that a significant finding reflects a true relationship is the ratio

$$\mathrm{PPV}=\frac{(1-\beta )\mu +u\beta \mu }{(1-\beta )\mu +u\beta \mu +\alpha c+u(1-\alpha )c}.$$

Dividing numerator and denominator by $c$ and substituting $R=\mu /c$ yields the stated formula. The bias term $u$ enters both pools, but because the null pool $c$ is typically much larger than the true pool $\mu$ when $R$ is small, the denominator grows faster than the numerator, and PPV falls. In the limit $u\to 1$, $\mathrm{PPV}\to R/(R+1)$, independent of $\alpha$ and $\beta$ — the literature becomes pure noise in which the fraction of true findings equals the pre-study base rate. $\square$

Proposition (Simmons-Nelson-Simonsohn QRP inflation, independence lower bound). Let a researcher with access to $k$ independent researcher degrees of freedom conduct $2^k$ analyses of the same dataset, each at nominal level $\alpha$, and report the most significant. Then the family-wise probability that at least one analysis reaches significance, in the absence of any real effect, is $1-(1-\alpha {)}^{2^k}$.

Proof. Under the null hypothesis, each of the $2^k$ analyses yields a significant result with probability $\alpha$, independently (by assumption). The probability that no analysis reaches significance is $(1-\alpha {)}^{2^k}$, and the probability that at least one does is the complement $1-(1-\alpha {)}^{2^k}$. For $k=4$ and $\alpha =0.05$, this gives $1-0.95^{16}\approx 0.56$: a researcher with four researcher degrees of freedom and no real effect can manufacture $p<0.05$ more often than not. The independence assumption gives a lower bound; positively-correlated analyses (the realistic case, since they are computed on overlapping subsets of the same data) inflate the rate further, as Simmons, Nelson and Simonsohn ^{[Simmons2011]} established by simulation. The remedy is preregistration: by fixing the primary analysis in advance, the researcher commits to one of the $2^k$ possible analyses before seeing the data, and the nominal $\alpha =0.05$ is restored. $\square$

Connections Master

• Introduction to psychology and research methods 29.01.01 is the immediate prerequisite and chapter anchor: the survey unit introduces p-values, effect sizes, the WEIRD-sample problem, and the replication crisis in summary form, and the present unit develops the crisis into a full theoretical-and-empirical treatment with the Ioannidis PPV framework, the Open Science Collaboration 2015 result, and the reform package. The flow from 29.01.01 to this unit is the move from "the crisis exists" to "here is why it exists, here is the quantitative theory that predicts its magnitude, and here is the institutional response".

• Psychometrics foundations 29.13.01 provides the measurement framework without which the replication crisis cannot be properly understood. A finding can fail to replicate because the original effect was false, because the replication used a different operationalisation of the construct, or because the measurement instrument was unreliable. The psychometric distinction between reliability (consistency of measurement) and validity (does the instrument measure the construct) is exactly the distinction that adjudicates between "false positive" and "construct-bound effect" when a direct replication fails. The PPV framework assumes a working measurement model; 29.13.01 supplies the theory of that model.

• Intelligence testing: Binet-Simon, Spearman, the Flynn effect, and the IQ controversy 29.13.04 is the historical peer for the conflation of statistical significance with substantive truth. The twentieth-century heritability controversies around IQ turned on the same error that the PPV framework corrects: small studies with selective reporting were treated as definitive evidence for strong claims about group differences, and the resulting literature was later shown to be substantially inflated. The Binet-Simon-to-modern-IQ tradition is a longitudinal case study in how a low-PPV literature can harden into apparent consensus, and the Open Science reforms apply directly to contemporary behaviour-genetic and psychometric research.

• The person-situation debate: Mischel 1968 and the density-distribution resolution 29.08.04 is the conceptual parallel. Mischel's 1968 Personality and Assessment attacked the trait-psychology literature on grounds that anticipate the modern replication crisis: small effects, inconsistent operationalisations, failure to replicate across situations. The resolution of the person-situation debate through density-distribution methodology (Fleeson 2001) is a methodological precedent for the Open Science reform package — rigorous measurement, large samples, and pre-specified analyses resolved a half-century-long empirical dispute that selective reporting had sustained.

Historical & philosophical context Master

The intellectual origin of the replication crisis is John Ioannidis's 2005 paper "Why Most Published Research Findings Are False" ^{[Ioannidis2005]}, published in PLoS Medicine, which modelled the probability that a published significant finding reflects a true effect as a function of pre-study odds, power, Type I error rate, and bias. The paper was largely ignored outside biomedicine for several years; its central claim was treated as a provocative theoretical curiosity. The framework became urgent in 2011, when Simmons, Nelson and Simonsohn ^{[Simmons2011]} published "False-Positive Psychology" in Psychological Science, demonstrating by simulation that four undisclosed researcher degrees of freedom inflate the nominal $\alpha =0.05$ to above $0.50$, and Bem published a methodologically flexible paper claiming evidence for precognition in the same journal — a result that the field could not accept but could not reject using the methods the field accepted.

The empirical confirmation came in 2015, when the Open Science Collaboration, a 270-person team led by Brian Nosek ^[OSC2015], published the Reproducibility Project: Psychology in Science, reporting that only $36\%$ of 100 replicated studies reproduced the original effect. The paper crystallised the crisis publicly and triggered reform across the discipline. Nosek's Center for Open Science, founded in 2013, became the institutional hub of the Open Science movement, hosting the Open Science Framework and driving the adoption of preregistration and registered reports. Chris Chambers's 2017 monograph The Seven Deadly Sins of Psychology ^{[Chambers2017]} made the case for registered reports as the central reform, and by 2020 over 300 journals had adopted the format. Munafò and colleagues' 2017 manifesto in Nature Human Behaviour ^[Munafo2017] synthesised the reform programme into the framework that now underlies most major funders' open-science mandates.

The crisis generalised rapidly. Ioannidis himself extended the framework to cancer biology and genetics; the Reproducibility Project: Cancer Biology began publishing in 2017. Camerer and colleagues replicated laboratory economics (2016) and social-science studies in Nature and Science (2018), finding replication rates around $60\%$ with effect-size halving — the same pattern as psychology. The Hagger et al. 2016 ^[Hagger2016] multi-lab preregistered replication of ego depletion, which found $d=0.04$ against an original meta-analytic estimate of $d\approx 0.62$, became the paradigmatic case of a high-profile finding failing under rigorous testing, and the ManyLabs programme of Klein et al. ^[Klein2014] established the methodological template for large-scale multi-site replication that has since spread across the behavioural sciences.

Bibliography Master

@article{Ioannidis2005,
  author  = {Ioannidis, John P. A.},
  title   = {Why Most Published Research Findings Are False},
  journal = {PLoS Medicine},
  volume  = {2},
  number  = {8},
  pages   = {e124},
  year    = {2005},
  doi     = {10.1371/journal.pmed.0020124},
}

@article{Simmons2011,
  author  = {Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri},
  title   = {False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant},
  journal = {Psychological Science},
  volume  = {22},
  number  = {11},
  pages   = {1359--1366},
  year    = {2011},
  doi     = {10.1177/0956797611417632},
}

@article{OSC2015,
  author  = {{Open Science Collaboration}},
  title   = {Estimating the Reproducibility of Psychological Science},
  journal = {Science},
  volume  = {349},
  number  = {6251},
  pages   = {aac4716},
  year    = {2015},
  doi     = {10.1126/science.aac4716},
}

@article{Munafo2017,
  author  = {Munaf{\`o}, Marcus R. and Nosek, Brian A. and Bishop, Dorothy V. M. and Button, Katherine S. and Chambers, Christopher D. and Du Sert, Nathalie Percie and Simonsohn, Uri and Wagenmakers, Eric-Jan and Ware, Jennifer J. and Ioannidis, John P. A.},
  title   = {A Manifesto for Reproducible Science},
  journal = {Nature Human Behaviour},
  volume  = {1},
  pages   = {0021},
  year    = {2017},
  doi     = {10.1038/s41562-016-0021},
}

@book{Chambers2017,
  author    = {Chambers, Christopher D.},
  title     = {The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice},
  publisher = {Princeton University Press},
  year      = {2017},
}

@article{Hagger2016,
  author  = {Hagger, Martin S. and Chatzisarantis, Nikos L. D. and Alberts, Hamilton and Anggono, Chinmoy O. and Batailler, Constance and Birt, Ann R. and Brand, Richard and Brand, Samantha and Connor, Mark and Conroy, Declan and et al.},
  title   = {A Multilab Preregistered Replication of the Ego-Depletion Effect},
  journal = {Perspectives on Psychological Science},
  volume  = {11},
  number  = {4},
  pages   = {546--573},
  year    = {2016},
  doi     = {10.1177/1745691616652873},
}

@article{Klein2014,
  author  = {Klein, Richard A. and Ratliff, Kate A. and Vianello, Michelangelo and Adams, Reginald B. and Bahn{\'\i}k, {\v{S}}t{\v{e}}p{\'a}n and Bernstein, Michael J. and Bocian, Konrad and Brand, Mark J. and Brooks, Brooke and Brumbaugh, Claudia C. and et al.},
  title   = {Investigating Variation in Replicability: A ``Many Labs'' Replication Project},
  journal = {Social Psychology},
  volume  = {45},
  number  = {3},
  pages   = {142--152},
  year    = {2014},
  doi     = {10.1027/1864-9335/a000178},
}

@article{Camerer2018,
  author  = {Camerer, Colin F. and Dreber, Anna and Holzmeister, Felix and Ho, Teck-Hua and Huber, J{\"u}rgen and Johannesson, Magnus and Kirchler, Michael and Nave, Gideon and Nosek, Brian A. and Pfeiffer, Thomas and et al.},
  title   = {Evaluating the Replicability of Social Science Experiments in {N}ature and {S}cience Between 2010 and 2015},
  journal = {Nature Human Behaviour},
  volume  = {2},
  pages   = {637--644},
  year    = {2018},
  doi     = {10.1038/s41562-018-0399-z},
}

@article{Rosenthal1979,
  author  = {Rosenthal, Robert},
  title   = {The ``File Drawer Problem'' and Tolerance for Null Results},
  journal = {Psychological Bulletin},
  volume  = {86},
  number  = {3},
  pages   = {638--641},
  year    = {1979},
}

@article{Simonsohn2014,
  author  = {Simonsohn, Uri and Nelson, Leif D. and Simmons, Joseph P.},
  title   = {$p$-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results},
  journal = {Perspectives on Psychological Science},
  volume  = {9},
  number  = {6},
  pages   = {666--681},
  year    = {2014},
}