29.01.03 · psychology / intro-methods

Statistical reasoning in psychology: effect size, power, replication crisis

stub3 tiersLean: nonepending prereqs

Anchor (Master): Cohen, J. — The statistical power of abnormal-social psychological research (1962)

Intuition Beginner

Statistical thinking matters in psychology because human behavior varies enormously from person to person and moment to moment. Researchers need tools to separate real patterns from the noise of that variation.

A p-value tells you how surprising your data would be if there were no real effect — but it does not tell you how large or important the effect is. An effect size does that: Cohen's d measures how far apart two groups sit in standard-deviation units.

A study with a large sample can find a statistically significant result for a tiny, meaningless effect. Significance asks "is there something here?" while effect size asks "how much does it matter?". A large study and a small study can give very different significance verdicts for the same effect size.

The replication crisis began around 2011, when psychologists realized many published findings could not be reproduced when other labs tried. Causes included small samples, selective reporting, and intense pressure to publish surprising results.

The reforms that followed are practical: pre-registering hypotheses before collecting data, sharing data openly, running larger samples, and reporting effect sizes with confidence intervals. These changes make findings harder to manufacture and easier for other labs to check.

Visual Beginner

The table below gives Cohen's conventions for the three effect-size families a psychologist meets most often. The numbers are rules of thumb, not laws, but they make "small, medium, large" comparable across studies.

Quantity	What it measures	"Small"	"Medium"	"Large"
Cohen's d	Mean difference in SD units	0.2	0.5	0.8
Pearson r	Strength of linear association	0.1	0.3	0.5
r-squared	Shared variance	1%	9%	25%

The diagram makes the three pillars concrete. The p-value answers a narrow question; the effect size answers the question you actually care about; power tells you whether your study was big enough to find the effect at all.

Worked example Beginner

A team tests whether a tiny color tweak to a website button increases clicks. With 50,000 visitors the tweaked button earns a "statistically significant" 0.1% higher click rate ( $p = .03$ ). The effect size is $d = 0.02$ — essentially zero.

A small therapy trial with 40 patients finds a large drop in anxiety ( $d = 0.9$ ) that just misses significance ( $p = .07$ ). The effect is large and plausibly real, but the sample was too small to rule out chance.

Significance tracks sample size, while effect size tracks importance. A finding can be significant but worthless, or meaningful but not yet significant. Reading both numbers together is the first habit of statistical reasoning.

Check your understanding Beginner

Exercise (easy, multiple choice).

A study reports $p = .04$ . Which statement best describes what this p-value means?

A. The probability that the null hypothesis is true is 4%. B. The probability of getting data this extreme or more extreme, assuming the null hypothesis is true, is 4%. C. The probability that the result will replicate in a new study is 96%. D. The probability that the effect is large is 4%.

Hint

The p-value is computed assuming the null is true. It is a probability about the data under a hypothesis, not a probability about the hypothesis itself.

Answer

Option B.

A p-value is the probability of observing data as extreme as, or more extreme than, what was found, given that the null hypothesis holds. It is not the probability that the null is true (option A), not a replication probability (option C), and says nothing about effect magnitude (option D). Confusing $P (data ∣ H_{0})$ with $P (H_{0} ∣ data)$ is the single most common statistical error in the field.

Exercise (easy, multiple choice).

A study of 10,000 people finds a "statistically significant" effect with $d = 0.03$ . What is the most reasonable conclusion?

A. The effect is practically important because it is significant. B. The effect is real and large. C. The effect is statistically significant but too small to matter much in practice. D. The study was underpowered.

Hint

With a very large sample, even tiny effects reach significance. What does $d = 0.03$ say about practical importance by Cohen's conventions?

Answer

Option C.

An effect of $d = 0.03$ is far below Cohen's "small" benchmark of $0.2$ , so it is negligible in practical terms even though the huge sample made it statistically significant. This is the classic illustration of why significance and importance are different questions. The study was if anything overpowered: it had enough participants to flag a meaningless effect.

Formal definition Intermediate+

Statistical reasoning in psychology rests on null hypothesis significance testing (NHST), a framework inherited from Fisher and Neyman-Pearson. This section defines the moving parts and the standard effect-size families.

Null hypothesis significance testing

NHST frames inference as a choice between two hypotheses: the null hypothesis $H_{0}$ (no effect, no difference, parameter equals zero) and the alternative hypothesis $H_{1}$ (there is an effect). Two kinds of error can occur:

Type I error ( $α$ , false positive): rejecting $H_{0}$ when it is true.
Type II error ( $β$ , false negative): failing to reject $H_{0}$ when $H_{1}$ is true.

The significance level $α$ is set in advance (conventionally $.05$ ) and bounds the false-positive rate. Statistical power is $1 - β$ , the probability of detecting an effect that genuinely exists. A study is underpowered when $β$ is large — typically when power falls below the conventional target of $.80$ .

The p-value and its misconceptions

The p-value is the probability of obtaining data as extreme as, or more extreme than, the observed data, assuming $H_{0}$ is true:

p = P (data \geq observed ∣ H_{0}) .

It is not $P (H_{0} ∣ data)$ , not the probability that the result is due to chance, and not the probability of successful replication. Three misconceptions dominate the literature:

The p-value is not the probability that the null hypothesis is true.
A non-significant result ( $p > .05$ ) is not evidence for the null — it may simply reflect low power.
A significant result does not imply a large or important effect.

Effect sizes

An effect size quantifies magnitude independent of sample size. Three families cover most of psychology.

Cohen's $d$ standardizes the difference between two group means:

d = \frac{x ˉ _{1} - x ˉ _{2}}{s _{pooled}} .

By Cohen's conventions, $d = 0.2$ is small, $0.5$ medium, $0.8$ large. Hedges' $g$ applies a small-sample correction to $d$ .

Pearson's $r$ measures linear association, ranging from $- 1$ to $+ 1$ :

r_{x y} = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2} \sum _{i} ( y _{i} - y ˉ ) ^{2}} .

Conventions: $r = 0.1$ small, $0.3$ medium, $0.5$ large. The coefficient of determination $r^{2}$ reports the proportion of shared variance.

Eta-squared ( $η^{2}$ ) and partial eta-squared ( $η_{p}^{2}$ ) measure the proportion of total (or residual) variance explained by a factor in ANOVA designs. Analogous benchmarks (small $.01$ , medium $.06$ , large $.14$ ) anchor their interpretation.

Confidence intervals

A 95% confidence interval is constructed by a procedure that, over repeated sampling, would capture the true parameter 95% of the time. It expresses the precision of an estimate: a narrow interval means a precise estimate, a wide one an uncertain one. The correct interpretation concerns the procedure, not the probability that a single computed interval contains the parameter.

Statistical power and its determinants

Power is a function of three quantities:

Effect size. Larger effects are easier to detect.
Sample size. More participants raise power.
Alpha. A stricter significance level (smaller $α$ ) lowers power.

For a one-sample $z$ -test with standardized effect $δ = (μ - μ_{0}) / σ$ , approximate power against a one-sided alternative is

1 - β = Φ (δ n - z_{1 - α}),

where $Φ$ is the standard-normal CDF and $z_{1 - α}$ the corresponding critical value. The lesson generalizes: power grows with $d n$ , so halving the effect size quadruples the sample size required to hold power fixed.

Multiple comparisons

Running many tests inflates the chance of at least one false positive. With $m$ independent tests at level $α$ , the family-wise error rate reaches $1 - (1 - α)^{m}$ . Corrections include the Bonferroni adjustment (test each comparison at $α / m$ ) and the false discovery rate (FDR) approach of Benjamini and Hochberg, which controls the expected proportion of false positives among the rejections — a less conservative and often preferable target.

Questionable research practices

Three practices degrade the published record:

Publication bias (the file drawer problem): studies with significant results are published; null results are filed away, distorting the literature toward false positives.
p-hacking: exploiting analyst degrees of freedom — which variables to report, which outliers to drop, when to stop collecting data — until $p < .05$ .
HARKing (Hypothesizing After Results are Known): presenting a post-hoc, data-driven finding as if it were a pre-registered prediction.

Each practice inflates the false-positive rate far above the nominal $α$ and is a primary driver of the replication crisis.

Evidence pattern: diagnosing significance testing gone wrong Intermediate+

The replication crisis is, at its core, an evidence pattern: a systematic discrepancy between what published studies claim and what careful replications find. The diagnosis rests on recognizable statistical signatures.

The signature of low power

Cohen's 1962 survey of the Journal of Abnormal and Social Psychology found that the median study had power below .50 to detect a medium effect. A field running underpowered studies suffers two linked consequences: most true effects are missed (Type II errors abound), and the few that are found tend to overestimate their magnitude, because only the luckiest, largest draws cross the significance threshold. This winner's curse means published effect sizes are systematically inflated.

The signature of publication bias

When published studies are a nonrandom sample of all studies — biased toward significance — two diagnostic patterns appear. A funnel plot of effect size against precision should be symmetric; asymmetry, with small studies clustering at large effects, signals suppressed nulls. p-curve analysis examines the distribution of significant p-values: a right-skewed curve (many low p-values) indicates real effects, whereas a flat or left-skewed curve suggests the significance came from p-hacking rather than signal.

The signature of p-hacking

When researchers exploit flexibility, significant p-values bunch just below $.05$ . A spike in the histogram of p-values between $.04$ and $.05$ — and an unnatural scarcity just above $.05$ — is the fingerprint of selective reporting. The "garden of forking paths" (Gelman and Loken) formalizes why: even without conscious cheating, the many defensible analytic choices guarantee that some path will reach significance.

The corrective pattern: what good evidence looks like

A trustworthy body of evidence shows the opposite signatures: power calculated and reported a priori, pre-registered analyses matched to reported analyses, effect sizes accompanied by confidence intervals, open data and code, and independent replications whose effect sizes match the originals within sampling error. These are not aesthetic preferences; they are the conditions under which the statistical machinery above does what it claims to do.

Exercises Intermediate+

Exercise 1 (easy, multiple choice).

A researcher runs a study with power $= .30$ to detect a medium effect. She finds a non-significant result ( $p = .21$ ). What is the strongest conclusion she can draw?

A. The null hypothesis is probably true. B. There is no effect. C. The study was too underpowered to rule out a real effect. D. The effect is in the opposite direction.

Hint

With power of only .30, what is the probability of detecting a true medium effect?

Answer

Option C.

With power of $.30$ , the study had only a 30% chance of detecting a genuine medium effect — so a 70% chance of a false negative. A non-significant result under such low power is essentially uninformative about whether the effect exists. Concluding "no effect" (options A and B) commits the error of treating absence of evidence as evidence of absence.

Exercise 2 (medium, conceptual).

A researcher reports a between-groups comparison with $\overset{x}{ˉ}_{1} = 24$ , $\overset{x}{ˉ}_{2} = 20$ , and a pooled standard deviation $s_{pooled} = 10$ . Compute Cohen's $d$ , classify it by Cohen's conventions, and explain in one sentence why a reviewer should care about the effect size in addition to the p-value.

Hint

Cohen's $d$ is the mean difference divided by the pooled standard deviation. Compare the result to the 0.2 / 0.5 / 0.8 benchmarks.

Answer

d = \frac{24 - 20}{10} = \frac{4}{10} = 0.40.

This falls between the "small" ( $0.2$ ) and "medium" ( $0.5$ ) benchmarks, so it is a small-to-medium effect. A reviewer should care because a p-value alone conflates magnitude with sample size: with enough participants this $d = 0.40$ effect would be highly significant, while with few participants it might be non-significant, yet the practical size of the effect — the quantity a reader actually wants — is constant at $0.40$ regardless of $n$ .

Exercise 3 (medium, conceptual).

A journal publishes 8 out of 10 submitted studies on a given effect, all of which found $p < .05$ . The 2 unpublished studies both found null results. Explain how this scenario illustrates the file drawer problem, and name one statistical or graphical tool that could detect the resulting distortion in a meta-analysis.

Hint

Which studies enter the published record, and which do not? What would a plot of effect size versus precision look like if some small null studies were missing?

Answer

The file drawer problem is the selective publication of significant results and suppression of nulls. Here, the two null studies — which would have shrunk the pooled estimate toward zero — never enter the literature, so the published record overstates the effect's size and reliability.

A funnel plot would likely reveal this: with the small null studies missing, the plot would show asymmetry (small studies clustering at large effect sizes, no corresponding small studies near zero). An Egger test or a p-curve analysis could test the asymmetry or the credibility of the significant p-value distribution more formally. Any of these tools could flag the distortion.

Exercise 4 (medium, conceptual).

A researcher tests 20 separate outcome variables in a single dataset and reports the 3 that reached $p < .05$ , presenting them as the primary hypotheses. Identify two distinct questionable research practices in this scenario, and state one methodological fix for each.

Hint

One problem is the sheer number of tests; the other is the framing of the analysis after seeing the data. Think about how to control the former and how to prevent the latter.

Answer

Two practices:

Multiple comparisons without correction. Testing 20 outcomes at $α = .05$ yields an expected one false positive purely by chance, and the researcher reported the survivors. The fix is a correction: Bonferroni (test each at $α /20 = .0025$ ) for family-wise control, or FDR (Benjamini-Hochberg) for a less stringent balance.
HARKing. Presenting the 3 surviving outcomes as if they were pre-specified hypotheses is Hypothesizing After Results are Known. The fix is pre-registration: committing to the primary outcomes and analysis plan before data collection, so that the reported hypotheses genuinely preceded the results.

Exercise 5 (hard, conceptual).

Two studies estimate the same effect. Study A reports $d = 0.70$ , $p = .01$ , $n = 40$ per group. Study B (a direct replication) reports $d = 0.25$ , $p = .22$ , $n = 40$ per group. A commentator claims Study B "failed to replicate" Study A. Evaluate this claim by considering the power of Study B to detect the original effect, and explain what the comparison actually shows.

Hint

What power would a study with $n = 40$ per group have to detect $d = 0.70$ ? If that power is modest, how much should a single non-significant replication weigh against the original?

Answer

A study with $n = 40$ per group has roughly 60-70% power to detect an effect as large as $d = 0.70$ , and only about 20-25% power to detect the smaller $d = 0.25$ that Study B actually found. So Study B was underpowered for the smaller effect, but reasonably powered for the original's claimed magnitude.

The non-significant result in Study B is therefore suggestive but not conclusive: there is a 30-40% chance of missing a genuine $d = 0.70$ effect at this sample size. The more informative reading is the effect-size shrinkage — from $0.70$ to $0.25$ — which is the canonical signature of an original study that capitalized on the winner's curse (its point estimate was inflated by a lucky draw). "Failed to replicate" overstates a single underpowered comparison; the honest conclusion is that the original effect was probably overstated, and a larger, pre-registered replication is needed to pin down the true magnitude. Meta-analytic combination of the two studies, rather than a binary verdict on Study B alone, is the appropriate inferential move.

The replication crisis, reform, and alternatives to NHST Master

The statistical machinery above is sound when used as intended. The crisis is that, for decades, it was not. This section follows the empirical exposé of that failure, the reforms proposed in response, and the alternative inferential frameworks vying to replace or supplement NHST.

The reproducibility project

The Open Science Collaboration's 2015 replication of 100 studies — published in Science — coordinated dozens of independent labs to re-run studies sampled from three top psychology journals. The results were sobering: only 36% of the original significant effects reached significance on replication, and the mean replicated effect size was roughly half the original. The shrinkage was systematic: effects did not scatter randomly around the originals; they collapsed toward smaller, more modest magnitudes.

The study also identified predictors of replication. Effects were more likely to replicate when the original study had a lower p-value (not just below .05, but well below), a larger effect size, was conducted by an independent rather than original-author team, and was less surprising on its face. Counterintuitively, the most "exciting" findings — those featured in textbooks and media — were the least likely to survive replication. This pattern is exactly what the file-drawer and winner's-curse signatures predict: surprising results are surprising precisely because they benefited from a lucky draw.

The phenomenon is not unique to psychology. The cancer biology replication projects (Errington et al.) and replications in economics (Camerer et al.) found similar, sometimes worse, shrinkage rates.

The false-positive rate critique

Ioannidis's 2005 essay "Why Most Published Research Findings Are False" formalized the worry with a simple Bayesian-in-spirit argument. If many true effects are small and most studies are underpowered, while the prior probability of any given hypothesis being true is low and bias inflates false positives, then the positive predictive value of a single significant result can fall below 50%. The essay's model made the replication crisis predictable before the large-scale replications confirmed it, and it spread the diagnosis from medicine to every field relying on NHST.

Bayesian alternatives to NHST

Several reformers argue the problem is not just misused p-values but NHST itself, and propose Bayesian frameworks.

Bayes factors (Wagenmakers, Rouder, Morey) quantify the evidence that the data provide for one hypothesis relative to another:

BF_{10} = \frac{P ( data ∣ H _{1} )}{P ( data ∣ H _{0} )} .

A Bayes factor of 10 means the data are ten times more likely under $H_{1}$ than under $H_{0}$ . Unlike the p-value, the Bayes factor can express evidence for the null ( $BF_{10} < 1$ ), distinguishing "no evidence" from "evidence of no effect" — a distinction NHST cannot make.

Kruschke's BEST (Bayesian Estimation Supersedes the t Test) replaces the dichotomous test with full posterior distributions over the parameters of interest, reporting credible intervals and the probability that an effect falls in a region of practical importance. The shift is from "is there an effect?" to "how big is it, and how certain are we?".

Estimation and the "new statistics"

Cumming's "new statistics" advocates abandoning dichotomous testing in favor of estimation: report effect sizes, confidence intervals, and meta-analytic accumulation, and treat each study as one input to a cumulative evidence base rather than a self-contained verdict. The argument is pragmatic rather than philosophical: even without leaving frequentism, emphasizing estimation over significance mitigates the worst abuses and aligns reporting with what readers actually want to know.

Reform infrastructure

A cluster of institutional reforms target the incentives that produced the crisis:

Pre-registration (OSF, AsPredicted) fixes the analysis plan before data are seen, eliminating the flexibility that enables p-hacking and HARKing.
Registered Reports (Chambers, Cortex 2013) move peer review and acceptance to before data collection: the journal commits to publish the result whatever it is, eliminating publication bias for registered studies by construction.
Open data and code (OSF, TOP guidelines) let reviewers and readers verify analyses and re-derive results.
The TOP (Transparency and Openness Promotion) guidelines tier journals by their openness requirements, from disclosure to required data deposit.

Power, optional stopping, and equivalence testing

Three technical reforms round out the picture.

A priori power analysis requires researchers to compute the sample size needed for adequate power (conventionally $.80$ ) to detect the smallest effect of interest, before collecting data. Button et al. (2013) showed that low power not only misses effects but inflates the false-positive rate among reported findings, making power a precondition for reliable inference rather than a nicety.

Optional stopping — peeking at the data as they arrive and stopping once significance is reached — inflates the false-positive rate unless corrected. Sequential analysis (group-sequential designs, alpha-spending functions) provides valid corrected stopping rules that preserve the Type I error rate.

Equivalence testing (TOST — two one-sided tests) inverts the usual logic: instead of testing for an effect, it tests whether the effect falls within a pre-specified equivalence band around zero, allowing researchers to assert "no meaningful effect" with the same machinery NHST uses to assert one. PCA (principal component analysis), used for item reduction in scale construction, addresses a related multiplicity problem: collapsing many correlated questionnaire items into a smaller set of components before testing hypotheses about them.

Connections Master

Introduction to psychology and research methods 29.01.01. The foundational unit introduced descriptive statistics and the logic of inference; this unit formalizes inferential statistics, effect sizes, and the credibility concerns that follow when inference is misused.
Research designs 29.01.02. Every quantity here — power, effect size, confidence interval — is computed for a particular design. The research-designs unit explained why randomization licenses causal claims; this unit explains why even a well-randomized study can mislead if its statistics are mishandled. Power analysis is the direct quantitative link between the two: design choices determine $n$ , which determines power.
Probability and statistics [26.XX]. NHST, the Bayes factor, and confidence intervals are all applications of the probability and inference theory developed in the statistics strand. The replication crisis is a live case study in the gap between a method's mathematical guarantees and its application under perverse incentives.
Logic and critical thinking [49.XX]. The p-value misconceptions (confusing $P (D ∣ H)$ with $P (H ∣ D)$ ) are instances of the base-rate / inverse-probability fallacy studied in the logic strand. HARKing and p-hacking are forms of motivated reasoning, and the reforms are institutional safeguards against them.
Philosophy of science [20.08.XX]. The replication crisis is a real-time episode in the social epistemology of science: it concerns which claims a community is justified in believing, how the publication system distorts the evidence base, and whether self-correction is a structural property of science or an aspirational one.
Learning, memory, and cognition [29.04.01 / 29.05.01]. Many of the most-studied effects in cognition (priming, ego depletion, stereotype threat) were at the center of the replication crisis. The statistical reasoning here is what lets a student evaluate whether a classic finding still stands.
Psychological disorders and therapy 29.09.01. Clinical trials of treatments are evaluated with the same effect-size and power machinery, and meta-analysis of treatment outcomes is the basis for evidence-based practice. Underpowered trials and publication bias directly distort clinical guidelines.

Historical and philosophical context Master

Fisher, Neyman, and Pearson: two incompatible traditions fused

The significance test psychology inherited is a hybrid of two frameworks that their inventors never reconciled. R. A. Fisher developed the p-value as a continuous measure of evidence against the null: a small p-value meant the data were improbable under $H_{0}$ , and the researcher judged whether to be suspicious of it. Jerzy Neyman and Egon Pearson rejected Fisher's evidential interpretation and recast testing as a decision procedure: pre-specify $α$ and $β$ , choose a test that minimizes Type II error for fixed Type I error, and accept or reject mechanically. Fisher's framework had no alternative hypothesis and no power; Neyman-Pearson had no notion of graded evidence.

The "NHST" taught in psychology textbooks is a synthesis neither founder would have endorsed: Fisher's p-value glued onto Neyman-Pearson's accept/reject logic, with $α = .05$ as an inherited convention. This hybrid is the source of much confusion — in particular, the persistent misreading of p-values as evidence measures and as long-run error rates, depending on which part of the hybrid the reader has in mind.

Cohen and the power problem

Jacob Cohen's 1962 survey was the first systematic measurement of statistical power in a psychology literature. Finding that the typical study had less than a coin flip's chance of detecting a medium effect, Cohen spent the next three decades arguing that underpowered research was the field's central methodological failing, and he produced the effect-size conventions and power tables ( $d = 0.2/0.5/0.8$ ) that made power analysis practical. The field largely ignored him for thirty years. The replication crisis vindicated his diagnosis: low power produces a literature of inflated, non-reproducible effects, precisely as he warned.

The crisis timeline

The crisis had several precipitating events. Bem's (2011) publication of ostensibly precognitive findings in a top journal — results that defied replication — dramatized how easily flawed methods could pass review. Simmons, Nelson, and Simonsohn's (2011) "False-Positive Psychology" demonstrated that entirely fabricated significant results could be manufactured from random data using common, defensible analytic choices. The Open Science Collaboration's (2015) large-scale replication then provided the empirical scale: this was not a few bad studies but a field-wide pattern. Ioannidis's (2005) modeling had predicted it; the behavioral data confirmed it.

Reform as a return to discipline

The reform movement — pre-registration, Registered Reports, open data, the new statistics, Bayesian methods — is sometimes framed as a revolution, but it is more accurately a return to discipline. Fisher insisted on randomization before analysis; the reforms insist on pre-registration. Neyman and Pearson insisted on specifying $α$ and $β$ in advance; the reforms insist on a priori power analysis. Cohen insisted on reporting effect sizes; the reforms insist on estimation over dichotomy. What collapsed in the postwar rush to publish was not the statistical theory but the methodological restraint that made it valid. The replication crisis is the bill for that collapse; the reforms are the attempt to pay it.

Bibliography Master

Fisher, R. A., Statistical Methods for Research Workers (Oliver & Boyd, 1925). Introduced the p-value and the analysis of variance as instruments of inference.
Neyman, J. and Pearson, E. S., "On the Problem of the Most Efficient Tests of Statistical Hypotheses," Philosophical Transactions of the Royal Society of London A 231 (1933), 289-337. The Neyman-Pearson lemma and the Type I / Type II error framework.
Cohen, J., "The Statistical Power of Abnormal-Social Psychological Research," Journal of Abnormal and Social Psychology 65(3) (1962), 145-153. The original survey showing that most psychology studies were severely underpowered.
Rosenthal, R., "The File Drawer Problem and Tolerance for Null Results," Psychological Bulletin 86(3) (1979), 638-641. Named publication bias and introduced the fail-safe N.
Cohen, J., Statistical Power Analysis for the Behavioral Sciences, 2nd ed. (Lawrence Erlbaum, 1988). The effect-size conventions and power tables that made a priori power analysis practical.
Benjamini, Y. and Hochberg, Y., "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing," Journal of the Royal Statistical Society B 57(1) (1995), 289-300. The FDR correction as a less conservative alternative to family-wise error control.
Ioannidis, J. P. A., "Why Most Published Research Findings Are False," PLoS Medicine 2(8) (2005), e124. The modeling argument predicting low positive predictive value for underpowered, biased literatures.
Wagenmakers, E.-J., "A Practical Solution to the Pervasive Problems of p Values," Psychonomic Bulletin & Review 14(5) (2007), 779-804. The case for Bayes factors as a replacement for p-value inference.
Cumming, G., "The New Statistics: Why and How," Psychological Science 25(1) (2014), 7-29. The estimation-based alternative: effect sizes, confidence intervals, and meta-analytic accumulation over dichotomous testing.
Simmons, J. P., Nelson, L. D., and Simonsohn, U., "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant," Psychological Science 22(11) (2011), 1359-1366. The p-hacking demonstration.
Bem, D. J., "Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect," Journal of Personality and Social Psychology 100(3) (2011), 407-425. The precognition paper that dramatized how flawed methods pass review.
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., and Munafo, M. R., "Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience," Nature Reviews Neuroscience 14(5) (2013), 365-376. Low power inflates both false negatives and false positives among reported findings.
Chambers, C. D., "Registered Reports: A New Publishing Initiative at Cortex," Cortex 49(3) (2013), 609-610. Introduced Registered Reports, moving peer review and acceptance to before data collection.
Kruschke, J. K., "Bayesian Estimation Supersedes the t Test," Journal of Experimental Psychology: General 142(2) (2013), 573-603. The BEST framework for posterior-based group comparison.
Open Science Collaboration, "Estimating the Reproducibility of Psychological Science," Science 349(6251) (2015), aac4716. The 100-study replication showing only 36% significance and halved effect sizes.
Gelman, A. and Loken, E., "The Statistical Crisis in Science," American Scientist 102(6) (2014), 460-465. The "garden of forking paths" account of how multiplicity produces false positives without conscious p-hacking.
Nosek, B. A., Alter, G., Banks, G. C., et al., "Promoting an Open Research Culture: Author Guidelines for Journals Could Help to Promote Transparency, Openness, and Reproducibility," Science 348(6242) (2015), 1422-1425. The TOP (Transparency and Openness Promotion) guidelines.
Lakens, D., McLorie, W., Isager, P., and Scheel, A. M., "Improving Transparency, Falsifiability, and Rigour by Making Hypothesis Tests Machine-Readable," Advances in Methods and Practices in Psychological Science 1 (2018), and Lakens, D., "Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses," Social Psychological and Personality Science 8(4) (2017), 355-362. The TOST equivalence-testing framework.
Benjamin, D. J., Berger, J. O., Johannesson, M., et al., "Redefine Statistical Significance," Nature Human Behaviour 2(1) (2018), 6-10. The proposal to move the default significance threshold from .05 to .005.
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., and Mellor, D. T., "The Preregistration Revolution," Proceedings of the National Academy of Sciences 115(11) (2018), 2600-2606. The case for pre-registration and Registered Reports as the central reform.