45.04.10 · mathematical-statistics / 04-asymptotic-statistics

Asymptotic Consistency of the Bootstrap

shipped3 tiersLean: none

Anchor (Master): van der Vaart 1998 Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics) Ch. 23 (the bootstrap: §23.2 the empirical bootstrap and conditional convergence in probability, §23.2.1 the multiplier/exchangeable-weights bootstrap, the delta-method bootstrap for Hadamard-differentiable functionals); Bickel & Freedman 1981 (Ann. Statist.) (the Mallows-metric consistency proof and its failures); Hall 1992 The Bootstrap and Edgeworth Expansion (Springer) Ch. 3-4 (second-order accuracy of the bootstrap-t)

Intuition Beginner

You have one sample of data, and you computed one number from it — an average, say. That number would have come out a little different if you had drawn a different sample, and you want to know how much it wobbles. The honest way to find out is to go back and collect many fresh samples, compute the number on each, and watch the spread. But you cannot: you have only the one sample you already paid for.

The bootstrap is a clever substitute. It treats your sample as if it were the whole population, and draws new "samples" from it — by picking observations at random, with replacement, until you have a new batch the same size as the original. Some real observations get picked twice, some not at all. You compute your number on each fake sample. The spread of those numbers stands in for the spread you would have seen across real fresh samples.

The promise is that, when you have enough data, the resampled spread matches the true spread closely. That promise is what "consistency" means, and it is what this unit certifies — together with the cases where the promise quietly fails.

The one-sentence takeaway: the bootstrap fakes the experiment of redrawing your whole sample by redrawing from the sample you have, and for smooth quantities with enough data the fake spread reliably matches the real one.

Visual Beginner

Picture three rows. The top row is the unreachable ideal: many fresh samples from the real population, each giving one dot on a number line, the dots forming a bell. The middle row is what you actually have: one sample. The bottom row is the bootstrap: from that one sample you draw many resamples — each a reshuffle with some repeats — and each gives a dot, and those dots also form a bell. The bootstrap claim is that the bottom bell, centred at your sample's number, has the same shape and width as the top bell, centred at the truth.

what you wish you could do	what the bootstrap does instead	when the two match
draw many fresh samples from the population	draw many resamples from your one sample	the estimate is smooth and the sample is large
measure the true spread of the estimate	measure the spread across resamples	the quantity is an average or a gentle function of one
(impossible with one dataset)	(always possible by computer)	not for sharp edge quantities like the maximum

The last row is the warning. The bottom bell copies the top bell for averages and smooth functions, but for a quantity that depends on the single most extreme observation, the copy is wrong.

Worked example Beginner

We use a tiny sample so every resample can be checked by hand. Your data are four measurements: $2, 4, 4, 6$ . The number you care about is the average, which for this sample is $(2 + 4 + 4 + 6) /4 = 4$ . You want a feel for how much that average would wobble, using the bootstrap.

Step 1. The recipe. Draw four observations at random, with replacement, from ${2, 4, 4, 6}$ . Each draw is equally likely to land on any of the four slots, so a $4$ is twice as likely as a $2$ . Compute the average of the resample. Repeat.

Step 2. A few resamples by hand. Resample ${4, 4, 4, 6}$ has average $4.5$ . Resample ${2, 2, 4, 6}$ has average $3.5$ . Resample ${6, 6, 4, 4}$ has average $5$ . Resample ${2, 4, 4, 4}$ has average $3.5$ . Each is a number near $4$ , scattered above and below.

Step 3. Read off the spread. Across many such resamples the averages pile up around $4$ with a typical wobble. The variance of one resampled draw equals the sample variance of the data divided by the size: the data values have spread, and dividing by $4$ gives the wobble of the resample average.

Step 4. The number. The sample values have an average squared deviation from $4$ of $(4 + 0 + 0 + 4) /4 = 2$ . So one resample average has variance about $2/4 = 0.5$ , and a typical wobble of $0.5 \approx 0.71$ .

Step 5. What this tells us. Without ever collecting a second dataset, you read a standard error of about $0.71$ for the average straight off the resampling. The bootstrap turned "how would this number wobble across fresh samples" into a computation you can run on the one sample you have — and for an average, the answer it gives is the right one as the sample grows.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_{1}, \dots, X_{n}$ be i.i.d. from a law $P$ on $R^{d}$ , with empirical measure $P_{n} = n^{- 1} \sum_{i = 1}^{n} δ_{X_{i}}$ and empirical distribution function $F_{n}$ . Let $T_{n} = T_{n} (X_{1}, \dots, X_{n})$ be a statistic estimating a parameter $θ = θ (P)$ , and write $R_{n} = n (T_{n} - θ)$ for the centred-and-scaled error, whose law $L_{n} = L_{P} (R_{n})$ is the sampling distribution the statistician wants but cannot observe. The modes of stochastic convergence $d, P$ and the $o_{P} / O_{P}$ calculus are those of 45.04.01.

Definition (bootstrap sample and bootstrap statistic). Conditional on the data $X_{1}, \dots, X_{n}$ , draw $X_{1}^{*}, \dots, X_{n}^{*}$ i.i.d. from $P_{n}$ — that is, each $X_{i}^{*}$ is a uniform draw with replacement from the observed values. The bootstrap statistic is $T_{n}^{*} = T_{n} (X_{1}^{*}, \dots, X_{n}^{*})$ , computed by the same rule applied to the resample. The bootstrap empirical measure is $P_{n}^{*} = n^{- 1} \sum_{i = 1}^{n} δ_{X_{i}^{*}}$ .

Definition (the bootstrap principle). The bootstrap estimates the unknown sampling distribution $L_{n} = L_{P} (n (T_{n} - θ))$ by the conditional law of the bootstrapped error,

\hat{L}_{n} = L (n (T_{n}^{*} - T_{n}) X_{1}, \dots, X_{n}) .

The recentring is at $T_{n}$ , not $θ$ : in the bootstrap world the "truth" is the empirical distribution $P_{n}$ , whose parameter is $θ (P_{n}) = T_{n}$ for a plug-in statistic. The principle replaces $(P, θ)$ by $(P_{n}, T_{n})$ and reads the sampling distribution off the resampling experiment.

Definition (the bounded-Lipschitz metric). For probability laws $μ, ν$ on $R^{k}$ , the bounded-Lipschitz distance is

d_{BL} (μ, ν) = sup {\int f d μ - \int f d ν : ∥ f ∥_{\infty} \leq 1, Lip (f) \leq 1} .

It metrises weak convergence: $μ_{n} d μ$ iff $d_{BL} (μ_{n}, μ) \to 0$ .

Definition (the Mallows-Wasserstein distance). For laws $μ, ν$ on $R^{k}$ with finite second moment, the Mallows distance (the $L^{2}$ -Wasserstein distance) is

d_{2} (μ, ν) = (X, Y) in f (E ∣ X - Y ∣^{2})^{1/2},

the infimum over all couplings of random vectors $X \sim μ$ , $Y \sim ν$ . Convergence $d_{2} (μ_{n}, μ) \to 0$ holds iff $μ_{n} d μ$ together with convergence of second moments. The distance is shift-equivariant and scales as $d_{2} (a X, aY) = ∣ a ∣ d_{2} (X, Y)$ , and it satisfies the convolution bound $d_{2}^{2} (\sum X_{i}, \sum Y_{i}) \leq \sum d_{2}^{2} (X_{i}, Y_{i})$ for independent summands under a common coupling.

Definition (bootstrap consistency). The bootstrap is consistent (weakly) for $T_{n}$ at $P$ if the random distance between the bootstrap law and the true sampling limit vanishes in probability:

d_{BL} (\hat{L}_{n}, L) P 0, where L_{n} d L .

Equivalently, the conditional law $\hat{L}_{n}$ and the unconditional law $L_{n}$ have the same weak limit, the convergence of $\hat{L}_{n}$ holding in (outer) probability over the data. Strong consistency replaces $P$ by almost-sure convergence.

Counterexamples to common slips Intermediate+

Recentre at $T_{n}$ , not at $θ$ . The bootstrap quantity is $n (T_{n}^{*} - T_{n})$ ; using $n (T_{n}^{*} - θ)$ injects the unknown $θ$ the resampling cannot see and breaks the mimicry. The empirical world's parameter is $T_{n}$ .
Consistency is conditional convergence in probability, not a pointwise statement. The bootstrap law $\hat{L}_{n}$ is itself random (it depends on the data); the claim is that, as a random element of the space of laws, it approaches the fixed limit $L$ in probability. Reading it as an ordinary distributional limit conflates the two sources of randomness.
A bootstrap that reproduces the variance need not reproduce the law. For the sample maximum the bootstrap variance estimate can even be sensible while the bootstrap distribution is wrong, because the conditional law puts an atom at zero with limiting probability $1 - e^{- 1}$ . Matching one moment is weaker than matching the law.
Finite variance is a hypothesis, not a formality. The Bickel-Freedman mean theorem needs $Var_{P} (X_{1}) < \infty$ . In the infinite-variance / stable-domain case the bootstrap of the mean is inconsistent, and resampling fewer than $n$ points is required.

Key theorem with proof Intermediate+

The signature result is the Bickel-Freedman consistency theorem for the sample mean: the conditional law of the bootstrapped mean converges, in the Mallows metric and in probability over the data, to the same Gaussian limit as the true sampling distribution. The proof is a conditional central limit theorem made quantitative by the Mallows distance, which controls weak convergence and second moments simultaneously and turns the two sources of randomness into two distance bounds ^{[Bickel, P. J. & Freedman, D. A. — Some asymptotic theory for the bootstrap]}.

Theorem (consistency of the bootstrap mean). Let $X_{1}, X_{2}, \dots$ be i.i.d. with mean $μ$ and finite variance $σ^{2} > 0$ , and let $\overset{ˉ}{X}_{n} = P_{n} id$ . Conditional on $X_{1}, \dots, X_{n}$ , let $X_{1}^{*}, \dots, X_{n}^{*}$ be i.i.d. from $P_{n}$ with bootstrap mean $\overset{ˉ}{X}_{n}^{*} = n^{- 1} \sum_{i} X_{i}^{*}$ . Then, for almost every sequence $(X_{i})$ ,

d_{2} (L (n (\overset{ˉ}{X}_{n}^{*} - \overset{ˉ}{X}_{n}) ∣ X_{1}, \dots, X_{n}), N (0, σ^{2})) ⟶ 0,

and consequently $sup_{t} Pr^{*} (n (\overset{ˉ}{X}_{n}^{*} - \overset{ˉ}{X}_{n}) \leq t) - Φ (t / σ) \to 0$ a.s., where $Pr^{*}$ is the conditional law.

Proof. Conditional on the data, $n (\overset{ˉ}{X}_{n}^{*} - \overset{ˉ}{X}_{n}) = n^{- 1/2} \sum_{i = 1}^{n} (X_{i}^{*} - \overset{ˉ}{X}_{n})$ is a normalised sum of i.i.d. summands drawn from $P_{n}$ , each with conditional mean $0$ (since $\mathbb{E}^\astX_i^\ast = \bar X_n$ ) and conditional variance

\overset{σ}{^}_{n}^{2} := Var^{*} (X_{i}^{*}) = n^{- 1} i = 1 \sum n (X_{i} - \overset{ˉ}{X}_{n})^{2} .

By the strong law of large numbers, $\overset{σ}{^}_{n}^{2} \to σ^{2}$ a.s. The conditional law of the standardised summand $X_{1}^{*} - \overset{ˉ}{X}_{n}$ therefore converges, in $d_{2}$ , to $N (0, σ^{2})$ in the following sense. Couple $X_{1}^{*}$ to a $N (0, σ^{2})$ variable $W$ as follows: by the SLLN the empirical law $P_{n}$ of the centred values $X_{i} - \overset{ˉ}{X}_{n}$ converges to the law of $X_{1} - μ$ in $d_{2}$ (weak convergence plus second-moment convergence, both delivered by the SLLN applied to first and second moments), so $d_{2} (P_{n}^{c}, P^{c}) \to 0$ a.s., where $P_{n}^{c}, P^{c}$ are the centred laws.

Now apply the Mallows-distance central-limit bound. For i.i.d. centred summands with law $ν$ of variance $τ^{2}$ , the normalised sum satisfies $d_{2} (L (n^{- 1/2} \sum_{i} ξ_{i}), N (0, τ^{2})) \to 0$ ; this is the $d_{2}$ form of the central limit theorem 37.03.02, and the convergence is uniform over families of summand laws whose $d_{2}$ -distance to a fixed law tends to zero. Concretely, the triangle inequality in $d_{2}$ gives

d_{2} (L^{*} (n^{- 1/2} i \sum (X_{i}^{*} - \overset{ˉ}{X}_{n})), N (0, σ^{2})) \leq d_{2} (L^{*} (n^{- 1/2} i \sum (X_{i}^{*} - \overset{ˉ}{X}_{n})), N (0, \overset{σ}{^}_{n}^{2})) + d_{2} (N (0, \overset{σ}{^}_{n}^{2}), N (0, σ^{2})) .

The second term is $∣ \overset{σ}{^}_{n} - σ ∣ \to 0$ a.s. by the scaling property $d_{2} (N (0, a^{2}), N (0, b^{2})) = ∣ a - b ∣$ . For the first term, by the convolution bound for independent summands under a common coupling,

d_{2}^{2} (L^{*} (n^{- 1/2} i \sum (X_{i}^{*} - \overset{ˉ}{X}_{n})), N (0, \overset{σ}{^}_{n}^{2})) \leq n^{- 1} i = 1 \sum n d_{2}^{2} (P_{n}^{c}, N (0, \overset{σ}{^}_{n}^{2})) = d_{2}^{2} (P_{n}^{c}, N (0, \overset{σ}{^}_{n}^{2})),

where we couple each bootstrap summand to an independent Gaussian.

This bound does not yet vanish, so we sharpen it by the Lindeberg route: the conditional Lindeberg condition holds because $n^{- 1} \sum_{i} (X_{i} - \overset{ˉ}{X}_{n})^{2} 1 {∣ X_{i} - \overset{ˉ}{X}_{n} ∣ > ε n} \to 0$ a.s. (the truncated second moment of $P_{n}^{c}$ converges to that of $P^{c}$ , which is $σ^{2}$ , so the tail piece vanishes). The Lindeberg-Feller theorem 37.03.02, applied conditionally to the triangular array ${(X_{i}^{*} - \overset{ˉ}{X}_{n}) / n}$ , gives $L^{*} (n^{- 1/2} \sum_{i} (X_{i}^{*} - \overset{ˉ}{X}_{n})) d N (0, σ^{2})$ for a.e. data sequence; combined with the conditional-variance convergence $\overset{σ}{^}_{n}^{2} \to σ^{2}$ , the second moments also converge, upgrading weak convergence to $d_{2}$ -convergence. Since $L_{n} = L (n (\overset{ˉ}{X}_{n} - μ)) d N (0, σ^{2})$ by the classical CLT, the bootstrap law and the true sampling law share the limit, which is the assertion. The Berry-Esseen reduction of $d_{2}$ to the supremum distance gives the stated uniform statement. $□$

Bridge. This theorem builds toward the general justification of resampling for every smooth statistic, and the same conditional-CLT mechanism appears again in the delta-method bootstrap, where the bootstrapped empirical process replaces the bootstrapped sum and Hadamard differentiability replaces the linearity of the mean. The foundational reason the bootstrap works is that the empirical distribution $P_{n}$ converges to $P$ fast enough — in $d_{2}$ , hence in both weak topology and second moment — that resampling from $P_{n}$ is, to first order, the same experiment as sampling from $P$ ; this is exactly the plug-in principle of 45.04.04 applied not to a point estimate but to an entire sampling distribution. Putting these together, the bootstrap generalises the delta method 45.04.02: where the delta method transports one limit law through a derivative, the bootstrap regenerates the limit law by simulation, and the two agree precisely when the functional is differentiable. The bridge is the recognition that consistency of the bootstrap is consistency of $P_{n}$ as an estimator of $P$ in a metric strong enough to carry the statistic's limit law.

Exercises Intermediate+

Exercise 4 (medium, symbolic).

(Delta-method bootstrap for a smooth function.) Suppose $n (T_{n} - θ) d N (0, σ^{2})$ and the bootstrap is consistent for $T_{n}$ . Let $g$ be continuously differentiable with $g^{'} (θ) \neq = 0$ . Argue that the bootstrap is consistent for $g (T_{n})$ , i.e. the conditional law of $n (g (T_{n}^{*}) - g (T_{n}))$ converges to $N (0, g^{'} (θ)^{2} σ^{2})$ in probability.

Hint

Apply the delta-method linearisation $g (T_{n}^{*}) - g (T_{n}) = g^{'} (T_{n}) (T_{n}^{*} - T_{n}) + o (\cdot)$ inside the bootstrap world, and use $g^{'} (T_{n}) P g^{'} (θ)$ .

Answer

Conditionally on the data, $T_{n}^{*} - T_{n} = o_{P^{*}} (1)$ (bootstrap consistency forces $n (T_{n}^{*} - T_{n}) = O_{P^{*}} (1)$ ), so a first-order Taylor expansion gives $n (g (T_{n}^{*}) - g (T_{n})) = g^{'} (T_{n}) n (T_{n}^{*} - T_{n}) + R_{n}^{*}$ with $R_{n}^{*} = o_{P^{*}} (1)$ by the same remainder bookkeeping as the delta method 45.04.02. By consistency $T_{n} P θ$ and continuity of $g^{'}$ , the slope $g^{'} (T_{n}) P g^{'} (θ)$ , a constant relative to the conditional law. The bootstrap consistency for $T_{n}$ gives $n (T_{n}^{*} - T_{n}) d^{*} N (0, σ^{2})$ in probability; multiplying by the convergent slope (Slutsky, conditionally) yields $n (g (T_{n}^{*}) - g (T_{n})) d^{*} N (0, g^{'} (θ)^{2} σ^{2})$ , matching the delta-method limit of $n (g (T_{n}) - g (θ))$ . The bootstrap inherits consistency through any differentiable transformation.

Exercise 5 (medium, symbolic).

(Why the studentised statistic is pivotal.) Let $T_{n}$ estimate $θ$ with $n (T_{n} - θ) d N (0, σ^{2})$ and let $\overset{σ}{^}_{n} P σ$ . Show the studentised statistic $S_{n} = n (T_{n} - θ) / \overset{σ}{^}_{n} d N (0, 1)$ , a limit free of unknown parameters, and explain why a quantity with a parameter-free limit is the right thing to bootstrap.

Hint

Divide the distributional limit by the convergent denominator using Slutsky; "pivotal" means the limit law has no unknowns.

Answer

By Slutsky's lemma, $S_{n} = n (T_{n} - θ) / \overset{σ}{^}_{n} d N (0, σ^{2}) / σ = N (0, 1)$ , a limit with no dependence on $θ$ or $σ$ . A statistic whose limit law is parameter-free is asymptotically pivotal: the gap between its finite-sample law and its limit is then governed by an Edgeworth expansion whose leading correction the bootstrap can match exactly, because the bootstrap reproduces the polynomial-in-cumulants structure of that expansion. Bootstrapping a non-pivotal quantity (e.g. $n (T_{n} - θ)$ itself, whose limit has the unknown $σ$ ) leaves a first-order error in the scale, so the studentised version is the one for which the bootstrap delivers its second-order accuracy.

Exercise 6 (medium, symbolic).

(Conditional Lindeberg condition.) In the bootstrap-mean proof, the conditional Lindeberg condition is $n^{- 1} \sum_{i = 1}^{n} (X_{i} - \overset{ˉ}{X}_{n})^{2} 1 {∣ X_{i} - \overset{ˉ}{X}_{n} ∣ > ε n} \to 0$ a.s. for each $ε > 0$ . Show this follows from $E X_{1}^{2} < \infty$ .

Hint

The indicator forces $∣ X_{i} - \overset{ˉ}{X}_{n} ∣ > ε n$ ; since $\overset{ˉ}{X}_{n} \to μ$ , at most finitely many fixed $X_{i}$ can satisfy this as $n$ grows, and the truncated second moment of a finite-variance law vanishes.

Answer

Fix $ε > 0$ . Since $\overset{ˉ}{X}_{n} \to μ$ a.s., for large $n$ the event ${∣ X_{i} - \overset{ˉ}{X}_{n} ∣ > ε n}$ implies $∣ X_{i} - μ ∣ > \frac{1}{2} ε n$ . Hence the sum is bounded by $n^{- 1} \sum_{i} (X_{i} - \overset{ˉ}{X}_{n})^{2} 1 {∣ X_{i} - μ ∣ > \frac{1}{2} ε n}$ , which is at most $2 n^{- 1} \sum_{i} [(X_{i} - μ)^{2} + (μ - \overset{ˉ}{X}_{n})^{2}] 1 {\cdot}$ . The second piece is $O ((μ - \overset{ˉ}{X}_{n})^{2}) = o (1)$ . The first piece is $2 n^{- 1} \sum_{i} (X_{i} - μ)^{2} 1 {∣ X_{i} - μ ∣ > \frac{1}{2} ε n}$ ; by the SLLN this converges a.s. to $2 E [(X_{1} - μ)^{2} 1 {∣ X_{1} - μ ∣ > c}]$ in the limit, and as $c = \frac{1}{2} ε n \to \infty$ the dominated-convergence tail of a finite-variance law is $0$ . So the truncated sum vanishes, verifying the condition.

Exercise 7 (hard, symbolic).

(Failure for the sample maximum.) Let $X_{1}, \dots, X_{n}$ be i.i.d. Uniform $(0, θ)$ , $T_{n} = X_{(n)} = max_{i} X_{i}$ . The true rescaled error has limit $n (θ - X_{(n)}) / θ d Exp (1)$ . Show that the bootstrap maximum $X_{(n)}^{*}$ satisfies $Pr^{*} (X_{(n)}^{*} = X_{(n)}) \to 1 - e^{- 1}$ , so the conditional law of $n (X_{(n)} - X_{(n)}^{*}) / X_{(n)}$ has a persistent atom at $0$ and cannot converge to the continuous $Exp (1)$ limit.

Hint

$X_{(n)}^{*} = X_{(n)}$ unless none of the $n$ bootstrap draws lands on the index of the sample maximum; that probability is $(1 - 1/ n)^{n}$ .

Answer

The bootstrap maximum equals $X_{(n)}$ exactly when at least one of the $n$ resampled indices equals the index $i^{*}$ achieving the sample maximum. The probability that a single draw misses $i^{*}$ is $1 - 1/ n$ , so $Pr^{*} (X_{(n)}^{*} < X_{(n)}) = Pr^{*} (all n draws miss i^{*}) = (1 - 1/ n)^{n} \to e^{- 1}$ . Therefore $Pr^{*} (X_{(n)}^{*} = X_{(n)}) \to 1 - e^{- 1} \approx 0.632$ . The conditional law of $n (X_{(n)} - X_{(n)}^{*}) / X_{(n)}$ thus places mass $1 - e^{- 1}$ on the single point $0$ for every large $n$ , a discrete atom that the continuous $Exp (1)$ limit does not have. So $d_{BL} (\hat{L}_{n}, Exp (1)) \neq \to 0$ : the naive $n$ -out-of- $n$ bootstrap is inconsistent for the maximum. The defect is the extreme dependence of $X_{(n)}$ on a single order statistic, exactly the non-smoothness the delta-method bootstrap excludes.

Exercise 8 (hard, symbolic).

(The m-out-of-n repair.) For the Uniform $(0, θ)$ maximum, the m-out-of-n bootstrap draws $m = m_{n}$ observations with replacement, $m \to \infty$ , $m / n \to 0$ . Show $Pr^{*} (X_{(m)}^{*} = X_{(n)}) = 1 - (1 - 1/ n)^{m} \to 0$ , so the atom disappears, and indicate why the rescaled $m (X_{(n)} - X_{(m)}^{*}) / X_{(n)}$ then recovers the $Exp (1)$ limit.

Hint

With $m$ draws the chance of hitting the maximising index is $1 - (1 - 1/ n)^{m} \approx m / n \to 0$ ; rescale by $m$ rather than $n$ .

Answer

Drawing $m$ indices with replacement, the chance that at least one equals $i^{*}$ is $1 - (1 - 1/ n)^{m} \approx 1 - e^{- m / n} \approx m / n \to 0$ . So $Pr^{*} (X_{(m)}^{*} = X_{(n)}) \to 0$ and the degenerate atom vanishes. Conditionally, $X_{(m)}^{*}$ is the maximum of $m$ i.i.d. draws from $P_{n}$ , which (since $P_{n} \to Unif (0, θ)$ and $m \to \infty$ , $m / n \to 0$ ) behaves like the maximum of $m$ genuine Uniform $(0, X_{(n)})$ draws; the extreme-value scaling at sample size $m$ gives $m (X_{(n)} - X_{(m)}^{*}) / X_{(n)} d^{*} Exp (1)$ in probability, matching the true limit (with $X_{(n)} \to θ$ ). The condition $m / n \to 0$ keeps the resample from over-using the single sample extreme, while $m \to \infty$ supplies the asymptotics; this is the general m-out-of-n principle for boundary and extreme-value parameters ^{[Bickel, P. J., Götze, F. & van Zwet, W. R. — Resampling fewer than n observations]}.

Advanced results Master

The empirical-bootstrap consistency for the mean extends to all Hadamard-differentiable functionals through the bootstrapped empirical process, the studentised statistic gains an extra order of accuracy through the Edgeworth expansion, and the boundary and extreme-value cases that defeat the $n$ -out-of- $n$ scheme are repaired by resampling $m ≪ n$ points. The smooth-functional case is the centre; the failures and their fixes are the boundary of validity.

Theorem 1 (consistency of the bootstrap mean; Mallows route). For i.i.d. $X_{i}$ with $Var (X_{1}) = σ^{2} < \infty$ , the conditional law of $n (\overset{ˉ}{X}_{n}^{*} - \overset{ˉ}{X}_{n})$ converges to $N (0, σ^{2})$ in the Mallows distance $d_{2}$ for almost every data sequence, hence the bootstrap is strongly consistent for the mean. The mechanism is that $P_{n} \to P$ in $d_{2}$ by the strong law applied to the first two moments, and $d_{2}$ controls both the weak limit and the variance simultaneously, so the conditional CLT inherits the correct scale. The hypothesis $σ^{2} < \infty$ is sharp: in the infinite-variance / stable domain of attraction the conditional law of the suitably normalised bootstrap mean does not converge to the stable sampling limit, and the bootstrap is inconsistent ^{[Bickel, P. J. & Freedman, D. A. — Some asymptotic theory for the bootstrap]}.

Theorem 2 (the delta-method bootstrap; Hadamard differentiability). Let $ϕ$ be a functional that is Hadamard-differentiable at $F$ tangentially to a subspace, with derivative $ϕ_{F}^{'}$ , and let $\hat{F}_{n}$ be the empirical distribution with bootstrap version $\hat{F}_{n}^{*}$ . If the empirical process $G_{n} = n (\hat{F}_{n} - F)$ converges weakly to a tight Gaussian process $G$ (a Donsker condition, 45.05.03) and the bootstrap empirical process $n (\hat{F}_{n}^{*} - \hat{F}_{n})$ converges conditionally to the same $G$ in probability, then

n (ϕ (\hat{F}_{n}^{*}) - ϕ (\hat{F}_{n})) d ϕ_{F}^{'} (G) conditionally, in probability,

matching the limit of $n (ϕ (\hat{F}_{n}) - ϕ (F))$ . The functional delta method for the bootstrap is the chain rule applied to the bootstrapped process: Hadamard differentiability is exactly the regularity that lets the limit pass through $ϕ$ , and it is the same hypothesis that makes the ordinary functional delta method 45.05.06 valid. Quantiles, trimmed means, L-statistics, M-estimators 45.04.04, and copula functionals fall under this theorem; the sample maximum, a non-Hadamard-differentiable functional, does not ^{[van der Vaart — Asymptotic Statistics]}.

Theorem 3 (second-order accuracy of the bootstrap-t). For a smooth statistic admitting an Edgeworth expansion, the studentised (asymptotically pivotal) statistic $S_{n} = n (T_{n} - θ) / \overset{σ}{^}_{n}$ has distribution

Pr (S_{n} \leq x) = Φ (x) + n^{- 1/2} q_{1} (x) ϕ (x) + O (n^{- 1}),

with $q_{1}$ an even polynomial in the population cumulants. The bootstrap distribution $Pr^{*} (S_{n}^{*} \leq x)$ has the same expansion with the cumulants replaced by their $n$ -consistent empirical estimates, so $q_{1}$ is reproduced up to an $O_{P} (n^{- 1/2})$ error in its coefficients, and the bootstrap matches the $n^{- 1/2}$ term:

Pr * (S_{n}^{*} \leq x) - Pr (S_{n} \leq x) = O_{P} (n^{- 1}) .

The coverage error of a one-sided bootstrap-t interval is therefore $O (n^{- 1})$ , against $O (n^{- 1/2})$ for the normal approximation and for the percentile interval applied to the non-pivotal $n (T_{n} - θ)$ . Pivoting removes the leading error term; this is the precise sense in which the bootstrap-t "beats" the normal approximation ^{[Hall, P. — The Bootstrap and Edgeworth Expansion]}.

Theorem 4 (failure at the boundary and the m-out-of-n repair). When the limit law of $n (T_{n} - θ)$ depends discontinuously on $P$ — the sample extremes, a parameter on the boundary of its space, super-efficient or shrinkage estimators, the largest eigenvalue at a multiplicity change — the $n$ -out-of-n bootstrap is inconsistent: the conditional law fails to track the sampling law (the maximum's persistent atom at $0$ is the prototype). The m-out-of-n bootstrap, resampling $m = m_{n}$ points with $m \to \infty$ and $m / n \to 0$ , restores consistency whenever the sampling distribution has a nondegenerate limit and the convergence is locally uniform, because the smaller resample size decouples the bootstrap from the over-counted sample extremes while preserving the asymptotics; subsampling without replacement (Politis-Romano) is the closely related construction with even weaker requirements ^{[Bickel, P. J., Götze, F. & van Zwet, W. R. — Resampling fewer than n observations]}.

Synthesis. The foundational reason the bootstrap is consistent for smooth statistics is that $P_{n}$ converges to $P$ in a metric strong enough — the Mallows $d_{2}$ for the mean, the uniform Donsker topology for functionals — to carry the statistic's entire limit law, so resampling from $P_{n}$ is asymptotically the same experiment as sampling from $P$ ; this is exactly the plug-in principle of 45.04.04 lifted from a point estimate to a sampling distribution. The central insight is that consistency is differentiability: the delta-method bootstrap 45.04.02 succeeds precisely for Hadamard-differentiable functionals, because the bootstrapped empirical process passes through the derivative as the genuine process does, and fails exactly where that derivative does not exist — the sample maximum, the boundary parameter — which is dual to the kink obstruction delimiting the ordinary delta method. Putting these together, second-order accuracy refines the same picture: the bootstrap reproduces not just the Gaussian term but the first Edgeworth correction, and studentisation aligns the bootstrap's empirical cumulants with the population ones at order $n^{- 1/2}$ , generalising variance stabilisation from making one moment constant to making the whole expansion match. The bridge from failure back to validity is the m-out-of-n repair, which trades exact mimicry of $P_{n}$ for a coarser resample that no longer over-weights the sample extremes; the theory is one principle — estimate the sampling distribution by the resampling distribution — valid to the order at which the empirical law approximates the true law in a metric matched to the statistic's smoothness.

Full proof set Master

The sample-mean consistency (Theorem 1) is proved in the Key theorem; the delta-method inheritance, the studentised pivotality, the maximum's failure, and the m-out-of-n repair are Exercises 4, 5, 7, and 8. The remaining Master claims are recorded here.

Proposition 1 (conditional convergence in probability via the bounded-Lipschitz metric). The bootstrap is consistent for $T_{n}$ — $d_{BL} (\hat{L}_{n}, L) P 0$ — if and only if for every bounded Lipschitz $f$ , $E^{\astf} (n (T_{n}^{*} - T_{n})) P E f (L)$ , where $E^{*}$ is the conditional expectation given the data.

Proof. The bounded-Lipschitz distance is the supremum of $∣ E^{\astf} (n (T_{n}^{*} - T_{n})) - E f (L) ∣$ over the unit ball $F_{1} = {f : ∥ f ∥_{\infty} \leq 1, Lip (f) \leq 1}$ . If $d_{BL} (\hat{L}_{n}, L) P 0$ then the supremum, hence each fixed $f$ -difference, tends to $0$ in probability. Conversely, $F_{1}$ is separable for the topology of uniform convergence on compacta and the functionals $μ \mapsto \int f d μ$ are $1$ -Lipschitz in $d_{BL}$ uniformly over $F_{1}$ ; a standard tightness-plus-countable-dense-subset argument upgrades pointwise (in $f$ ) convergence in probability to convergence of the supremum in probability, because $d_{BL}$ is attained as a sup over a $d_{BL}$ -equicontinuous family. Thus the two statements are equivalent, which is why consistency is phrased as conditional weak convergence in probability. $□$

Proposition 2 (Mallows-distance contraction for normalised sums). Let $ν, ρ$ be laws on $R$ with equal means and finite second moments, and let $ξ_{1}, \dots, ξ_{n}$ be i.i.d.\ $ν$ and $η_{1}, \dots, η_{n}$ i.i.d.\ $ρ$ . Then $d_{2} (L (n^{- 1/2} \sum_{i} ξ_{i}), L (n^{- 1/2} \sum_{i} η_{i})) \leq d_{2} (ν, ρ)$ .

Proof. Let $(ξ, η)$ be an optimal coupling of $ν, ρ$ attaining $d_{2} (ν, ρ)^{2} = E (ξ - η)^{2}$ , and take $(ξ_{i}, η_{i})$ i.i.d.\ copies of it; this is a valid coupling of the two product laws and hence of the two sums. Then

d_{2}^{2} (L (n^{- 1/2} i \sum ξ_{i}), L (n^{- 1/2} i \sum η_{i})) \leq E (n^{- 1/2} i \sum (ξ_{i} - η_{i}))^{2} = \frac{1}{n} i \sum E (ξ_{i} - η_{i})^{2} = d_{2}^{2} (ν, ρ),

using independence and the equal-mean assumption to drop cross terms. This is the convolution bound underlying the bootstrap-mean proof: applying it with $ν = P_{n}^{c}$ the centred empirical law and $ρ = N (0, \overset{σ}{^}_{n}^{2})$ controls the bootstrap sum by $d_{2} (P_{n}^{c}, N (0, \overset{σ}{^}_{n}^{2}))$ , which the conditional CLT sends to $0$ . $□$

Proposition 3 (consistency of the bootstrap variance estimate). Under $E X_{1}^{4} < \infty$ , the bootstrap estimate of $Var (n \overset{ˉ}{X}_{n})$ , namely $Var^{*} (n \overset{ˉ}{X}_{n}^{*}) = \overset{σ}{^}_{n}^{2} = n^{- 1} \sum_{i} (X_{i} - \overset{ˉ}{X}_{n})^{2}$ , converges a.s.\ to $σ^{2}$ .

Proof. Expand $\overset{σ}{^}_{n}^{2} = n^{- 1} \sum_{i} X_{i}^{2} - \overset{ˉ}{X}_{n}^{2}$ . By the strong law $n^{- 1} \sum_{i} X_{i}^{2} \to E X_{1}^{2}$ a.s.\ and $\overset{ˉ}{X}_{n} \to μ$ a.s., so $\overset{σ}{^}_{n}^{2} \to E X_{1}^{2} - μ^{2} = σ^{2}$ a.s. The fourth-moment hypothesis is not needed for this first-moment statement but guarantees the further consistency of the bootstrap estimate of $Var (\overset{σ}{^}_{n}^{2})$ , securing the variance of the variance. The conditional variance is the exact (not merely asymptotic) variance of the bootstrap mean, which is why the bootstrap reproduces the plug-in standard error before any limit is taken. $□$

Proposition 4 (Edgeworth matching of the bootstrap-t at order $n^{- 1/2}$ ). Suppose $S_{n}$ admits the Edgeworth expansion $Pr (S_{n} \leq x) = Φ (x) + n^{- 1/2} q_{1} (x; κ) ϕ (x) + O (n^{- 1})$ where $q_{1}$ depends smoothly on a finite vector $κ$ of population cumulants, and the empirical cumulants satisfy $\overset{κ}{^} = κ + O_{P} (n^{- 1/2})$ . Then $Pr^{*} (S_{n}^{*} \leq x) - Pr (S_{n} \leq x) = O_{P} (n^{- 1})$ .

Proof. The bootstrap statistic $S_{n}^{*}$ is the studentised statistic computed in the empirical world, whose population is $P_{n}$ with cumulants $\overset{κ}{^}$ . By the same Edgeworth expansion, valid uniformly over cumulants in a neighbourhood of $κ$ , $Pr^{*} (S_{n}^{*} \leq x) = Φ (x) + n^{- 1/2} q_{1} (x; \overset{κ}{^}) ϕ (x) + O_{P} (n^{- 1})$ . Subtracting the population expansion,

Pr * (S_{n}^{*} \leq x) - Pr (S_{n} \leq x) = n^{- 1/2} (q_{1} (x; \overset{κ}{^}) - q_{1} (x; κ)) ϕ (x) + O_{P} (n^{- 1}) .

Smoothness of $q_{1}$ in $κ$ and $\overset{κ}{^} - κ = O_{P} (n^{- 1/2})$ give $q_{1} (x; \overset{κ}{^}) - q_{1} (x; κ) = O_{P} (n^{- 1/2})$ , so the displayed first term is $n^{- 1/2} \cdot O_{P} (n^{- 1/2}) = O_{P} (n^{- 1})$ , whence the whole difference is $O_{P} (n^{- 1})$ . The crucial input is that $q_{1}$ is even in $x$ for a studentised statistic, so the bootstrap and the truth agree on the $n^{- 1/2}$ skewness correction; for the non-pivotal $n (T_{n} - θ)$ the scale enters $q_{1}$ at order $n^{0}$ in a way the bootstrap mis-estimates at $O_{P} (n^{- 1/2})$ , leaving the larger coverage error. $□$

Proposition 5 (the maximum's atom is a non-Hadamard obstruction). For the Uniform $(0, θ)$ maximum the naive bootstrap is inconsistent, and the obstruction is the failure of $F \mapsto sup {x : F (x) < 1}$ to be Hadamard-differentiable at a distribution with a jump-free right endpoint.

Proof. Exercise 7 establishes $Pr^{*} (X_{(n)}^{*} = X_{(n)}) \to 1 - e^{- 1}$ , a persistent atom incompatible with the continuous $Exp (1)$ limit, so $d_{BL} (\hat{L}_{n}, Exp (1)) \neq \to 0$ . For the structural reason, the endpoint functional $ϕ (F) = sup {x : F (x) < 1}$ is not Hadamard-differentiable at $Unif (0, θ)$ : perturbing $F$ by $t h_{t}$ along a sequence $h_{t} \to h$ moves the endpoint by an amount that depends on the local behaviour of $h$ at the single point $θ$ rather than linearly on $h$ in any tangent norm, so no continuous linear derivative $ϕ_{F}^{'}$ exists. Theorem 2 requires Hadamard differentiability precisely to transport the limit through $ϕ$ ; its absence is the exact dual of the kink obstruction that delimits the ordinary delta method 45.04.02, and it is why the maximum needs the m-out-of-n repair (Exercise 8) rather than the delta-method bootstrap. $□$

Connections Master

The delta method and the second-order delta method 45.04.02 are the conceptual core: bootstrap consistency for a smooth statistic is the delta-method linearisation carried out inside the resampling world, and the boundary of validity coincides — the bootstrap is consistent exactly where the functional is differentiable, and the maximum's failure is the resampling face of the kink obstruction that delimits the delta method, while second-order accuracy refines the Gaussian leading term with the Edgeworth correction that pivoting aligns.

M-estimators and Z-estimators 45.04.04 supply the plug-in principle the bootstrap lifts from a point estimate to a sampling distribution, and they are a principal client: because an M-estimator is an asymptotically linear functional with influence function $A^{- 1} ψ_{θ_{0}}$ , the delta-method bootstrap (Theorem 2) certifies that resampling reproduces its sandwich covariance, so the bootstrap is an alternative to the analytic sandwich estimator for standard errors and confidence regions.

The functional delta method via Hadamard differentiability 45.05.06 is the exact hypothesis under which Theorem 2 holds: the bootstrap inherits consistency through any Hadamard-differentiable map of the empirical process, so quantiles, trimmed means, and L-statistics are bootstrappable for the same reason they obey the functional delta method, and the non-differentiable endpoint functional is excluded by both at once.

Donsker classes and the empirical process 45.05.03 provide the infinite-dimensional input the functional bootstrap consumes: the conditional weak convergence of the bootstrap empirical process $n (\hat{F}_{n}^{*} - \hat{F}_{n})$ to the same Brownian bridge as $n (\hat{F}_{n} - F)$ is a bootstrap Donsker theorem, and it is the infinite-dimensional analogue of the conditional CLT proved for the mean here, with stochastic equicontinuity over the class controlling the resampled fluctuation uniformly.

Nonparametric methods and resampling 26.08.01 is the elementary companion that defines the bootstrap algorithm — resample with replacement, recompute, read off percentiles — without the consistency theory; this unit is the rigorous justification of that recipe, establishing when the percentile and bootstrap-t intervals it describes actually attain their nominal coverage and when (the maximum, the boundary, infinite variance) they silently fail.

Historical & philosophical context Master

The bootstrap was introduced by Bradley Efron in his 1979 Annals of Statistics paper ^{[Efron, B. — Bootstrap methods: another look at the jackknife]}, which defined resampling with replacement from the empirical distribution as a general way to estimate the sampling distribution of any statistic, unifying and extending the jackknife of Quenouille and Tukey. Efron's move was to take the plug-in principle seriously at the level of the whole distribution: replace the unknown $P$ by $P_{n}$ everywhere, including inside the sampling-distribution operator, and evaluate the result by Monte Carlo when no closed form exists.

The asymptotic justification followed quickly. Peter Bickel and David Freedman, in their 1981 Annals of Statistics paper ^{[Bickel, P. J. & Freedman, D. A. — Some asymptotic theory for the bootstrap]}, proved consistency for the sample mean and a range of smooth statistics using the Mallows-Wasserstein distance, and in the same work exhibited the canonical failures — the sample maximum, the mean with infinite variance — that showed the bootstrap is not universally valid. The metric-space formulation of consistency as conditional weak convergence in probability, and the extension to Hadamard-differentiable functionals via the bootstrap empirical process, were consolidated by Aad van der Vaart and Jon Wellner, with van der Vaart's 1998 Asymptotic Statistics giving the textbook treatment used here ^{[van der Vaart — Asymptotic Statistics]}.

The second-order theory is due to Peter Hall, whose 1992 monograph The Bootstrap and Edgeworth Expansion ^{[Hall, P. — The Bootstrap and Edgeworth Expansion]} explained, through Edgeworth and Cornish-Fisher expansions, why the studentised bootstrap-t achieves $O (n^{- 1})$ coverage error against the $O (n^{- 1/2})$ of the normal approximation, locating the bootstrap's advantage in its matching of the skewness correction for asymptotically pivotal statistics. The repair of the inconsistent cases by resampling fewer than $n$ points was developed by Bickel, Götze, and van Zwet in 1997 ^{[Bickel, P. J., Götze, F. & van Zwet, W. R. — Resampling fewer than n observations]} and, in the subsampling form, by Politis and Romano.

Bibliography Master

@article{Efron1979,
  author  = {Efron, Bradley},
  title   = {Bootstrap methods: another look at the jackknife},
  journal = {Annals of Statistics},
  volume  = {7},
  number  = {1},
  year    = {1979},
  pages   = {1--26}
}

@article{BickelFreedman1981,
  author  = {Bickel, Peter J. and Freedman, David A.},
  title   = {Some asymptotic theory for the bootstrap},
  journal = {Annals of Statistics},
  volume  = {9},
  number  = {6},
  year    = {1981},
  pages   = {1196--1217}
}

@book{Hall1992,
  author    = {Hall, Peter},
  title     = {The Bootstrap and Edgeworth Expansion},
  series    = {Springer Series in Statistics},
  publisher = {Springer-Verlag},
  address   = {New York},
  year      = {1992}
}

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@article{BickelGotzeVanZwet1997,
  author  = {Bickel, Peter J. and G\"otze, Friedrich and van Zwet, Willem R.},
  title   = {Resampling fewer than $n$ observations: gains, losses, and remedies for losses},
  journal = {Statistica Sinica},
  volume  = {7},
  number  = {1},
  year    = {1997},
  pages   = {1--31}
}

@book{EfronTibshirani1993,
  author    = {Efron, Bradley and Tibshirani, Robert J.},
  title     = {An Introduction to the Bootstrap},
  series    = {Monographs on Statistics and Applied Probability},
  publisher = {Chapman \& Hall},
  address   = {New York},
  year      = {1993}
}

@article{PolitisRomano1994,
  author  = {Politis, Dimitris N. and Romano, Joseph P.},
  title   = {Large sample confidence regions based on subsamples under minimal assumptions},
  journal = {Annals of Statistics},
  volume  = {22},
  number  = {4},
  year    = {1994},
  pages   = {2031--2050}
}

Prerequisites

45.04.02
45.04.04

Tier anchors

beginner: Efron & Tibshirani 1993 An Introduction to the Bootstrap (Chapman & Hall) Ch. 6-7 (the plug-in principle and the percentile interval, informally); Wasserman 2004 All of Statistics (Springer) §8.3 (the bootstrap as a way to read the standard error off resampled data)
intermediate: van der Vaart 1998 Asymptotic Statistics (Cambridge) §23.1-23.2 (the empirical bootstrap, conditional weak convergence, and consistency for the sample mean); Shao & Tu 1995 The Jackknife and Bootstrap (Springer) Ch. 3 (consistency via the Mallows distance)
master: van der Vaart 1998 Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics) Ch. 23 (the bootstrap: §23.2 the empirical bootstrap and conditional convergence in probability, §23.2.1 the multiplier/exchangeable-weights bootstrap, the delta-method bootstrap for Hadamard-differentiable functionals); Bickel & Freedman 1981 (Ann. Statist.) (the Mallows-metric consistency proof and its failures); Hall 1992 The Bootstrap and Edgeworth Expansion (Springer) Ch. 3-4 (second-order accuracy of the bootstrap-t)

References

van der Vaart — Asymptotic Statistics · Cambridge University Press, Cambridge Series in Statistical and Probabilistic Mathematics, 1998; Ch. 23 (the bootstrap: the empirical bootstrap §23.2, conditional weak convergence in probability and the bounded-Lipschitz metric, the delta-method bootstrap via Hadamard differentiability §23.2.1, the exchangeably-weighted/multiplier bootstrap)
Bickel, P. J. & Freedman, D. A. — Some asymptotic theory for the bootstrap · Annals of Statistics 9 (1981), 1196-1217 (consistency of the bootstrap for the sample mean via the Mallows-Wasserstein distance d_2; the conditional CLT for the bootstrap mean; failure of the bootstrap for the sample maximum and for the mean with infinite variance)
Hall, P. — The Bootstrap and Edgeworth Expansion · Springer Series in Statistics, 1992; Ch. 2-4 (Edgeworth and Cornish-Fisher expansions, the second-order accuracy of the bootstrap-t and BCa intervals, why pivoting/studentisation yields O(n^{-1}) coverage error against O(n^{-1/2}) for the normal approximation)
Efron, B. — Bootstrap methods: another look at the jackknife · Annals of Statistics 7 (1979), 1-26 (the original definition of the bootstrap as resampling with replacement from the empirical distribution, the plug-in estimate of the sampling distribution, and the connection to the jackknife)
Bickel, P. J., Götze, F. & van Zwet, W. R. — Resampling fewer than n observations · Statistica Sinica 7 (1997), 1-31 (the m-out-of-n bootstrap as a repair for inconsistent cases: boundary parameters, the sample extremes, and non-smooth functionals; consistency under m -> infinity, m/n -> 0)

Estimated time

beginner: 20m
intermediate: 55m
master: 95m