45.03.04 · mathematical-statistics / 03-bayesian-inference

The Bernstein-von Mises Theorem

shipped3 tiersLean: none

Anchor (Master): van der Vaart 1998 Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics) Ch. 10 §10.2 (Bernstein-von Mises) with the LAN background of Ch. 7; Le Cam 1986 Asymptotic Methods in Statistical Decision Theory (Springer) Part 3 (limit experiments and the LAN/LAMN structure); Ghosal & van der Vaart 2017 Fundamentals of Nonparametric Bayesian Inference (Cambridge) Ch. 8, 10, 12 (posterior contraction rates and the failure / repair of BvM in infinite dimensions)

Intuition Beginner

A Bayesian starts with a prior — a belief about an unknown number before seeing data — and updates it into a posterior once data arrive. A frequentist ignores priors and reads a single best guess straight off the data, the maximum-likelihood estimate. These two camps often argue. The Bernstein-von Mises theorem is the peace treaty: with enough data, they end up in almost exactly the same place.

Here is the picture. As you collect more and more observations, the posterior stops looking like your prior and starts looking like a bell curve. That bell is centred on the maximum-likelihood estimate — the frequentist's answer — and its width is set by the information in the data, the same width the frequentist would quote. Two analysts who began with different priors, one optimistic and one skeptical, watch their posteriors converge onto the same bell. The prior gets washed out.

Why does this matter? Because it tells you that, in large samples, a Bayesian "95% credible interval" — the central chunk holding 95% of the posterior — is essentially the same as a frequentist "95% confidence interval." The two traditions report nearly identical numbers, so a practitioner can use whichever machinery is convenient and trust the answer.

The one-sentence version: with plenty of data, the posterior becomes a bell curve sitting on the maximum-likelihood estimate with the data-determined width, the prior fades away, and Bayesian and frequentist intervals agree.

Visual Beginner

Picture the same posterior drawn for three sample sizes, with two analysts using different priors at each size. With few observations the two posteriors are far apart and each is shaped by its prior. With more observations they pull together and start to look bell-shaped. With a large sample the two posteriors lie almost on top of each other, both bell-shaped, both centred on the maximum-likelihood estimate marked on the axis, both with the same width.

sample size	two priors: how far apart?	shape	centre
small	far apart, prior-driven	irregular	near each prior
medium	closing the gap	roughly bell-shaped	drifting toward the estimate
large	nearly identical	a narrow bell	on the maximum-likelihood estimate

The takeaway: the prior matters when data are scarce and fades when data are plentiful; in the large-sample limit every reasonable prior gives the same bell-shaped posterior, centred where the frequentist estimate sits.

Worked example Beginner

Suppose you flip a coin $n = 400$ times and see $240$ heads, and you want the rate $p$ of heads. Two friends use different priors. Friend A uses a flat prior, the same as $Beta (1, 1)$ . Friend B uses $Beta (20, 20)$ , a belief that the coin is close to fair.

Step 1. The data summary. The sample proportion is $240/400 = 0.60$ . This is the maximum-likelihood estimate, the frequentist's number.

Step 2. Friend A's posterior. With a flat prior the posterior is $Beta (1 + 240, 1 + 160) = Beta (241, 161)$ . Its mean is $241/402 \approx 0.5995$ .

Step 3. Friend B's posterior. With the $Beta (20, 20)$ prior the posterior is $Beta (20 + 240, 20 + 160) = Beta (260, 180)$ . Its mean is $260/440 \approx 0.5909$ .

Step 4. Compare the two. The two posterior means are $0.5995$ and $0.5909$ , both close to the estimate $0.60$ and close to each other — within about $0.009$ . Friend B's stronger pull toward $0.5$ shows up only as a small tug.

Step 5. The width. The frequentist standard error is $0.6 \times 0.4/400 = 0.00060 \approx 0.0245$ . Both posteriors have spread very near this value, so both friends' central 95% intervals are about $0.60 \pm 0.048$ , essentially the frequentist interval.

What this tells us: even with two quite different priors, the large sample pushes both posteriors onto the same bell, centred near $0.60$ with the data-set width. With ten flips instead of four hundred, the two friends would have disagreed sharply — the agreement is a large-sample effect.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_{1}, \dots, X_{n}$ be i.i.d. from a regular parametric family ${P_{θ} : θ \in Θ}$ , $Θ \subseteq R^{k}$ open, dominated with densities $f (\cdot ∣ θ)$ , with true value $θ_{0}$ interior to $Θ$ . The family is assumed differentiable in quadratic mean at $θ_{0}$ with score $\dot{ℓ}_{θ_{0}}$ and nonsingular Fisher information $I (θ_{0})$ , the regularity used in 45.04.03 and 45.01.05. Let $Π$ be a prior on $Θ$ with Lebesgue density $π$ , and let $Π (\cdot ∣ X_{1}, \dots, X_{n})$ denote the posterior, the regular conditional law of $θ$ given the sample, as in 45.03.01.

Definition (total-variation distance). For probability measures $P, Q$ on a common space, the total-variation distance is

∥ P - Q ∥_{TV} = B sup ∣ P (B) - Q (B) ∣ = \frac{1}{2} \int ∣ p - q ∣ d μ,

the supremum over measurable sets, equal to half the $L^{1}$ distance of densities against any dominating $μ$ .

Definition (local parameter and the recentred posterior). Fix an estimator sequence $\hat{θ}_{n}$ (the MLE of 45.04.03, or any efficient estimator). The local parameter is $h = n (θ - \hat{θ}_{n})$ . The recentred-rescaled posterior is the law of $h$ under the posterior — the pushforward of $Π (\cdot ∣ X_{1}, \dots, X_{n})$ by $θ \mapsto n (θ - \hat{θ}_{n})$ . Write it $Π_{n}^{h}$ .

Definition (Bernstein-von Mises property). The model with prior $Π$ has the Bernstein-von Mises (BvM) property at $θ_{0}$ if

Π_{n}^{h} - N (0, I (θ_{0})^{- 1})_{TV} P_{θ_{0}} 0,

the convergence being in $P_{θ_{0}}$ -probability. Equivalently, on the original scale, the posterior is asymptotically $N (\hat{θ}_{n}, (n I (θ_{0}))^{- 1})$ in total variation. The centring $\hat{θ}_{n}$ is the MLE and the spread is the inverse Fisher information — the frequentist asymptotic law of 45.04.03 — and the limit does not involve the prior.

Definition (credible and confidence sets). A level- $(1 - α)$ credible set $C_{n}$ is any (data-measurable) set with posterior mass $Π (C_{n} ∣ X_{1}, \dots, X_{n}) = 1 - α$ . A level- $(1 - α)$ confidence set has frequentist coverage $P_{θ_{0}} (θ_{0} \in C_{n}) \to 1 - α$ . The BvM corollary is that natural credible sets are asymptotically confidence sets.

Definition (posterior consistency and contraction rate). The posterior is consistent at $θ_{0}$ if for every neighbourhood $U$ of $θ_{0}$ , $Π (U^{c} ∣ X_{1}, \dots, X_{n}) P_{θ_{0}} 0$ . It contracts at rate $ε_{n} ↓ 0$ (in a metric $d$ ) if $Π ({θ : d (θ, θ_{0}) > M_{n} ε_{n}} ∣ X_{1}, \dots, X_{n}) P_{θ_{0}} 0$ for every $M_{n} \to \infty$ . In the regular parametric case the rate is $ε_{n} = n^{- 1/2}$ , the scale on which BvM operates.

Counterexamples to common slips Intermediate+

BvM is about the whole posterior, not just its mean. The conclusion is total-variation closeness of the posterior to a Gaussian, which implies (but is strictly stronger than) posterior-mean $\approx$ MLE and matching variance. A model can have posterior-mean consistency without the full BvM total-variation statement.
The centring is the MLE (or any efficient estimator), not $θ_{0}$ . The posterior of $n (θ - θ_{0})$ is asymptotically $N (Δ_{n, θ_{0}}, I^{- 1})$ with a random Gaussian shift $Δ_{n, θ_{0}} = I^{- 1} n^{- 1/2} \sum_{i} \dot{ℓ}_{θ_{0}} (X_{i})$ ; recentring at $\hat{θ}_{n}$ removes that shift and leaves the clean $N (0, I^{- 1})$ . Writing the limit as centred at $θ_{0}$ misplaces the random sampling fluctuation.
Prior positivity at $θ_{0}$ is needed. If the prior assigns zero density (or zero mass to every neighbourhood) at the true value, the posterior cannot concentrate there and BvM fails; "any prior" means any prior with a continuous, strictly positive density near $θ_{0}$ and sufficiently light tails.
BvM is a parametric phenomenon. In infinite-dimensional (nonparametric) models the local-asymptotic-normality scaling and the Gaussian limit need not hold, and credible sets can have asymptotic coverage zero; the theorem of this unit is for fixed finite-dimensional $Θ$ .

Key theorem with proof Intermediate+

The signature result is that, under parametric regularity and a prior positive and continuous at the truth, the recentred posterior converges in total variation to the fixed Gaussian $N (0, I (θ_{0})^{- 1})$ , independently of the prior. The engine is the local-asymptotic-normality (LAN) expansion of 45.04.06: the log-likelihood, viewed as a function of the local parameter, is asymptotically a quadratic with a random linear term, so the posterior — likelihood times a locally-constant prior — is asymptotically Gaussian.

Theorem (Bernstein-von Mises). Let ${P_{θ}}$ be differentiable in quadratic mean at the interior point $θ_{0}$ with nonsingular Fisher information $I (θ_{0})$ , and suppose a uniformly consistent sequence of tests of $θ_{0}$ against ${∣ θ - θ_{0} ∣ \geq ε}$ exists for every $ε > 0$ (so the posterior is consistent). Let the prior $Π$ have a density that is continuous and positive in a neighbourhood of $θ_{0}$ . Then for the MLE $\hat{θ}_{n}$ ,

Π (n (θ - \hat{θ}_{n}) \in \cdot X_{1}, \dots, X_{n}) - N (0, I (θ_{0})^{- 1})_{TV} P_{θ_{0}} 0.

Proof. Work with the local parameter $h = n (θ - θ_{0})$ ; the change of centre to $\hat{θ}_{n}$ is made at the end using $n (\hat{θ}_{n} - θ_{0}) = Δ_{n} + o_{P} (1)$ with $Δ_{n} = I (θ_{0})^{- 1} n^{- 1/2} \sum_{i} \dot{ℓ}_{θ_{0}} (X_{i})$ , the linearisation of 45.04.03.

Step 1 (LAN expansion). By differentiability in quadratic mean, the log-likelihood ratio of $P_{θ_{0} + h / n}^{\otimes n}$ to $P_{θ_{0}}^{\otimes n}$ admits the LAN expansion of 45.04.06:

lo g \frac{\prod _{i} f ( X _{i} ∣ θ _{0} + h / n )}{\prod _{i} f ( X _{i} ∣ θ _{0} )} = h^{⊤} Δ_{n} I (θ_{0})^{- 1} \cdot I (θ_{0}) - \frac{1}{2} h^{⊤} I (θ_{0}) h + r_{n} (h),

which we write compactly as $h^{⊤} I (θ_{0}) Δ_{n}^{*} - \frac{1}{2} h^{⊤} I (θ_{0}) h + r_{n} (h)$ with $Δ_{n}^{*} = n (\hat{θ}_{n} - θ_{0}) = I (θ_{0})^{- 1} n^{- 1/2} \sum_{i} \dot{ℓ}_{θ_{0}} (X_{i}) + o_{P} (1)$ , where the remainder satisfies $r_{n} (h) P_{θ_{0}} 0$ for each fixed $h$ and, by the consistency hypothesis, uniformly over $h$ in compacta after the posterior is restricted to a shrinking neighbourhood of $θ_{0}$ .

Step 2 (posterior density in the local parameter). The posterior density of $h$ is proportional to likelihood times prior:

π_{n}^{h} (h) \propto exp (h^{⊤} I Δ_{n}^{*} - \frac{1}{2} h^{⊤} I h + r_{n} (h)) π (θ_{0} + h / n),

with $I = I (θ_{0})$ . The prior factor is the value of $π$ at $θ_{0} + h / n$ . By continuity and positivity of $π$ at $θ_{0}$ , $π (θ_{0} + h / n) \to π (θ_{0}) > 0$ for each $h$ , so the prior contributes a factor tending to the constant $π (θ_{0})$ on compacta — it cancels in the normalisation. This is the analytic content of "the prior washes out".

Step 3 (Laplace/Gaussian limit on compacta). Dropping the prior factor (it tends to a constant) and the vanishing remainder, the exponent is $h^{⊤} I Δ_{n}^{*} - \frac{1}{2} h^{⊤} I h$ , which completes the square to $- \frac{1}{2} (h - Δ_{n}^{*})^{⊤} I (h - Δ_{n}^{*}) + \frac{1}{2} (Δ_{n}^{*})^{⊤} I Δ_{n}^{*}$ . The $h$ -dependent part is the kernel of $N (Δ_{n}^{*}, I^{- 1})$ . Hence the unnormalised local posterior density converges, for each fixed $h$ , to a constant multiple of the $N (Δ_{n}^{*}, I^{- 1})$ density. Recentring at $\hat{θ}_{n}$ , i.e. replacing $h$ by $h + Δ_{n}^{*}$ , shifts this to $N (0, I^{- 1})$ .

Step 4 (from pointwise to total variation). Let $g_{n} (h)$ be the recentred local posterior density and $ϕ (h)$ the $N (0, I^{- 1})$ density. Posterior consistency forces the posterior mass outside any fixed ball around the centre to vanish, so the tails of $g_{n}$ are negligible. On compacta $g_{n} \to ϕ$ pointwise in $P_{θ_{0}}$ -probability by Steps 2-3. Scheffé's lemma upgrades pointwise convergence of densities (with the integrable-dominated tails supplied by consistency) to $L^{1}$ convergence: $\int ∣ g_{n} - ϕ ∣ P_{θ_{0}} 0$ . Since total variation is half the $L^{1}$ distance, $∥ Π_{n}^{h} - N (0, I^{- 1}) ∥_{TV} P_{θ_{0}} 0$ . $□$

Bridge. This theorem builds toward the credible-set coverage corollary and the posterior-contraction theory of the Advanced section, and it appears again in the asymptotic equivalence of Bayes and maximum-likelihood point estimates 45.03.01 and in likelihood-based confidence regions 45.04.03. This is exactly the LAN expansion of 45.04.06 read through Bayes' rule rather than through the score equation: where 45.04.03 inverts the observed information to get the MLE's law, BvM observes that the posterior — likelihood times a locally-constant prior — is the same quadratic exponentiated, so it must be the same Gaussian. The theorem generalises the finite-sample conjugate-normal calculation, where the posterior is exactly $N (\hat{θ}_{n}, (n I)^{- 1})$ , to every regular family asymptotically. The central insight is that at scale $n^{- 1/2}$ a smooth prior is locally flat, so the posterior shape is dictated entirely by the likelihood; the prior survives only in the normalising constant and therefore cannot affect the limiting law. Putting these together, the posterior mean is asymptotically the MLE and inherits its efficiency, so the Bayesian estimator of 45.03.01 is dual to the frequentist estimator of 45.04.03 in the large-sample limit, and the two inferential traditions report the same intervals.

Exercises Intermediate+

Exercise 3 (medium, symbolic).

For $X_{i} ∣ θ \sim N (θ, σ^{2})$ with known $σ^{2}$ and conjugate prior $θ \sim N (μ_{0}, τ^{2})$ , write the exact posterior and show directly that, as $n \to \infty$ , it is $N (\hat{θ}_{n}, σ^{2} / n) + o (1)$ with $\hat{θ}_{n} = \overset{ˉ}{X}_{n}$ , verifying BvM in closed form.

Hint

The posterior is normal with precision $n / σ^{2} + 1/ τ^{2}$ ; track the prior precision as $n \to \infty$ .

Answer

The posterior is $N (μ_{n}, τ_{n}^{2})$ with $τ_{n}^{- 2} = n / σ^{2} + 1/ τ^{2}$ and $μ_{n} = τ_{n}^{2} (n \overset{ˉ}{X}_{n} / σ^{2} + μ_{0} / τ^{2})$ . As $n \to \infty$ , $τ_{n}^{- 2} = (n / σ^{2}) (1 + σ^{2} / (n τ^{2})) = (n / σ^{2}) (1 + o (1))$ , so $τ_{n}^{2} = (σ^{2} / n) (1 + o (1))$ , the inverse Fisher information $I (θ)^{- 1} / n = σ^{2} / n$ . The mean is $μ_{n} = \overset{ˉ}{X}_{n} + τ_{n}^{2} (μ_{0} / τ^{2} - \overset{ˉ}{X}_{n} / τ^{2}) = \overset{ˉ}{X}_{n} + O (1/ n)$ , so $μ_{n} = \hat{θ}_{n} + o (n^{- 1/2})$ . Recentring, $n (θ - \hat{θ}_{n})$ has posterior law $N (n (μ_{n} - \overset{ˉ}{X}_{n}), n τ_{n}^{2}) \to N (0, σ^{2})$ , the BvM Gaussian. The prior parameters $μ_{0}, τ^{2}$ disappear at rate $1/ n$ .

Exercise 4 (medium, symbolic).

(Credible-set coverage.) Let $C_{n} = [\hat{θ}_{n} - z_{1 - α /2} \overset{s}{^}_{n}, \hat{θ}_{n} + z_{1 - α /2} \overset{s}{^}_{n}]$ with $\overset{s}{^}_{n}^{2} = (n \hat{I}_{n})^{- 1}$ a consistent variance estimate. Using BvM, show $Π (C_{n} ∣ X_{1}, \dots, X_{n}) \to 1 - α$ in probability, and that $C_{n}$ also has frequentist coverage $1 - α$ .

Hint

Express the posterior mass of $C_{n}$ as a Gaussian probability via the recentred posterior; then use 45.04.03 for the sampling distribution of $\hat{θ}_{n}$ .

Answer

In the local parameter, $C_{n}$ corresponds to ${h : ∣ h ∣ \leq z_{1 - α /2} n \overset{s}{^}_{n}} = {∣ h ∣ \leq z_{1 - α /2} \hat{I}_{n}^{- 1/2}}$ . By BvM the posterior of $h$ is $N (0, I^{- 1})$ in TV, and TV convergence preserves probabilities of fixed sets, so $Π (C_{n} ∣ \cdot) \to N (0, I^{- 1}) ({∣ h ∣ \leq z_{1 - α /2} I^{- 1/2}}) = Φ (z_{1 - α /2}) - Φ (- z_{1 - α /2}) = 1 - α$ . For frequentist coverage, $θ_{0} \in C_{n} ⟺ ∣ n (\hat{θ}_{n} - θ_{0}) ∣ \hat{I}_{n}^{1/2} \leq z_{1 - α /2}$ , and by 45.04.03 $n (\hat{θ}_{n} - θ_{0}) \hat{I}_{n}^{1/2} d N (0, 1)$ , giving coverage $1 - α$ . The Bayesian and frequentist guarantees coincide.

Exercise 5 (medium, symbolic).

Show that under BvM the posterior mean $\hat{θ}_{n}^{Bayes} = E [θ ∣ X_{1}, \dots, X_{n}]$ satisfies $n (\hat{θ}_{n}^{Bayes} - \hat{θ}_{n}) P_{θ_{0}} 0$ , hence the Bayes estimator of 45.03.01 is asymptotically efficient.

Hint

TV convergence with a uniformly integrable first moment implies convergence of means; the recentred posterior mean tends to the mean of $N (0, I^{- 1})$ , which is zero.

Answer

The recentred posterior mean is $E [n (θ - \hat{θ}_{n}) ∣ \cdot] = n (\hat{θ}_{n}^{Bayes} - \hat{θ}_{n})$ . BvM gives TV convergence of the recentred posterior to $N (0, I^{- 1})$ ; with the (standard, separately verified) uniform integrability of $h$ under the posterior, the first moment converges to the mean of $N (0, I^{- 1})$ , namely $0$ . Hence $n (\hat{θ}_{n}^{Bayes} - \hat{θ}_{n}) P_{θ_{0}} 0$ . Since $n (\hat{θ}_{n} - θ_{0}) d N (0, I^{- 1})$ is efficient by 45.04.03, so is the Bayes estimator: it shares the MLE's asymptotic law to first order.

Exercise 6 (medium, symbolic).

(Random shift.) Show that the posterior of $n (θ - θ_{0})$ (centred at the truth, not the MLE) is asymptotically $N (Δ_{n}^{*}, I^{- 1})$ with $Δ_{n}^{*} = n (\hat{θ}_{n} - θ_{0})$ , and explain why this is not asymptotically pivotal while the $\hat{θ}_{n}$ -centred version is.

Hint

The completing-the-square step in the Key theorem leaves the centre at $Δ_{n}^{*}$ ; that quantity is itself random and asymptotically $N (0, I^{- 1})$ .

Answer

Step 3 of the Key-theorem proof shows the $θ_{0}$ -centred local posterior is $N (Δ_{n}^{*}, I^{- 1})$ . The shift $Δ_{n}^{*} = n (\hat{θ}_{n} - θ_{0})$ is a function of the data and is itself asymptotically $N (0, I^{- 1})$ under $P_{θ_{0}}$ , so the $θ_{0}$ -centred posterior law depends on the (unknown) sampling fluctuation and is not pivotal. Recentring at $\hat{θ}_{n}$ subtracts $Δ_{n}^{*}$ , giving the data-free limit $N (0, I^{- 1})$ — a pivot. This is why BvM is always stated with the MLE (or an efficient estimator) as the centre.

Exercise 7 (hard, symbolic).

(Misspecified BvM.) When the data law $P$ is not in the model, the posterior concentrates at the Kullback-Leibler projection $θ_{*}$ , and the local posterior is asymptotically $N (0, V^{- 1})$ with $V = - E_{P} \nabla^{2} lo g f (\cdot ∣ θ_{*})$ , while the frequentist law of the MLE has the sandwich covariance $V^{- 1} W V^{- 1}$ with $W = Cov_{P} \nabla lo g f (\cdot ∣ θ_{*})$ . Show that credible sets then have wrong frequentist coverage unless $V = W$ .

Hint

Compare the posterior spread $V^{- 1}$ with the sampling spread $V^{- 1} W V^{- 1}$ ; coverage is correct only when the two match.

Answer

A BvM credible interval has half-width set by the posterior spread $V^{- 1}$ , so it is $\hat{θ}_{n} \pm z_{1 - α /2} (nV)^{- 1/2}$ scaled by $V^{- 1/2}$ . But $n (\hat{θ}_{n} - θ_{*}) d N (0, V^{- 1} W V^{- 1})$ , so correct frequentist coverage requires the interval half-width to track $(V^{- 1} W V^{- 1})^{1/2}$ . These agree only when $W = V$ , i.e. when the information equality (second Bartlett identity) holds — which under correct specification gives $V = W = I (θ_{0})$ . Under misspecification $V \neq = W$ in general, and the credible interval over- or under-covers; the fix is a sandwich-corrected ("BvM with the wrong variance") posterior or an explicit sandwich interval. This is the Bayesian face of the Huber sandwich of 45.04.03.

Exercise 8 (hard, symbolic).

(Boundary failure.) Let $X_{i} ∣ θ \sim N (θ, 1)$ with the constraint $θ \geq 0$ and prior uniform on $[0, \infty)$ , and suppose the true value is $θ_{0} = 0$ (on the boundary). Show that the recentred posterior does not converge to a (non-degenerate) Gaussian, so BvM fails, and identify the limit.

Hint

The posterior is a normal density restricted to $[0, \infty)$ ; with $θ_{0} = 0$ the MLE is $\hat{θ}_{n} = max (\overset{ˉ}{X}_{n}, 0)$ and the relevant local scale puts mass on a half-line.

Answer

With the flat prior on $[0, \infty)$ the posterior of $θ$ is $N (\overset{ˉ}{X}_{n}, 1/ n)$ truncated to $θ \geq 0$ . In the local parameter $h = n (θ - \hat{θ}_{n})$ with $\hat{θ}_{n} = max (\overset{ˉ}{X}_{n}, 0)$ , the support is $h \geq - n \hat{θ}_{n}$ , which when $θ_{0} = 0$ (so $\overset{ˉ}{X}_{n} = O_{P} (n^{- 1/2})$ , $\hat{θ}_{n} = O_{P} (n^{- 1/2})$ ) is asymptotically $h \geq - U$ with $U = (n \hat{θ}_{n}) = max (Z, 0)$ , $Z \sim N (0, 1)$ . The limiting local posterior is a standard normal truncated to a random half-line, not the full $N (0, 1)$ ; its shape depends on the random boundary distance and it is not the prior-free Gaussian BvM predicts. The interior-point hypothesis $θ_{0} \in int Θ$ is exactly what this example violates, and credible sets built from the symmetric Gaussian undercover near the boundary.

Advanced results Master

Posterior consistency: Doob and Schwartz

Two consistency theorems sit beneath BvM. Doob's theorem gives almost-everywhere consistency for free: for a dominated model with an identifiable parameter, the posterior is consistent at $θ_{0}$ for $Π$ -almost every $θ_{0}$ , by a martingale-convergence argument on the joint law ^{[Ghosh & Ramamoorthi]}. Its weakness is the null exceptional set, which can contain the very $θ_{0}$ a frequentist cares about. Schwartz's theorem repairs this with two checkable conditions: the prior must charge every Kullback-Leibler neighbourhood of $P_{θ_{0}}$ (the KL support condition), and there must exist a uniformly exponentially consistent sequence of tests of $θ_{0}$ against the complement of each neighbourhood. The KL-support condition pushes prior mass toward the truth; the test condition controls the likelihood ratio away from it, and Bayes' rule does the rest. In regular finite-dimensional models both conditions hold whenever the prior is positive at $θ_{0}$ , which is why the BvM hypothesis is stated so simply. The contraction rate refines consistency: matching the prior's local mass (a lower bound on $Π$ of an $ε_{n}$ -ball) against the metric entropy of the model and the exponential power of the tests yields $Π (d (θ, θ_{0}) > M ε_{n} ∣ X^{n}) \to 0$ , with $ε_{n} = n^{- 1/2}$ in the parametric case and slower polynomial rates in nonparametric problems ^{[Ghosal & van der Vaart]}.

The LAN structure and the limit experiment

BvM is the Bayesian shadow of local asymptotic normality. Under differentiability in quadratic mean the sequence of local models ${P_{θ_{0} + h / n}^{\otimes n} : h \in R^{k}}$ converges, in the Le Cam sense, to the Gaussian shift experiment ${N (h, I (θ_{0})^{- 1}) : h}$ of 45.04.06. In that limit experiment the posterior for a flat prior is exactly the Gaussian centred at the observation, and BvM is the statement that the finite- $n$ posterior inherits this limit. Contiguity of $P_{θ_{0}}^{\otimes n}$ and $P_{θ_{0} + h / n}^{\otimes n}$ , a consequence of LAN, is what lets the proof transfer the pointwise density convergence to total-variation convergence uniformly over local alternatives ^{[Le Cam & Yang]}. The same limit experiment explains the convolution and local-asymptotic-minimax efficiency of 45.04.03: the Gaussian shift is the canonical experiment whose optimal estimator is the observation itself, and both the MLE and the Bayes estimator realise it.

Semiparametric BvM and nonparametric failure

In semiparametric problems — a finite-dimensional parameter of interest $ψ (θ)$ alongside an infinite-dimensional nuisance — a BvM theorem holds for the marginal posterior of $n (ψ - \hat{ψ}_{n})$ provided the efficient influence function exists and a no-bias condition controls the nuisance, giving the efficient information bound as the limiting variance. The full infinite-dimensional case is where BvM most dramatically fails. Freedman's example exhibits a prior for which the posterior is consistent yet credible sets have asymptotic frequentist coverage zero: the rate of contraction and the geometry of the credible set diverge, so the prior-free Gaussian picture collapses. The repair is delicate. Either one restricts to functionals that are $n$ -estimable (where a semiparametric BvM survives), or one rescales credible sets by an inflation factor calibrated to the contraction rate, or one accepts that nonparametric Bayesian uncertainty quantification requires separate frequentist validation ^{[Ghosal & van der Vaart]}. The dimension-dependence is structural: the local prior is flat to $o (1)$ only when the perturbation $h / n$ shrinks faster than the model's effective complexity grows, which fails when the parameter dimension is comparable to $n$ .

Synthesis. The central insight is that at the $n^{- 1/2}$ scale a smooth prior is locally constant, so the posterior shape is the exponentiated LAN quadratic of 45.04.06 and nothing else, and this is exactly why the prior survives only in a normalising constant and washes out of the limit. The foundational reason Bayesian and frequentist inference agree asymptotically is that both read off the same Gaussian shift limit experiment: the MLE of 45.04.03 is its optimal estimator and the posterior of 45.03.01 is its Gaussian posterior, so the posterior mean is dual to the MLE and credible sets generalise into confidence sets. Putting these together, posterior consistency (Schwartz), the LAN expansion, and total-variation closure by Scheffé are one argument: KL-support pushes mass to the truth, contiguity transfers the quadratic expansion to local alternatives, and the prior-flatness cancellation delivers the prior-free Gaussian. The bridge is the limit experiment — it carries the efficiency of 45.04.03, the loss-indexed point estimates of 45.03.01, and the credible-set coverage corollary as facets of one Gaussian shift, while its breakdown in infinite dimensions, where the local prior is no longer flat relative to the model's complexity, is precisely where the Bayesian-frequentist agreement dissolves and credible sets stop being confidence sets.

Full proof set Master

Proposition 1 (the prior factor is locally constant). If $π$ is continuous and positive at $θ_{0}$ , then $sup_{∣ h ∣ \leq K} π (θ_{0} + h / n) - π (θ_{0}) \to 0$ for every $K$ , and $π (θ_{0}) > 0$ .

Proof. Continuity at $θ_{0}$ gives, for each $δ > 0$ , a neighbourhood on which $∣ π (θ) - π (θ_{0}) ∣ < δ$ . For $∣ h ∣ \leq K$ the argument $θ_{0} + h / n$ lies within $K / n$ of $θ_{0}$ , inside that neighbourhood once $n > (K / r)^{2}$ where $r$ is the neighbourhood radius. Hence the supremum is below $δ$ for large $n$ , and positivity of $π (θ_{0})$ is the hypothesis. $□$

Proposition 2 (LAN expansion of the local log-posterior). Under differentiability in quadratic mean at $θ_{0}$ with information $I = I (θ_{0})$ , the recentred local log-likelihood satisfies, for each fixed $h$ ,

lo g \frac{\prod _{i} f ( X _{i} ∣ θ ^ _{n} + h / n )}{\prod _{i} f ( X _{i} ∣ θ ^ _{n} )} = - \frac{1}{2} h^{⊤} I h + \tilde{r}_{n} (h), \tilde{r}_{n} (h) P_{θ_{0}} 0.

Proof. Expand both the numerator and denominator log-likelihoods about $θ_{0}$ using the 45.04.06 LAN form $lo g \prod_{i} f (X_{i} ∣ θ_{0} + u / n) / \prod_{i} f (X_{i} ∣ θ_{0}) = u^{⊤} I Δ_{n}^{*} - \frac{1}{2} u^{⊤} I u + r_{n} (u)$ with $r_{n} (u) P 0$ . Take $u = n (\hat{θ}_{n} - θ_{0}) + h = Δ_{n}^{*} + h$ for the numerator and $u = Δ_{n}^{*}$ for the denominator, and subtract:

(Δ_{n}^{*} + h)^{⊤} I Δ_{n}^{*} - \frac{1}{2} (Δ_{n}^{*} + h)^{⊤} I (Δ_{n}^{*} + h) - [(Δ_{n}^{*})^{⊤} I Δ_{n}^{*} - \frac{1}{2} (Δ_{n}^{*})^{⊤} I Δ_{n}^{*}] + r_{n} (Δ_{n}^{*} + h) - r_{n} (Δ_{n}^{*}) .

The linear-in- $h$ terms are $h^{⊤} I Δ_{n}^{*} - \frac{1}{2} (2 h^{⊤} I Δ_{n}^{*}) = 0$ , leaving $- \frac{1}{2} h^{⊤} I h$ plus the remainder $\tilde{r}_{n} (h) = r_{n} (Δ_{n}^{*} + h) - r_{n} (Δ_{n}^{*})$ , which is $o_{P} (1)$ since $Δ_{n}^{*} = O_{P} (1)$ and $r_{n}$ vanishes on compacta. $□$

Proposition 3 (pointwise density convergence). Let $g_{n} (h)$ be the recentred local posterior density. Under Propositions 1-2, $g_{n} (h) P_{θ_{0}} ϕ_{I} (h)$ for each $h$ , where $ϕ_{I}$ is the $N (0, I^{- 1})$ density.

Proof. The unnormalised recentred density is $\tilde{g}_{n} (h) = exp (- \frac{1}{2} h^{⊤} I h + \tilde{r}_{n} (h)) π (\hat{θ}_{n} + h / n)$ . By Proposition 1 (with $\hat{θ}_{n} P θ_{0}$ so the argument is within $O_{P} (K / n)$ of $θ_{0}$ ) the prior factor tends to $π (θ_{0})$ , and by Proposition 2 the exponent tends to $- \frac{1}{2} h^{⊤} I h$ . Thus $\tilde{g}_{n} (h) P π (θ_{0}) e^{- h^{⊤} I h /2}$ . The normalising constant $\int \tilde{g}_{n}$ converges (by the same convergence plus the dominated-tail control of Proposition 4) to $π (θ_{0}) \int e^{- h^{⊤} I h /2} d h = π (θ_{0}) (2 π)^{k /2} det (I)^{- 1/2}$ . The ratio is $g_{n} (h) \to e^{- h^{⊤} I h /2} / [(2 π)^{k /2} det (I)^{- 1/2}] = ϕ_{I} (h)$ . $□$

Proposition 4 (total-variation convergence via Scheffé). Under posterior consistency and Propositions 1-3, $∥ g_{n} - ϕ_{I} ∥_{L^{1}} P_{θ_{0}} 0$ , hence $∥ Π_{n}^{h} - N (0, I^{- 1}) ∥_{TV} P_{θ_{0}} 0$ .

Proof. Posterior consistency gives, for every $η > 0$ , a $K$ with $\int_{∣ h ∣ > K} g_{n} \leq η$ eventually with probability tending to one (the posterior mass outside an $O (n^{- 1/2})$ -ball of the centre, i.e. outside ${∣ h ∣ \leq K}$ , vanishes), and $\int_{∣ h ∣ > K} ϕ_{I} \leq η$ by Gaussian tails. On the compact ${∣ h ∣ \leq K}$ , Proposition 3 gives $g_{n} \to ϕ_{I}$ pointwise and the densities are uniformly integrable there (bounded by an integrable Gaussian envelope after the exponent control), so the bounded-convergence form of Scheffé's lemma yields $\int_{∣ h ∣ \leq K} ∣ g_{n} - ϕ_{I} ∣ \to 0$ in probability. Combining, $\int ∣ g_{n} - ϕ_{I} ∣ \leq \int_{∣ h ∣ \leq K} ∣ g_{n} - ϕ_{I} ∣ + \int_{∣ h ∣ > K} g_{n} + \int_{∣ h ∣ > K} ϕ_{I} \leq o_{P} (1) + 2 η$ . As $η$ is arbitrary, the $L^{1}$ distance is $o_{P} (1)$ , and $TV = \frac{1}{2} L^{1}$ gives the claim. $□$

Proposition 5 (credible sets are confidence sets). Under BvM, the equal-tailed level- $(1 - α)$ credible interval $C_{n} = \hat{θ}_{n} + n^{- 1/2} [q_{α /2}^{n}, q_{1 - α /2}^{n}]$ , with $q^{n}$ the posterior quantiles of $h$ , satisfies $P_{θ_{0}} (θ_{0} \in C_{n}) \to 1 - α$ .

Proof. TV convergence to $N (0, I^{- 1})$ forces the posterior quantiles of $h$ to converge to the corresponding $N (0, I^{- 1})$ quantiles $q_{α /2}, q_{1 - α /2}$ (TV convergence implies convergence of the CDF at continuity points, hence of quantiles). So $C_{n} = \hat{θ}_{n} + n^{- 1/2} [q_{α /2} + o_{P} (1), q_{1 - α /2} + o_{P} (1)]$ . Then $θ_{0} \in C_{n} ⟺ n (θ_{0} - \hat{θ}_{n}) \in [q_{α /2} + o_{P} (1), q_{1 - α /2} + o_{P} (1)]$ . By 45.04.03, $n (\hat{θ}_{n} - θ_{0}) d N (0, I^{- 1})$ , so $n (θ_{0} - \hat{θ}_{n})$ has the same symmetric limit, and $P_{θ_{0}} (θ_{0} \in C_{n}) \to N (0, I^{- 1}) ([q_{α /2}, q_{1 - α /2}]) = 1 - α$ . $□$

Connections Master

Bayes estimation under loss 45.03.01 supplies the posterior summaries that BvM controls asymptotically: the posterior mean, median, and mode all collapse onto the MLE in the limit because the posterior becomes the symmetric Gaussian $N (\hat{θ}_{n}, (n I)^{- 1})$ , so the loss-indexed distinction that matters in finite samples vanishes and the Bayes estimator inherits the efficiency of the frequentist estimator.
Asymptotic normality and efficiency of the MLE 45.04.03 is the frequentist statement BvM matches: the same inverse-Fisher-information covariance $I (θ_{0})^{- 1}$ governs both the sampling law of $n (\hat{θ}_{n} - θ_{0})$ and the limiting posterior spread, which is why a Bayesian credible interval and a Wald confidence interval coincide asymptotically and why the posterior mean is asymptotically efficient.
Local asymptotic normality 45.04.06 is the engine of the proof: the LAN quadratic expansion of the log-likelihood ratio is what makes the local posterior — likelihood times a locally-flat prior — asymptotically Gaussian, and the Gaussian shift limit experiment it defines is the common limit from which both the MLE's efficiency and the posterior's BvM Gaussian are read off.
Fisher information and the Cramér-Rao bound 45.01.05 determines the spread of the limiting posterior: $I (θ_{0})^{- 1}$ is the inverse of the information matrix defined there, so the asymptotic credible-set width is the inverse-information scale, and the information equality $I = - E [\nabla^{2} lo g f]$ is what forces the misspecification sandwich $V^{- 1} W V^{- 1}$ to collapse to $I^{- 1}$ under correct specification.
Chi-squared limits of the Wald, score, and likelihood-ratio tests 45.06.01 share BvM's mechanism from the Bayesian side: the posterior credible region is a quadratic form in $n (θ - \hat{θ}_{n})$ whose Gaussian law gives the same $χ^{2}$ calibration as the Wald test, so Bayesian and frequentist hypothesis assessments also align asymptotically through the common LAN quadratic.

Historical & philosophical context Master

The result is named for two strands. Sergei Bernstein (1917) and Richard von Mises (1931) observed, for specific models, that the large-sample posterior becomes approximately normal and free of the prior, an early signal that subjective and objective inference converge with data ^{[von Mises 1931]}. Pierre-Simon Laplace had used the same Gaussian approximation to the posterior a century earlier in his work on inverse probability, so the asymptotic normality of the posterior is sometimes called the Laplace approximation.

The modern rigorous theorem, with explicit regularity conditions and a total-variation conclusion, is due to Lucien Le Cam, whose 1953 study of maximum-likelihood and related Bayes estimates placed the phenomenon inside the local-asymptotic-normality framework and the theory of limit experiments ^{[Le Cam 1953]}. Le Cam's programme, developed in his 1986 Asymptotic Methods in Statistical Decision Theory and with Grace Lo Yang, identified the Gaussian shift experiment as the structure unifying the efficiency of the MLE and the BvM behaviour of the posterior. Joseph Doob (1949) gave the almost-everywhere posterior-consistency theorem via martingale convergence, and Lorraine Schwartz (1965) gave the checkable KL-support-plus-tests consistency theorem that supplies the BvM hypotheses. The infinite-dimensional limits of the phenomenon were charted by David Freedman, whose 1963 and later examples showed that nonparametric posteriors can be consistent yet produce credible sets with vanishing frequentist coverage ^{[Ghosh & Ramamoorthi]}, and by Subhashis Ghosal and Aad van der Vaart, whose contraction-rate theory and nonparametric BvM analysis ^{[Ghosal & van der Vaart]} delineate when the parametric agreement survives and when it breaks.

Bibliography Master

@article{bernstein1917,
  author    = {Bernstein, Sergei N.},
  title     = {Theory of Probability},
  note      = {In Russian; the early posterior-normality observation},
  year      = {1917}
}

@book{vonMises1931,
  author    = {von Mises, Richard},
  title     = {Wahrscheinlichkeitsrechnung und ihre Anwendung in der Statistik und theoretischen Physik},
  publisher = {Franz Deuticke},
  address   = {Leipzig and Vienna},
  year      = {1931}
}

@article{leCam1953,
  author  = {Le Cam, Lucien},
  title   = {On some asymptotic properties of maximum likelihood estimates and related Bayes' estimates},
  journal = {University of California Publications in Statistics},
  volume  = {1},
  year    = {1953},
  pages   = {277--330}
}

@article{doob1949,
  author    = {Doob, Joseph L.},
  title     = {Application of the theory of martingales},
  journal   = {Le Calcul des Probabilit\'es et ses Applications (Colloques Internationaux du CNRS)},
  year      = {1949},
  pages     = {23--27}
}

@article{schwartz1965,
  author  = {Schwartz, Lorraine},
  title   = {On Bayes procedures},
  journal = {Zeitschrift f\"ur Wahrscheinlichkeitstheorie und Verwandte Gebiete},
  volume  = {4},
  year    = {1965},
  pages   = {10--26}
}

@book{leCam1986,
  author    = {Le Cam, Lucien},
  title     = {Asymptotic Methods in Statistical Decision Theory},
  series    = {Springer Series in Statistics},
  publisher = {Springer},
  year      = {1986}
}

@book{leCamYang2000,
  author    = {Le Cam, Lucien and Yang, Grace Lo},
  title     = {Asymptotics in Statistics: Some Basic Concepts},
  edition   = {2},
  series    = {Springer Series in Statistics},
  publisher = {Springer},
  year      = {2000}
}

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  publisher = {Cambridge University Press},
  year      = {1998}
}

@book{ghoshRamamoorthi2003,
  author    = {Ghosh, Jayanta K. and Ramamoorthi, R. V.},
  title     = {Bayesian Nonparametrics},
  series    = {Springer Series in Statistics},
  publisher = {Springer},
  year      = {2003}
}

@book{ghosalvanderVaart2017,
  author    = {Ghosal, Subhashis and van der Vaart, Aad W.},
  title     = {Fundamentals of Nonparametric Bayesian Inference},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  publisher = {Cambridge University Press},
  year      = {2017}
}

Prerequisites

45.03.01
45.04.03

Tier anchors

beginner: van der Vaart 1998 Asymptotic Statistics (Cambridge) §10.2 (the large-sample posterior is approximately normal, centred at the MLE, with spread the inverse Fisher information); Wasserman 2004 All of Statistics (Springer) §11.4-11.5 (the posterior concentrates and Bayesian credible sets become frequentist confidence sets)
intermediate: van der Vaart 1998 Asymptotic Statistics (Cambridge) §10.2 'Bernstein-von Mises Theorem' (the total-variation statement via the LAN expansion); Le Cam & Yang 2000 Asymptotics in Statistics (Springer) Ch. 6-8 (local asymptotic normality and the limit experiment)
master: van der Vaart 1998 Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics) Ch. 10 §10.2 (Bernstein-von Mises) with the LAN background of Ch. 7; Le Cam 1986 Asymptotic Methods in Statistical Decision Theory (Springer) Part 3 (limit experiments and the LAN/LAMN structure); Ghosal & van der Vaart 2017 Fundamentals of Nonparametric Bayesian Inference (Cambridge) Ch. 8, 10, 12 (posterior contraction rates and the failure / repair of BvM in infinite dimensions)

References

van der Vaart, A. W. — Asymptotic Statistics · Cambridge University Press, Cambridge Series in Statistical and Probabilistic Mathematics, 1998; Ch. 10 §10.2 (the Bernstein-von Mises theorem: posterior total-variation convergence to N(thetaHat, (nI)^{-1}), the LAN-based proof, asymptotic frequentist coverage of credible sets) with the local-asymptotic-normality background of Ch. 7
Le Cam, L. & Yang, G. L. — Asymptotics in Statistics: Some Basic Concepts (2nd ed.) · Springer Series in Statistics, 2000; Ch. 6-8 (local asymptotic normality, contiguity, the Gaussian limit experiment, and Bernstein-von Mises as the Bayesian face of LAN)
Ghosal, S. & van der Vaart, A. W. — Fundamentals of Nonparametric Bayesian Inference · Cambridge University Press, Cambridge Series in Statistical and Probabilistic Mathematics, 2017; Ch. 6-8 (posterior consistency, Schwartz's theorem, contraction rates) and Ch. 10, 12 (semiparametric and nonparametric Bernstein-von Mises, and its failure)
Ghosh, J. K. & Ramamoorthi, R. V. — Bayesian Nonparametrics · Springer Series in Statistics, 2003; Ch. 1-4 (parametric Bernstein-von Mises, posterior consistency, Doob's theorem, Schwartz's theorem) and Ch. 5 (Dirichlet-process priors and consistency failures)
Le Cam, L. — On some asymptotic properties of maximum likelihood estimates and related Bayes' estimates · University of California Publications in Statistics 1 (1953), 277-330 (the modern rigorous form of the Bernstein-von Mises phenomenon for regular parametric models)

Estimated time

beginner: 20m
intermediate: 55m
master: 95m