Consistency of Estimators and the Modes of Stochastic Convergence in Statistics

Intuition Beginner

An estimator is a recipe that turns a sample of data into a guess about some unknown number — the average height in a population, the failure rate of a part, the true mean of a measurement. You only ever see a finite sample, so your guess is itself random: a different sample gives a different number. The basic demand we make of a good recipe is modest but essential. As you collect more data, your guess should home in on the true value. A recipe with this property is called consistent, and a recipe without it is broken in a deep way: more data does not help.

Consistency is a promise about a long sequence of guesses, one for each sample size, not about any single guess. To make it precise you must say what it means for a sequence of random guesses to settle down to a fixed target. There turn out to be several distinct ways to say this, and they differ. One asks that big misses become rare as the sample grows. A stronger one asks that, for almost every imaginable run of data collection, the guesses actually converge as a sequence of numbers. A different one again tracks the whole spread of possible guesses and asks that this spread settle into a fixed shape.

These different notions of settling down are the modes of stochastic convergence. They form the grammar of the entire subject of large-sample statistics: every theorem about what happens to an estimator as data accumulate is phrased in one of them, and knowing which one — and how to trade one for another — is the core skill.

The one-sentence takeaway: an estimator is consistent when its guesses close in on the true value as the sample grows, and the modes of stochastic convergence are the precise vocabulary for saying how a sequence of random quantities settles down.

Visual Beginner

Picture a target with a bullseye marking the true value. Each sample size gives you a cloud of possible estimates — the spread of guesses you might get from different samples of that size. As the sample size grows, watch the cloud.

Sample size	Where the cloud sits	What it means
Small	Wide cloud, centered roughly on the bullseye	Guesses vary a lot; big misses common
Medium	Tighter cloud, hugging the bullseye	Guesses cluster closer; big misses rarer
Large	A tight knot right on the bullseye	Almost every guess lands near the true value
Growing without bound	The knot collapses to the bullseye	The estimator is consistent

The picture captures the weakest useful mode: the chance of landing far from the bullseye shrinks toward zero. Stronger modes add promises about whole runs of data or about the exact shape of the shrinking cloud, but the collapsing-cloud image is the one to keep in mind first.

Worked example Beginner

We check that the sample average is a consistent estimator of the true mean, using a small numerical run with a fair six-sided die, whose true mean is $3.5$.

Step 1. The recipe. Roll the die $n$ times and report the running average of the outcomes as your guess for the true mean. This is the sample average, the most basic estimator there is.

Step 2. A short run. Suppose the first eight rolls are $2,5,1,6,3,4,2,5$. The running averages are: after 1 roll $2$; after 2 rolls $3.5$; after 3 rolls $2.67$; after 4 rolls $3.5$; after 5 rolls $3.4$; after 6 rolls $3.5$; after 7 rolls $3.29$; after 8 rolls $3.5$. The guesses swing around but stay in the neighborhood of $3.5$.

Step 3. A longer run. Extend to 100 rolls and suppose the total is $352$. The average is $352/100=3.52$, within $0.02$ of the target. Extend to 10000 rolls with total $34850$: the average is $3.485$, within $0.015$.

Step 4. Why it keeps improving. The spread of the sample average shrinks as the sample grows: doubling the data does not move the center, which sits at the true mean, but it pulls the typical-size miss down. So the chance of a guess landing more than any fixed distance from $3.5$ keeps dropping toward zero.

Step 5. What this tells us. The sample average is a consistent estimator of the true mean: feed it more rolls and its guess closes in on $3.5$. This is the most important single instance of consistency, and the reason it works is the law of large numbers — the same principle that says long-run averages settle to the expected value. Every more elaborate estimator is judged against this baseline.

Check your understanding Beginner

Formal definition Intermediate+

Throughout, $(\Omega ,\mathcal{F},\mathbb{P})$ is a probability space, random vectors are Borel-measurable maps into ${\mathbb{R}}^k$, and $\stackrel{d}{\to }$, $\stackrel{\mathbb{P}}{\to }$, $\stackrel{a.s.}{\to }$, $\stackrel{L^p}{\to }$ denote the four standard modes defined below. The underlying limit theorems — the strong law of large numbers 37.02.02 and the central limit theorem through the Lévy continuity theorem 37.03.01 — are taken as given; this unit organises the modes of convergence and the order calculus that statistics builds on top of them.

Definition (the four modes). Let $X_n,X$ be random vectors in ${\mathbb{R}}^k$.

• $X_n\stackrel{a.s.}{\to }X$ (almost surely) if $\mathbb{P}(\{\omega :X_n(\omega )\to X(\omega )\})=1$.
• $X_n\stackrel{\mathbb{P}}{\to }X$ (in probability) if $\mathbb{P}(|X_n-X|>\epsilon )\to 0$ for every $\epsilon >0$, where $|\cdot |$ is the Euclidean norm.
• $X_n\stackrel{L^p}{\to }X$ (in $L^p$, $p\ge 1$) if $\mathbb{E}|X_n-X{|}^p\to 0$.
• $X_n\stackrel{d}{\to }X$ (in distribution, or weakly) if $\mathbb{E}f(X_n)\to \mathbb{E}f(X)$ for every bounded continuous $f:{\mathbb{R}}^k\to \mathbb{R}$; equivalently $F_n(x)\to F(x)$ at every continuity point $x$ of the limit distribution function $F$.

Convergence in distribution depends only on the laws and the limit may be specified by its law alone; the other three modes are statements about the random vectors as functions on $\Omega$.

Definition (the implication lattice). The four modes are ordered by strength. Almost-sure convergence and $L^p$-convergence each imply convergence in probability; convergence in probability implies convergence in distribution; convergence in distribution to a constant implies convergence in probability to that constant. Higher $L^p$ implies lower: $L^q\Rightarrow L^p$ for $q\ge p$. No other implications hold in general. Symbolically, with $\Rightarrow$ meaning "implies",

$$\left(X_n\stackrel{a.s.}{\to }X\right)\text{or}\left(X_n\stackrel{L^p}{\to }X\right)\Rightarrow \left(X_n\stackrel{\mathbb{P}}{\to }X\right)\Rightarrow \left(X_n\stackrel{d}{\to }X\right),$$

and the last arrow reverses when $X\equiv c$ is degenerate.

Definition (stochastic order: $o_{\mathbb{P}}$ and $O_{\mathbb{P}}$). Let $(R_n)$ be a sequence of positive random variables. A sequence $(X_n)$ of random vectors is $o_{\mathbb{P}}(R_n)$ if $X_n/R_n\stackrel{\mathbb{P}}{\to }0$, and $O_{\mathbb{P}}(R_n)$ (bounded in probability relative to $R_n$) if $X_n/R_n$ is uniformly tight: for every $\epsilon >0$ there is an $M$ with ${\sup }_n\mathbb{P}(|X_n/R_n|>M)<\epsilon$. Thus $X_n=o_{\mathbb{P}}(1)$ means $X_n\stackrel{\mathbb{P}}{\to }0$, and $X_n=O_{\mathbb{P}}(1)$ means $(X_n)$ is uniformly tight. The notation is read as an identity but means containment in the indicated stochastic-order class, exactly mirroring the deterministic Landau symbols.

Definition (consistency of an estimator). Let $\{P_{\theta }:\theta \in \Theta \}$ be a statistical model with $\Theta \subseteq {\mathbb{R}}^d$, and let $T_n=T_n(X_1,\dots ,X_n)$ be a sequence of estimators of a parameter $\psi (\theta )$. The sequence $(T_n)$ is weakly consistent (at $\theta$) if $T_n\stackrel{{\mathbb{P}}_{\theta }}{\to }\psi (\theta )$, and strongly consistent if $T_n\stackrel{a.s.}{\to }\psi (\theta )$, under $P_{\theta }$, for every $\theta \in \Theta$. Unqualified "consistent" means weakly consistent. Consistency is the minimal large-sample demand: an inconsistent estimator does not improve with data.

Counterexamples to common slips Intermediate+

• Convergence in probability does not give almost-sure convergence. On $([0,1],\mathcal{B},\text{Lebesgue})$ the "typewriter" sequence of indicators of dyadic subintervals $[k/2^j,(k+1)/2^j]$, enumerated so the lengths shrink, converges to $0$ in probability but at no point $\omega$ converges to $0$. A consistency proof via the law of large numbers that uses only the weak law delivers only weak consistency.

• Convergence in distribution does not give convergence in probability. If $X\sim N(0,1)$ and $X_n=(-1{)}^{n}X$, then $X_n\stackrel{d}{\to }N(0,1)$ (each $X_n$ is standard normal) yet $X_n$ does not converge in probability to anything, since $X_n-X_{n+1}=\pm 2X$ does not vanish. Distributional convergence to a non-degenerate limit says nothing about the random variables jointly.

• The exception is a degenerate limit. $X_n\stackrel{d}{\to }c$ for a constant $c$ does upgrade to $X_n\stackrel{\mathbb{P}}{\to }c$, because the event $\{|X_n-c|>\epsilon \}$ is read off the limiting distribution function, which is a step at $c$. This is the form in which the weak law and Slutsky's lemma are usually invoked.

• $L^p$ and almost-sure convergence are incomparable. The typewriter sequence converges in $L^1$ but not a.s.; conversely $X_n=n{1}_{[0,1/n]}$ on $([0,1],\text{Lebesgue})$ converges to $0$ a.s. (and in probability) but $\mathbb{E}|X_n|=1\nrightarrow 0$, so it fails $L^1$. Neither mode implies the other; both imply convergence in probability.

• $O_{\mathbb{P}}(1)$ is tightness, not boundedness. A sequence can be $O_{\mathbb{P}}(1)$ while ${\sup }_n|X_n|=\infty$ almost surely; the requirement is only that the mass not escape to infinity uniformly in $n$. Confusing $O_{\mathbb{P}}(1)$ with almost-sure boundedness breaks the order calculus.

Key theorem with proof Intermediate+

Theorem (the implication lattice, Slutsky's lemma, and the continuous-mapping theorem). Let $X_n,X,Y_n$ be random vectors in ${\mathbb{R}}^k$ and $c\in {\mathbb{R}}^k$ a constant.

(i) Lattice. $X_n\stackrel{a.s.}{\to }X$ implies $X_n\stackrel{\mathbb{P}}{\to }X$; $X_n\stackrel{L^p}{\to }X$ implies $X_n\stackrel{\mathbb{P}}{\to }X$; and $X_n\stackrel{\mathbb{P}}{\to }X$ implies $X_n\stackrel{d}{\to }X$.

(ii) Continuous mapping. If $g:{\mathbb{R}}^k\to {\mathbb{R}}^m$ is continuous at every point of a set $C$ with $\mathbb{P}(X\in C)=1$, then $X_n\stackrel{d}{\to }X$ implies $g(X_n)\stackrel{d}{\to }g(X)$; the same implication holds with $\stackrel{\mathbb{P}}{\to }$ throughout.

(iii) Slutsky. If $X_n\stackrel{d}{\to }X$ and $Y_n\stackrel{\mathbb{P}}{\to }c$, then $(X_n,Y_n)\stackrel{d}{\to }(X,c)$; consequently $X_n+Y_n\stackrel{d}{\to }X+c$, $Y_n^{\mathsf{T}}{X}_n\stackrel{d}{\to }{c}^{\mathsf{T}}X$, and $X_n/Y_n\stackrel{d}{\to }X/c$ when $k=1$ and $c\ne 0$.

Proof. (i). For the first arrow, set $A_n^{\epsilon }=\{{\sup }_{m\ge n}|X_m-X|>\epsilon \}$. The events $A_n^{\epsilon }$ decrease in $n$, and their intersection is contained in $\{X_n\nrightarrow X\}$, a null event by hypothesis; by continuity of measure $\mathbb{P}(A_n^{\epsilon })\to 0$, and $\mathbb{P}(|X_n-X|>\epsilon )\le \mathbb{P}(A_n^{\epsilon })\to 0$. For the second arrow, Markov's inequality gives $\mathbb{P}(|X_n-X|>\epsilon )\le {\epsilon }^{-p}\mathbb{E}|X_n-X{|}^p\to 0$. For the third arrow, let $f$ be bounded and Lipschitz with constant $L$ and bound $\Vert f{\Vert }_{\infty }=B$. For $\epsilon >0$,

$$|\mathbb{E}f(X_n)-\mathbb{E}f(X)|\le \mathbb{E}|f(X_n)-f(X)|\le L\epsilon +2B\mathbb{P}(|X_n-X|>\epsilon ).$$

The second term vanishes as $n\to \infty$, leaving ${\mathrm{limsup}}_n|\mathbb{E}f(X_n)-\mathbb{E}f(X)|\le L\epsilon$; let $\epsilon \downarrow 0$. Convergence of $\mathbb{E}f(X_n)$ for all bounded Lipschitz $f$ is equivalent to weak convergence (the portmanteau theorem, stated below), so $X_n\stackrel{d}{\to }X$.

(ii). Take the distributional case; the in-probability case is the typewriter-style argument along subsequences. Let $f$ be bounded continuous; then $f\circ g$ is bounded and continuous at every point of $C$. The portmanteau theorem in the form "$X_n\stackrel{d}{\to }X$ iff $\mathbb{E}h(X_n)\to \mathbb{E}h(X)$ for every bounded $h$ continuous off a ${\mu }_X$-null set" applies to $h=f\circ g$, whose discontinuity set lies in $C^c$ with $\mathbb{P}(X\in C^c)=0$. Hence $\mathbb{E}f(g(X_n))\to \mathbb{E}f(g(X))$, which is $g(X_n)\stackrel{d}{\to }g(X)$.

(iii). First, $Y_n\stackrel{\mathbb{P}}{\to }c$ with $c$ constant implies $Y_n\stackrel{d}{\to }c$, and joint tightness of $(X_n,Y_n)$ follows from tightness of each coordinate. It suffices to show $(X_n,Y_n)\stackrel{d}{\to }(X,c)$; the named consequences then follow from the continuous-mapping theorem applied to $(x,y)\mapsto x+y$, $(x,y)\mapsto y^{\mathsf{T}}x$, and $(x,y)\mapsto x/y$ (continuous on $\{y\ne 0\}$, a full-measure set for the limit $(X,c)$ since $c\ne 0$). For a bounded Lipschitz $f$ on ${\mathbb{R}}^{2k}$ with constant $L$,

$$|\mathbb{E}f(X_n,Y_n)-\mathbb{E}f(X_n,c)|\le \mathbb{E}\min (2\Vert f{\Vert }_{\infty },L|Y_n-c|)\to 0$$

by dominated convergence, using $Y_n\stackrel{\mathbb{P}}{\to }c$. And $\mathbb{E}f(X_n,c)\to \mathbb{E}f(X,c)$ because $x\mapsto f(x,c)$ is bounded continuous and $X_n\stackrel{d}{\to }X$. Combining, $\mathbb{E}f(X_n,Y_n)\to \mathbb{E}f(X,c)$, which is the joint convergence. $\square$

Portmanteau theorem (stated). For random vectors in ${\mathbb{R}}^k$ the following are equivalent: $X_n\stackrel{d}{\to }X$; $\mathbb{E}f(X_n)\to \mathbb{E}f(X)$ for all bounded Lipschitz $f$; ${\mathrm{liminf}}_n\mathbb{P}(X_n\in G)\ge \mathbb{P}(X\in G)$ for every open $G$; ${\mathrm{limsup}}_n\mathbb{P}(X_n\in F)\le \mathbb{P}(X\in F)$ for every closed $F$; $\mathbb{P}(X_n\in B)\to \mathbb{P}(X\in B)$ for every Borel $B$ with $\mathbb{P}(X\in \partial B)=0$; and $\mathbb{E}h(X_n)\to \mathbb{E}h(X)$ for every bounded $h$ continuous off a ${\mu }_X$-null set. (Proof in 37.03.01 and its successors; used here as a black box.)

Bridge. This lattice-plus-Slutsky package builds toward the delta method and the asymptotic distribution theory of every estimator in the chapter, and it appears again in the proof of asymptotic normality of M-estimators, where a score expansion is multiplied by a consistent estimate of an information matrix and Slutsky's lemma reassembles the pieces. The foundational reason consistency-plus-asymptotic-normality arguments work is that convergence in distribution and convergence in probability obey exactly these algebraic rules: a remainder shown to be $o_{\mathbb{P}}(1)$ may be discarded, and a nuisance factor shown to converge in probability to a constant may be frozen at its limit. This is exactly the mechanism Slutsky isolates, and putting these together with the continuous-mapping theorem gives the calculus that turns a one-line stochastic expansion into a limit law. The bridge is the passage from the raw limit theorems of the probability spine — the law of large numbers 37.02.02 and the central limit theorem 37.03.01 — to a working algebra of stochastic limits, and that algebra is what the rest of asymptotic statistics manipulates.

Exercises Intermediate+

Advanced results Master

Theorem 1 (Prohorov's theorem and the role of tightness). A family of probability measures on ${\mathbb{R}}^k$ (more generally on a Polish space) is relatively compact in the topology of weak convergence if and only if it is tight: for every $\epsilon >0$ there is a compact $K$ with ${\inf }_n{\mu }_n(K)>1-\epsilon$. In the order language, $(X_n)$ is $O_{\mathbb{P}}(1)$ precisely when $(\text{law of }{X}_n)$ is tight, so $O_{\mathbb{P}}(1)$ is the exact hypothesis under which every subsequence has a distributionally convergent further subsequence. This is the engine behind argmax-consistency and behind every "extract a convergent subsequence" step in asymptotics ^{[van der Vaart 1998]}.

Theorem 2 (the Mann-Wald stochastic-order calculus). The symbols $o_{\mathbb{P}}$ and $O_{\mathbb{P}}$ obey the algebra of their deterministic counterparts: $o_{\mathbb{P}}(1)+o_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$; $O_{\mathbb{P}}(1)+O_{\mathbb{P}}(1)=O_{\mathbb{P}}(1)$; $o_{\mathbb{P}}(1)O_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$; $o_{\mathbb{P}}(R_n)=R_n{o}_{\mathbb{P}}(1)$; and if $g$ is a function with $g(h)=o(|h{|}^p)$ as $h\to 0$, then $g(X_n)=o_{\mathbb{P}}(|X_n{|}^p)$ whenever $X_n\stackrel{\mathbb{P}}{\to }0$. These identities reduce stochastic Taylor expansions — the backbone of the delta method and of M-estimator asymptotics — to bookkeeping, with each discarded remainder certified as $o_{\mathbb{P}}$ of the leading term ^{[Mann-Wald 1943]}.

Theorem 3 (continuous mapping, full form, and Slutsky as a corollary). If $g$ is measurable and its discontinuity set $D_g$ satisfies $\mathbb{P}(X\in D_g)=0$, then $X_n\stackrel{d}{\to }X$ implies $g(X_n)\stackrel{d}{\to }g(X)$. Slutsky's lemma is the special case $g(x,y)=x+y$, $xy$, or $x/y$ applied to $(X_n,Y_n)\stackrel{d}{\to }(X,c)$, the joint convergence following because a constant limit forces joint from marginal convergence. The portmanteau characterisation — weak convergence is equivalent to $\mathrm{liminf}{\mu }_n(G)\ge \mu (G)$ on open sets, to $\mathrm{limsup}{\mu }_n(F)\le \mu (F)$ on closed sets, and to ${\mu }_n(B)\to \mu (B)$ on $\mu$-continuity sets — is what makes the discontinuity-set hypothesis the precise one ^{[van der Vaart 1998]}.

Theorem 4 (uniform law of large numbers and Glivenko-Cantelli, previewed). For i.i.d. $(X_n)$ with distribution function $F$, the empirical distribution function $F_n(t)=n^{-1}{\sum }_{k=1}^n{1}_{\{X_k\le t\}}$ satisfies ${\sup }_t|F_n(t)-F(t)|\stackrel{a.s.}{\to }0$. More generally, for a class $\mathcal{F}$ of integrable functions, ${\sup }_{f\in \mathcal{F}}|n^{-1}{\sum }_{k}f(X_k)-\mathbb{E}f(X_1)|\stackrel{a.s.}{\to }0$ when $\mathcal{F}$ is Glivenko-Cantelli; finite bracketing numbers in $L^1(P)$ suffice. The pointwise law of large numbers at each $t$ is upgraded to uniformity by monotonicity of $F$ and a finite-grid argument; the abstract version, with bracketing and entropy conditions, is the engine of M-estimator consistency. The full theory is developed in 45.05.02 ^{[Glivenko 1933]}.

Theorem 5 (Wald's consistency theorem for maximum likelihood). Under identifiability, a dominating measure, upper-semicontinuity of $\theta \mapsto \log p_{\theta }(x)$, and an integrable envelope controlling ${\sup }_{\theta }\log p_{\theta }$, the maximum-likelihood estimator is strongly consistent: ${\hat{\theta }}_n\stackrel{a.s.}{\to }{\theta }_0$. Wald's argument is the argmax route — the Kullback-Leibler divergence $\theta \mapsto {\mathbb{E}}_{{\theta }_0}\log (p_{{\theta }_0}/p_{\theta })$ has a well-separated minimum at ${\theta }_0$, and a one-sided uniform law of large numbers over shrinking-and-far neighbourhoods, via a compactness covering, forces the maximiser into every neighbourhood of ${\theta }_0$. The full M-estimator development, including the modern empirical-process treatment, is 45.04.04 ^{[Wald 1949]}.

Synthesis. The foundational reason large-sample statistics is computable is that the modes of convergence form a strict lattice and the stochastic-order symbols obey an algebra: a remainder certified as $o_{\mathbb{P}}$ of the leading term may be discarded, a nuisance factor converging in probability to a constant may be frozen by Slutsky's lemma, and a continuous transformation may be passed through the limit. This is exactly the calculus that converts the raw limit theorems of the probability spine — the law of large numbers 37.02.02 and the central limit theorem through the Lévy continuity theorem 37.03.01 — into statements about estimators. The central insight is that tightness, encoded as $O_{\mathbb{P}}(1)$, is dual to relative compactness in the weak topology through Prohorov's theorem, so every subsequence of a tight sequence has a distributionally convergent further subsequence; this is the device that powers argmax consistency and the subsequence criterion alike. Putting these together, consistency of plug-in estimators generalises the law of large numbers through the continuous-mapping theorem, consistency of M-estimators generalises it further through a uniform law of large numbers and a well-separated maximum, and the bridge from pointwise to uniform convergence — Glivenko-Cantelli — is the single structural fact that the entire empirical-process and M-estimation programme rests on. The lattice, the order calculus, Slutsky, and tightness are four faces of one apparatus, and asymptotic statistics is the study of what that apparatus delivers when applied to the score, the empirical measure, and the likelihood.

Full proof set Master

Proposition 1 (plug-in consistency via continuous mapping). Let $(X_n)$ be i.i.d. with $\mathbb{E}|X_1|<\infty$ and let $\varphi :\mathbb{R}\to \mathbb{R}$ be continuous at $\mathbb{E}{X}_1$. Then $\varphi ({\bar{X}}_n)\stackrel{a.s.}{\to }\varphi (\mathbb{E}{X}_1)$, so $\varphi ({\bar{X}}_n)$ is a strongly consistent estimator of $\varphi (\mathbb{E}{X}_1)$.

Proof. By the strong law of large numbers 37.02.02, ${\bar{X}}_n\stackrel{a.s.}{\to }\mathbb{E}{X}_1$. On the full-measure event where ${\bar{X}}_n\to \mathbb{E}{X}_1$, continuity of $\varphi$ at $\mathbb{E}{X}_1$ gives $\varphi ({\bar{X}}_n)\to \varphi (\mathbb{E}{X}_1)$. The intersection with the (full-measure) convergence event is full-measure, so $\varphi ({\bar{X}}_n)\stackrel{a.s.}{\to }\varphi (\mathbb{E}{X}_1)$. This is the continuous-mapping theorem in its almost-sure form, and it is the standard route to consistency of any smooth functional of sample moments. $\square$

Proposition 2 ($L^2$-consistency implies weak consistency, with the converse failing). If $\mathbb{E}(T_n-\psi {)}^2\to 0$ then $T_n\stackrel{\mathbb{P}}{\to }\psi$. The converse fails.

Proof. Markov's inequality gives $\mathbb{P}(|T_n-\psi |>\epsilon )\le {\epsilon }^{-2}\mathbb{E}(T_n-\psi {)}^2\to 0$, which is weak consistency. The mean-squared error decomposes as $\mathbb{E}(T_n-\psi {)}^2=\mathrm{Var}(T_n)+(\mathbb{E}{T}_n-\psi {)}^2$, so $L^2$-consistency is variance-going-to-zero plus bias-going-to-zero, a convenient sufficient condition. For the failure of the converse, take $T_n=n{1}_{A_n}+\psi$ with $\mathbb{P}(A_n)=1/n$ and $A_n$ chosen independent: then $T_n\stackrel{\mathbb{P}}{\to }\psi$ since $\mathbb{P}(|T_n-\psi |>\epsilon )=1/n\to 0$, while $\mathbb{E}(T_n-\psi {)}^2=n^2\cdot (1/n)=n\to \infty$. So weak consistency does not control the second moment; the rare large excursion that probability ignores dominates the $L^2$ norm. $\square$

Proposition 3 (Slutsky combination for the studentised mean). Let $(X_n)$ be i.i.d. with mean $\mu$ and variance ${\sigma }^2\in (0,\infty )$. Then $\sqrt{n}({\bar{X}}_n-\mu )/S_n\stackrel{d}{\to }N(0,1)$, where $S_n^2=n^{-1}{\sum }_k(X_k-{\bar{X}}_n{)}^2$.

Proof. By the central limit theorem 37.03.01, $\sqrt{n}({\bar{X}}_n-\mu )\stackrel{d}{\to }N(0,{\sigma }^2)$. By the law of large numbers and the continuous-mapping theorem (Exercise 4), $S_n^2\stackrel{\mathbb{P}}{\to }{\sigma }^2$, hence $S_n\stackrel{\mathbb{P}}{\to }\sigma$ by continuity of the square root at ${\sigma }^2>0$. Slutsky's lemma applied to $X_n^{\prime }=\sqrt{n}({\bar{X}}_n-\mu )\stackrel{d}{\to }N(0,{\sigma }^2)$ and $Y_n=S_n\stackrel{\mathbb{P}}{\to }\sigma >0$ gives $X_n^{\prime }/Y_n\stackrel{d}{\to }N(0,{\sigma }^2)/\sigma =N(0,1)$, the named limit. The argument uses every component of the Key theorem: a distributional limit from the central limit theorem, a consistent scale estimate from the law of large numbers and continuous mapping, and Slutsky to divide. $\square$

Proposition 4 (the order calculus reduces a stochastic expansion). Suppose $T_n=\theta +n^{-1/2}{Z}_n+R_n$ with $Z_n\stackrel{d}{\to }N(0,{\sigma }^2)$ and $R_n=o_{\mathbb{P}}(n^{-1/2})$. Then $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2)$.

Proof. Multiply through by $\sqrt{n}$: $\sqrt{n}(T_n-\theta )=Z_n+\sqrt{n}{R}_n$. The hypothesis $R_n=o_{\mathbb{P}}(n^{-1/2})$ means $\sqrt{n}{R}_n\stackrel{\mathbb{P}}{\to }0$, i.e. $\sqrt{n}{R}_n=o_{\mathbb{P}}(1)$. By the lattice, $\sqrt{n}{R}_n\stackrel{d}{\to }0$, and since the limit is constant, Slutsky's lemma applied to $Z_n\stackrel{d}{\to }N(0,{\sigma }^2)$ and $\sqrt{n}{R}_n\stackrel{\mathbb{P}}{\to }0$ gives $Z_n+\sqrt{n}{R}_n\stackrel{d}{\to }N(0,{\sigma }^2)$. This is the canonical pattern by which the delta method and M-estimator asymptotics conclude: isolate a leading term with a known limit law and absorb the remainder as $o_{\mathbb{P}}(1)$ after rescaling. $\square$

Proposition 5 (pointwise to uniform: the Glivenko-Cantelli core). For i.i.d. $(X_n)$ with distribution function $F$, ${\sup }_t|F_n(t)-F(t)|\stackrel{a.s.}{\to }0$.

Proof. Fix a grid $-\infty =t_0<t_1<\cdots <t_m=+\infty$ with $F(t_j-)-F(t_{j-1})\le 1/m$ at the continuity points and accounting for jumps at the rest (possible since $F$ is monotone with at most countably many jumps). By the strong law of large numbers 37.02.02 applied to the bounded indicators, $F_n(t_j)\stackrel{a.s.}{\to }F(t_j)$ and $F_n(t_j-)\stackrel{a.s.}{\to }F(t_j-)$ for each of the finitely many grid points, off a single null event. For $t\in [t_{j-1},t_j)$, monotonicity gives

$$F_n(t)-F(t)\le F_n(t_j-)-F(t_{j-1})\le (F_n(t_j-)-F(t_j-))+1/m,$$

and symmetrically $F_n(t)-F(t)\ge (F_n(t_{j-1})-F(t_{j-1}))-1/m$. Taking the supremum over $t$ and then the limit in $n$, ${\mathrm{limsup}}_n{\sup }_t|F_n(t)-F(t)|\le 1/m$ almost surely. As $m$ is arbitrary the limsup is $0$, giving uniform almost-sure convergence. $\square$

Connections Master

The strong law of large numbers 37.02.02 is the ground floor: it is the limit theorem behind every plug-in consistency proof in this unit, supplying the almost-sure convergence of sample averages that the continuous-mapping theorem then transports through smooth functionals, and the truncation and Borel-Cantelli machinery of that unit reappears here in the subsequence criterion (Exercise 7) and in the Glivenko-Cantelli core (Proposition 5).

Characteristic functions and the Lévy continuity theorem 37.03.01 supply convergence in distribution itself: the portmanteau theorem used throughout this unit is the measure-theoretic face of Lévy continuity, the central limit theorem that feeds every studentisation argument is proved there, and the tightness step in the converse Lévy theorem is the same tightness that Prohorov's theorem packages as $O_{\mathbb{P}}(1)$ here.

The asymptotic normality of M-estimators and the delta method 45.04.02 consume this unit directly: a score expansion is rescaled, its remainder is discarded as $o_{\mathbb{P}}(1)$ by the order calculus, and Slutsky's lemma multiplies the leading term by a consistent estimate of the information matrix, exactly the pattern of Proposition 4. The consistency half proved here is the prerequisite that licenses the local expansion there.

Consistency of M-estimators and maximum likelihood 45.04.04 is the full development of the Wald argmax route previewed in Exercise 8 and Theorem 5: the well-separated-maximum condition and the uniform law of large numbers are stated here and proved in generality there, with the empirical-process tools that certify the uniform law.

Empirical processes and Glivenko-Cantelli classes 45.05.02 is the home of the uniform law of large numbers previewed in Theorem 4 and proved in the classical case in Proposition 5: bracketing numbers, entropy, and the abstract Glivenko-Cantelli theorem generalise the empirical-distribution-function result to function classes, supplying the uniform convergence that argmax consistency requires.

Historical & philosophical context Master

The modern modes of stochastic convergence were sorted out across the first third of the twentieth century. Evgeny Slutsky's 1925 Metron paper ^{[Slutsky 1925]} on stochastic asymptotes and limits introduced the convergence-in-probability calculus and the combination lemma that now bears his name, separating the algebraic behaviour of stochastic limits from that of ordinary limits. Henry Mann and Abraham Wald's 1943 Annals of Mathematical Statistics paper ^{[Mann-Wald 1943]} formalised the $o_{\mathbb{P}}$ and $O_{\mathbb{P}}$ notation and proved the algebraic rules that reduce stochastic Taylor expansions to bookkeeping, importing the Landau symbols into probability with their full calculus.

The consistency question for maximum likelihood was settled by Abraham Wald's 1949 Annals of Mathematical Statistics note ^{[Wald 1949]}, which gave the argmax route — identify a population objective with a well-separated optimum, establish a one-sided uniform law of large numbers by a compactness covering, and conclude that the sample maximiser is forced into every neighbourhood of the truth. Wald's hypotheses (identifiability, an integrable envelope, upper-semicontinuity) remain the template for M-estimator consistency. The uniform law itself traces to Valery Glivenko and Francesco Paolo Cantelli, whose 1933 papers in the Giornale dell'Istituto Italiano degli Attuari ^{[Glivenko 1933]} proved that the empirical distribution function converges to the true distribution function uniformly with probability one, the first uniform law of large numbers and the seed of the empirical-process theory that organises the entire chapter.

The conceptual content is that statistics requires a richer convergence vocabulary than analysis: an estimator is a random sequence, and the distinct demands one can place on it — rare large misses, almost-sure settling, mean-square closeness, convergence of the whole distribution — are genuinely different and genuinely needed, with consistency the weakest demand that still makes data worth collecting. The asymptotic programme codified by Wald and later by Le Cam and van der Vaart reads every estimator through this lattice, and the order calculus of Mann and Wald is the algebra that makes the reading mechanical.

Bibliography Master

@article{Slutsky1925,
  author  = {Slutsky, Evgeny E.},
  title   = {\"Uber stochastische Asymptoten und Grenzwerte},
  journal = {Metron},
  volume  = {5},
  number  = {3},
  year    = {1925},
  pages   = {3--89}
}

@article{MannWald1943,
  author  = {Mann, Henry B. and Wald, Abraham},
  title   = {On stochastic limit and order relationships},
  journal = {Annals of Mathematical Statistics},
  volume  = {14},
  number  = {3},
  year    = {1943},
  pages   = {217--226}
}

@article{Wald1949,
  author  = {Wald, Abraham},
  title   = {Note on the consistency of the maximum likelihood estimate},
  journal = {Annals of Mathematical Statistics},
  volume  = {20},
  number  = {4},
  year    = {1949},
  pages   = {595--601}
}

@article{Glivenko1933,
  author  = {Glivenko, Valery I.},
  title   = {Sulla determinazione empirica delle leggi di probabilit\`a},
  journal = {Giornale dell'Istituto Italiano degli Attuari},
  volume  = {4},
  year    = {1933},
  pages   = {92--99}
}

@article{Cantelli1933,
  author  = {Cantelli, Francesco Paolo},
  title   = {Sulla determinazione empirica delle leggi di probabilit\`a},
  journal = {Giornale dell'Istituto Italiano degli Attuari},
  volume  = {4},
  year    = {1933},
  pages   = {221--424}
}

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{Ferguson1996,
  author    = {Ferguson, Thomas S.},
  title     = {A Course in Large Sample Theory},
  publisher = {Chapman \& Hall},
  address   = {London},
  year      = {1996}
}

@book{Pollard1984,
  author    = {Pollard, David},
  title     = {Convergence of Stochastic Processes},
  publisher = {Springer-Verlag},
  address   = {New York},
  year      = {1984}
}