Asymptotic Normality and Efficiency of the Maximum-Likelihood Estimator

Intuition Beginner

You have a recipe for estimating an unknown number from data: turn a dial until your data look least surprising, and read off the setting. That recipe is maximum likelihood, and the setting it lands on is the maximum-likelihood estimate. The question of this unit is not what the recipe is, but how good it is when you have a lot of data — and the answer is about as good as anything can be.

With more and more data, two pleasant things happen. First, the estimate stops missing: it homes in on the true value, so it is consistent. Second, the leftover error settles into a familiar bell-shaped wobble centered on the truth, and the size of that wobble is set by exactly one number — the Fisher information, the sharpness of the likelihood peak from the estimation story. A sharp peak pins the estimate down tightly; a flat peak leaves it loose. The wobble shrinks like one over the information.

The deeper claim is that no fair-minded estimator can do better in the long run. The floor on precision — the Cramér-Rao bound — is met, in the limit, by maximum likelihood. So the recipe is not just one option among many; it is the asymptotically best one. This is why it is the default workhorse of statistics.

The one-sentence takeaway: with enough data the maximum-likelihood estimate is centered on the truth, wobbles like a bell curve whose spread is one over the Fisher information, and that spread is the smallest any fair estimator can reach.

Visual Beginner

Picture the spread of the maximum-likelihood estimate for three sample sizes, drawn as three bell-shaped humps over the candidate values, all centered on the same true value. The small-sample hump is wide; the medium-sample hump is narrower; the large-sample hump is a thin spike. Each is bell-shaped, and each is centered on the truth — that is the normality and the consistency together. The widths shrink in a fixed pattern: quadruple the data and the width halves.

sample size	shape of the estimate's spread	width
small	roughly bell-shaped, wide	large
medium	bell-shaped, narrower	medium
large	a sharp bell-shaped spike on the truth	one over the square root of the information times the count

The takeaway: the maximum-likelihood estimate is always centered on the truth, its spread is bell-shaped, and the spread shrinks toward the smallest value any fair estimator can reach — one over the Fisher information in the data.

Worked example Beginner

You watch a counting process — defects on a production line per shift — that follows a single-rate pattern with unknown average rate. You collect $n=100$ shifts and want to know the spread of the maximum-likelihood estimate of the rate when the true rate is $5$.

Step 1. The maximum-likelihood estimate. For this counting pattern the setting that makes the data least surprising is simply the average of the counts. So the estimate is the sample average — the same number you would have guessed anyway.

Step 2. The information in one observation. For this counting pattern the Fisher information in a single shift is one divided by the rate. At a true rate of $5$, one observation carries information $1/5=0.2$.

Step 3. Information adds up. One hundred independent shifts carry one hundred times the information: total information is $100\times 0.2=20$.

Step 4. The spread of the estimate. The large-sample theory says the variance of the maximum-likelihood estimate is one divided by the total information: $1/20=0.05$. The typical error is the square root, $\sqrt{0.05}\approx 0.224$.

Step 5. Check against the direct calculation. The average of $100$ counts has variance equal to the rate over the count, $5/100=0.05$ — the same number. The two routes agree because the average is, for this pattern, exactly the maximum-likelihood estimate and it already sits on the precision floor.

What this tells us: the large-sample recipe — take one over the total Fisher information — predicts the spread of the maximum-likelihood estimate, and that spread is the smallest any fair estimate of the rate can reach from this much data.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_1,\dots ,X_n$ be i.i.d. from a regular parametric family $\{f(\cdot \mid \theta ):\theta \in \Theta \}$, $\Theta \subseteq {\mathbb{R}}^k$ open, satisfying the regularity conditions of 45.01.05: the support of $f$ does not depend on $\theta$, $\theta \mapsto \log f(x\mid \theta )$ is smooth, and differentiation in $\theta$ passes through the expectation integral. The log-likelihood, score, and Fisher information are

$${\ell }_n(\theta )=\sum _{i=1}^n\log f(X_i\mid \theta ),s(\theta ;x)={\nabla }_{\theta }\log f(x\mid \theta ),I(\theta )={\mathrm{Cov}}_{\theta }(s(\theta ;X))=-{\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2\log f],$$

as in 45.01.05. Convergence in distribution $\stackrel{d}{\to }$, in probability $\stackrel{\mathbb{P}}{\to }$, and the $o_{\mathbb{P}}/O_{\mathbb{P}}$ calculus are as in 45.04.01.

Definition (maximum-likelihood estimator). A maximum-likelihood estimator (MLE) ${\hat{\theta }}_n$ is any (measurable) maximiser of the likelihood, ${\hat{\theta }}_n\in \arg {\max }_{\theta \in \Theta }{\ell }_n(\theta )$. When the maximum is interior and ${\ell }_n$ is differentiable, ${\hat{\theta }}_n$ solves the likelihood equations (the score equation) $s_n({\hat{\theta }}_n)={\sum }_{i=1}^{n}s({\hat{\theta }}_n;X_i)=0$.

Definition (asymptotic normality at rate $\sqrt{n}$). An estimator sequence is asymptotically normal with asymptotic covariance $V$ if $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,V)$ under the data law at the true parameter ${\theta }_0$. The matrix $V$ is the asymptotic variance.

Definition (asymptotic efficiency). Within the class of regular estimators — those whose limit distribution is suitably stable under local perturbations ${\theta }_0+h/\sqrt{n}$ of the parameter — an asymptotically normal estimator is asymptotically efficient at ${\theta }_0$ if its asymptotic covariance equals $I({\theta }_0{)}^{-1}$, the inverse Fisher information. By the asymptotic Cramér-Rao / convolution theorem this is the smallest attainable regular asymptotic covariance in the Loewner order; the asymptotic relative efficiency of two such estimators is the ratio of their asymptotic variances.

Definition (Wald confidence interval). For a scalar $\theta$, the Wald interval of asymptotic level $1-\alpha$ is

$${\hat{\theta }}_n\pm z_{1-\alpha /2}(n{\hat{I}}_n{)}^{-1/2},$$

where ${\hat{I}}_n$ is a consistent estimator of $I({\theta }_0)$ — the plug-in information $I({\hat{\theta }}_n)$ or the observed information $-n^{-1}{\ell }_n^{\prime \prime }({\hat{\theta }}_n)$ — and $z_{1-\alpha /2}$ is the standard-normal quantile. In $k$ dimensions the analogous ellipsoid uses ${\hat{\theta }}_n$ and $(n^{-1}{\hat{I}}_n{)}^{-1}$ with a chi-squared cutoff.

Definition (one-step estimator). Given a preliminary $\sqrt{n}$-consistent estimator ${\tilde{\theta }}_n$ (so $\sqrt{n}({\tilde{\theta }}_n-{\theta }_0)=O_{\mathbb{P}}(1)$), the one-step (Newton-Raphson / Fisher-scoring) estimator is one Newton iterate on the score equation started at ${\tilde{\theta }}_n$:

$${\hat{\theta }}_n^{(1)}={\tilde{\theta }}_n-[\nabla s_n({\tilde{\theta }}_n){]}^{-1}{s}_n({\tilde{\theta }}_n)\text{or}{\hat{\theta }}_n^{(1)}={\tilde{\theta }}_n+[n{\hat{I}}_n{]}^{-1}{s}_n({\tilde{\theta }}_n),$$

the second form replacing the observed information by an estimated Fisher information (the scoring variant).

Counterexamples to common slips Intermediate+

• The asymptotic variance is $I({\theta }_0{)}^{-1}$, not $-1$ over the curvature at the truth without an expectation. The limit uses the expected information $I({\theta }_0)$. In practice one estimates it by the observed information $-n^{-1}{\ell }_n^{\prime \prime }({\hat{\theta }}_n)$, which is consistent for $I({\theta }_0)$ but is a random approximation, not the limit itself.

• Consistency is a hypothesis of the normality proof, not a consequence of it. The Taylor expansion is performed about ${\theta }_0$ and evaluated at ${\hat{\theta }}_n$; without first knowing ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$ (Wald's theorem) the expansion is not licensed. A root of the score equation that is the wrong root need not be consistent.

• The MLE is efficient among regular estimators, not literally among all estimators. Superefficient estimators — Hodges' example below — beat $I({\theta }_0{)}^{-1}$ on a Lebesgue-null set of parameter values, at the price of worse local behaviour nearby. The efficiency claim is sharp only once regularity (or a local-asymptotic-minimax criterion) is imposed.

• Non-regular families break the rate, not just the constant. For the uniform-$(0,\theta )$ family the support moves with $\theta$, the score identity fails, and the MLE $X_{(n)}$ converges at rate $n$ with a non-normal (exponential) limit. The $\sqrt{n}$-normal theorem requires the regularity of 45.01.05.

Key theorem with proof Intermediate+

The signature result is that, under regularity, the maximum-likelihood estimator is consistent and asymptotically normal with asymptotic variance the inverse Fisher information, and that this variance attains the asymptotic Cramér-Rao floor of 45.01.05. The proof is a stochastic Taylor expansion of the score about the truth, controlled by a law of large numbers on the observed information and the central limit theorem on the score, assembled by Slutsky's lemma.

Theorem (asymptotic normality and efficiency of the MLE). Let $X_1,\dots ,X_n$ be i.i.d. from a regular family $\{f(\cdot \mid \theta ):\theta \in \Theta \}$, $\Theta \subseteq {\mathbb{R}}^k$ open, with true parameter ${\theta }_0$ interior to $\Theta$. Assume

(C1) consistency: ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$, for instance via Wald's conditions — identifiability of ${\theta }_0$ and a uniform law of large numbers giving ${\sup }_{\theta }|n^{-1}{\ell }_n(\theta )-{\mathbb{E}}_{{\theta }_0}\log f(X\mid \theta )|\stackrel{\mathbb{P}}{\to }0$ with a well-separated population maximiser;

(C2) twice-continuous differentiability of $\theta \mapsto \log f(x\mid \theta )$ in a neighbourhood of ${\theta }_0$, with each third partial derivative dominated by a fixed integrable function $M(x)$, ${\mathbb{E}}_{{\theta }_0}M(X)<\infty$, on that neighbourhood;

(C3) the Fisher information $I({\theta }_0)=-{\mathbb{E}}_{{\theta }_0}[{\nabla }^2\log f]$ exists, is finite, and is positive definite.

Then

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0{)}^{-1}),$$

and the asymptotic covariance $I({\theta }_0{)}^{-1}$ is the asymptotic Cramér-Rao bound, so ${\hat{\theta }}_n$ is asymptotically efficient.

Proof. By consistency (C1) the event $\{{\hat{\theta }}_n\in U\}$, with $U$ the neighbourhood of (C2), has probability tending to $1$, and on it ${\hat{\theta }}_n$ is an interior maximiser, so the likelihood equation $s_n({\hat{\theta }}_n)=\nabla {\ell }_n({\hat{\theta }}_n)=0$ holds. Apply a first-order Taylor expansion of the (vector) score $\theta \mapsto s_n(\theta )=\nabla {\ell }_n(\theta )$ about ${\theta }_0$, with the integral / mean-value remainder evaluated at intermediate points ${\bar{\theta }}_n$ on the segment from ${\theta }_0$ to ${\hat{\theta }}_n$:

$$0=s_n({\hat{\theta }}_n)=s_n({\theta }_0)+[\nabla s_n({\bar{\theta }}_n)]({\hat{\theta }}_n-{\theta }_0),$$

where $\nabla s_n(\theta )={\nabla }^2{\ell }_n(\theta )$ is the Hessian of the log-likelihood. Rearranging and inserting the rate,

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=[-\frac{1}{n}{\nabla }^2{\ell }_n({\bar{\theta }}_n){]}^{-1}\frac{1}{\sqrt{n}}{s}_n({\theta }_0),$$

valid once the bracketed matrix is invertible, which holds with probability tending to $1$ by the steps below.

The score term obeys a CLT. The summands $s({\theta }_0;X_i)=\nabla \log f(X_i\mid {\theta }_0)$ are i.i.d. with mean zero (the score identity of 45.01.05) and covariance $I({\theta }_0)$. The multivariate central limit theorem 37.03.02 gives

$$\frac{1}{\sqrt{n}}{s}_n({\theta }_0)=\frac{1}{\sqrt{n}}\sum _{i=1}^{n}s({\theta }_0;X_i)\stackrel{d}{\to }N(0,I({\theta }_0)).$$

The observed-information term obeys an LLN. Write $A_n(\theta )=-\frac{1}{n}{\nabla }^2{\ell }_n(\theta )=-\frac{1}{n}{\sum }_i{\nabla }^2\log f(X_i\mid \theta )$. At the truth the law of large numbers gives $A_n({\theta }_0)\stackrel{\mathbb{P}}{\to }-{\mathbb{E}}_{{\theta }_0}[{\nabla }^2\log f]=I({\theta }_0)$ by the information equality of 45.01.05. Evaluated at the random ${\bar{\theta }}_n$ (which lies between ${\theta }_0$ and ${\hat{\theta }}_n$, hence ${\bar{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$ by (C1)), the difference is controlled by (C2): for $\theta \in U$,

$$|A_n(\theta )-A_n({\theta }_0)|\le |\theta -{\theta }_0|\frac{1}{n}\sum _{i=1}^{n}M(X_i),$$

and $\frac{1}{n}{\sum }_{i}M(X_i)\stackrel{\mathbb{P}}{\to }{\mathbb{E}}_{{\theta }_0}M(X)<\infty$ while $|{\bar{\theta }}_n-{\theta }_0|\stackrel{\mathbb{P}}{\to }0$, so $A_n({\bar{\theta }}_n)-A_n({\theta }_0)\stackrel{\mathbb{P}}{\to }0$. Hence $A_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }I({\theta }_0)$.

Assemble by Slutsky. By (C3) $I({\theta }_0)\succ 0$, so the continuous-mapping theorem 45.04.01 gives $A_n({\bar{\theta }}_n{)}^{-1}\stackrel{\mathbb{P}}{\to }I({\theta }_0{)}^{-1}$ (matrix inversion is continuous on the invertible matrices). Slutsky's lemma multiplies the convergent-in-probability matrix factor by the convergent-in-distribution score factor:

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=A_n({\bar{\theta }}_n{)}^{-1}\frac{1}{\sqrt{n}}{s}_n({\theta }_0)\stackrel{d}{\to }I({\theta }_0{)}^{-1}N(0,I({\theta }_0))=N(0,I({\theta }_0{)}^{-1}I({\theta }_0)I({\theta }_0{)}^{-1})=N(0,I({\theta }_0{)}^{-1}),$$

using $I({\theta }_0{)}^{-1}\Sigma (I({\theta }_0{)}^{-1}{)}^{\mathsf{T}}=I({\theta }_0{)}^{-1}$ with $\Sigma =I({\theta }_0)$ and the symmetry of $I({\theta }_0)$. The limiting covariance $I({\theta }_0{)}^{-1}$ is the asymptotic Cramér-Rao floor of 45.01.05, so ${\hat{\theta }}_n$ is asymptotically efficient. $\square$

Bridge. This theorem builds toward every likelihood-based confidence region and test in the chapter, and the same score-expansion mechanism appears again in the asymptotic theory of M- and Z-estimators, where an estimating function $\psi$ replaces the score and the sandwich covariance $J^{-1}\Sigma J^{-1}$ replaces $I({\theta }_0{)}^{-1}$. The foundational reason the MLE is efficient is that the score is the canonical direction in the model — its variance is the Fisher information, the very floor of 45.01.05 — and the estimator inverts exactly that information to convert the score's Gaussian fluctuation into the estimate's. This is exactly the delta-method principle of 45.04.02 applied to the implicitly defined root of the score equation: a smooth map is locally linear and the linearisation error is upgraded to an $o_{\mathbb{P}}$ remainder because $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$ is bounded in probability. Putting these together, asymptotic normality generalises the central limit theorem 37.03.02 from sample averages to implicitly defined estimators, and the bridge is the passage from the finite-sample Cramér-Rao floor to its generic asymptotic attainment: what no estimator can beat at finite $n$ in the exponential family alone, the MLE reaches in the limit for every regular family.

Exercises Intermediate+

Advanced results Master

The scalar theorem extends verbatim to the multiparameter case with the information matrix, the efficiency claim is sharpened by the convolution and local-asymptotic-minimax theorems into a precise optimality statement among regular estimators, superefficiency is delimited to a null set, and the one-step construction shows that efficiency is purchasable from any $\sqrt{n}$-consistent start. The misspecified case replaces $I^{-1}$ by a sandwich, and the functional version is the M-estimator theory of 45.04.04.

Theorem 1 (multiparameter asymptotic normality and the sandwich for Z-estimators). Under (C1)-(C3) in ${\mathbb{R}}^k$, $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0{)}^{-1})$ with $I({\theta }_0)$ the Fisher information matrix. More generally, if ${\hat{\theta }}_n$ solves an estimating equation ${\sum }_i\psi (X_i;{\hat{\theta }}_n)=0$ with ${\mathbb{E}}_{{\theta }_0}\psi (X;{\theta }_0)=0$, the same Taylor-plus-Slutsky argument gives $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,J^{-1}\Sigma J^{-\mathsf{T}})$ where $J=-{\mathbb{E}}_{{\theta }_0}{\nabla }_{\theta }\psi$ and $\Sigma ={\mathrm{Cov}}_{{\theta }_0}\psi$. The MLE is the special case $\psi =s$, where $J=\Sigma =I({\theta }_0)$ and the sandwich collapses to $I({\theta }_0{)}^{-1}$ — the second Bartlett identity is exactly what makes maximum likelihood efficient among Z-estimators ^{[van der Vaart — Asymptotic Statistics]}.

Theorem 2 (the convolution theorem and asymptotic optimality of $I^{-1}$). For a regular estimator sequence ${\hat{\theta }}_n$ in a differentiable-in-quadratic-mean family, any limit law $L$ of $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$ admits a convolution factorisation $L=N(0,I({\theta }_0{)}^{-1})\ast M$ for some distribution $M$. Consequently the asymptotic covariance is $⪰I({\theta }_0{)}^{-1}$ in the Loewner order, with equality precisely when $M={\delta }_0$, i.e. when the estimator is asymptotically equivalent to the efficient one. The MLE achieves $M={\delta }_0$, so among regular estimators it is asymptotically best; the local-asymptotic-minimax theorem gives the same conclusion without a regularity restriction, as a minimax statement over shrinking neighbourhoods ${\theta }_0+h/\sqrt{n}$ ^{[van der Vaart — Asymptotic Statistics]}.

Theorem 3 (superefficiency on a null set; Le Cam). If ${\hat{\theta }}_n$ has asymptotic variance $v(\theta )$ at each $\theta$ and is asymptotically normal, then the set $\{\theta :v(\theta )<I(\theta {)}^{-1}\}$ on which it beats the Fisher bound has Lebesgue measure zero. Hodges' estimator (Exercise 8) realises the extreme case: variance $0$ at one point, variance $I^{-1}$ elsewhere, at the cost of unbounded local risk near the superefficiency point. Superefficiency is thus a measure-theoretic curiosity, not a route to uniformly better estimation, and it is the historical motivation for restricting attention to regular estimators or to the minimax criterion ^{[Le Cam — On some asymptotic properties of maximum likelihood estimates]}.

Theorem 4 (one-step efficiency and the discretised MLE). If ${\tilde{\theta }}_n$ is $\sqrt{n}$-consistent and ${\hat{I}}_n\stackrel{\mathbb{P}}{\to }I({\theta }_0)$, the one-step estimator ${\hat{\theta }}_n^{(1)}={\tilde{\theta }}_n+(n{\hat{I}}_n{)}^{-1}{s}_n({\tilde{\theta }}_n)$ satisfies $\sqrt{n}({\hat{\theta }}_n^{(1)}-{\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0{)}^{-1})$, asymptotically equivalent to the MLE. A technically clean variant uses a discretised preliminary estimator (rounding ${\tilde{\theta }}_n$ to a ${\sqrt{n}}^{-1}$-grid) so that the random expansion point can be treated as deterministic; this removes the need for the global maximisation that defines the MLE while retaining its efficiency, and it is the standard device for proving asymptotic efficiency of estimators that are awkward to maximise directly ^{[van der Vaart — Asymptotic Statistics]}.

Theorem 5 (misspecification: the Huber sandwich and the KL-projection). If the data law $P$ is not in the model, the MLE converges to the parameter ${\theta }_{\ast }$ minimising the Kullback-Leibler divergence $\mathrm{KL}(P\Vert f(\cdot \mid \theta ))$ (the pseudo-true value), and $\sqrt{n}({\hat{\theta }}_n-{\theta }_{\ast })\stackrel{d}{\to }N(0,A^{-1}B{A}^{-1})$ with $A=-{\mathbb{E}}_P{\nabla }^2\log f(\cdot \mid {\theta }_{\ast })$ and $B={\mathrm{Cov}}_P(\nabla \log f(\cdot \mid {\theta }_{\ast }))$. Under correct specification the information equality forces $A=B=I({\theta }_0)$ and the sandwich collapses to $I({\theta }_0{)}^{-1}$; the gap between the observed and expected information is therefore a diagnostic for misspecification ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Synthesis. The foundational reason maximum likelihood is the asymptotically optimal estimator is that the score is the canonical tangent direction of the model, so the central limit theorem 37.03.02 acting on the score produces a $N(0,I({\theta }_0))$ fluctuation, and inverting the observed information — which the law of large numbers pins to $I({\theta }_0)$ — converts it into the estimate's $N(0,I({\theta }_0{)}^{-1})$ law. The central insight is that this asymptotic covariance is exactly the inverse Fisher information, the finite-sample Cramér-Rao floor of 45.01.05, so what is attainable only by exponential-family sufficient statistics at finite $n$ becomes generically attainable in the limit: efficiency relaxes from a rigid algebraic identity to a universal asymptotic property. The convolution theorem makes this optimality precise — every regular limit law is $N(0,I^{-1})$ convolved with extra noise, so $I^{-1}$ is the irreducible variance and the MLE alone removes the surplus; this is dual to the Cramér-Rao computation, the inequality of the finite-sample world reappearing as a convolution factorisation in the asymptotic one.

Putting these together, superefficiency is the apparent exception that proves the rule — it lives on a Lebesgue-null set at the price of catastrophic local risk, which is exactly why the right notion of optimality is local-asymptotic-minimax rather than pointwise variance. The bridge from the MLE to the wider world is the Z-estimator sandwich $J^{-1}\Sigma J^{-1}$, which generalises $I^{-1}$ to misspecified models and estimating equations, collapsing to $I^{-1}$ precisely when the Bartlett identities $J=\Sigma =I$ hold; the one-step construction then shows efficiency is not tied to global maximisation but is a single Newton correction away from any $\sqrt{n}$-consistent guess.

Full proof set Master

The scalar and multiparameter asymptotic-normality statements are proved in the Key theorem (the multiparameter case is the same argument with vectors and matrices); the one-step and superefficiency claims are Exercises 7-8. The remaining Master claims are recorded here.

Proposition 1 (Wald consistency of the MLE). Suppose ${\theta }_0$ is identifiable, $\Theta$ is compact (or the search is restricted to a compact set containing ${\theta }_0$), $\theta \mapsto \log f(x\mid \theta )$ is upper semicontinuous, and a uniform law of large numbers holds: ${\sup }_{\theta \in \Theta }|n^{-1}{\ell }_n(\theta )-K(\theta )|\stackrel{\mathbb{P}}{\to }0$, where $K(\theta )={\mathbb{E}}_{{\theta }_0}\log f(X\mid \theta )$. Then any maximiser ${\hat{\theta }}_n$ of ${\ell }_n$ satisfies ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$.

Proof. By the information inequality $K(\theta )=K({\theta }_0)-\mathrm{KL}(f(\cdot \mid {\theta }_0)\Vert f(\cdot \mid \theta ))\le K({\theta }_0)$, with equality iff $f(\cdot \mid \theta )=f(\cdot \mid {\theta }_0)$, i.e. (by identifiability) iff $\theta ={\theta }_0$; thus ${\theta }_0$ is the unique, well-separated maximiser of the continuous population criterion $K$. Fix $\epsilon >0$ and let $\delta =K({\theta }_0)-{\sup }_{|\theta -{\theta }_0|\ge \epsilon }K(\theta )>0$ by well-separation. On the event $\{{\sup }_{\theta }|n^{-1}{\ell }_n-K|<\delta /2\}$, which has probability tending to $1$, the empirical criterion at ${\hat{\theta }}_n$ exceeds that at ${\theta }_0$ by definition of ${\hat{\theta }}_n$, so $K({\hat{\theta }}_n)>n^{-1}{\ell }_n({\hat{\theta }}_n)-\delta /2\ge n^{-1}{\ell }_n({\theta }_0)-\delta /2>K({\theta }_0)-\delta$, which forces $|{\hat{\theta }}_n-{\theta }_0|<\epsilon$. Hence $\mathbb{P}(|{\hat{\theta }}_n-{\theta }_0|\ge \epsilon )\to 0$. $\square$

Proposition 2 (the score is mean-zero with covariance $I$, and the leading term is asymptotically normal). Under regularity, ${\mathbb{E}}_{{\theta }_0}[s({\theta }_0;X)]=0$, ${\mathrm{Cov}}_{{\theta }_0}(s({\theta }_0;X))=I({\theta }_0)$, and $n^{-1/2}{s}_n({\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0))$.

Proof. The score identity ${\mathbb{E}}_{{\theta }_0}[s({\theta }_0;X)]=\int ({\nabla }_{\theta }\log f)fdx={\nabla }_{\theta }\int fdx={\nabla }_{\theta }1=0$ uses differentiation under the integral 45.01.05, and then ${\mathrm{Cov}}_{{\theta }_0}(s)={\mathbb{E}}_{{\theta }_0}[s{s}^{\mathsf{T}}]=I({\theta }_0)$ by definition of the information matrix. The summands $s({\theta }_0;X_i)$ are i.i.d., mean zero, with finite covariance $I({\theta }_0)$, so the multivariate Lindeberg-Lévy central limit theorem 37.03.02 gives $n^{-1/2}{\sum }_{i}s({\theta }_0;X_i)\stackrel{d}{\to }N(0,I({\theta }_0))$. $\square$

Proposition 3 (observed information converges to $I$ along a consistent sequence). Under (C1)-(C3), for any ${\theta }_n^{\ast }\stackrel{\mathbb{P}}{\to }{\theta }_0$ with ${\theta }_n^{\ast }\in U$ eventually, $-n^{-1}{\nabla }^2{\ell }_n({\theta }_n^{\ast })\stackrel{\mathbb{P}}{\to }I({\theta }_0)$.

Proof. At ${\theta }_0$, the LLN gives $-n^{-1}{\nabla }^2{\ell }_n({\theta }_0)=-n^{-1}{\sum }_i{\nabla }^2\log f(X_i\mid {\theta }_0)\stackrel{\mathbb{P}}{\to }-{\mathbb{E}}_{{\theta }_0}[{\nabla }^2\log f]=I({\theta }_0)$ (information equality, 45.01.05). The third-derivative domination (C2) gives, on $U$, the Lipschitz bound $\Vert -n^{-1}{\nabla }^2{\ell }_n({\theta }_n^{\ast })+n^{-1}{\nabla }^2{\ell }_n({\theta }_0)\Vert \le |{\theta }_n^{\ast }-{\theta }_0|n^{-1}{\sum }_{i}M(X_i)$. Since $n^{-1}{\sum }_{i}M(X_i)\stackrel{\mathbb{P}}{\to }{\mathbb{E}}_{{\theta }_0}M(X)<\infty$ and $|{\theta }_n^{\ast }-{\theta }_0|\stackrel{\mathbb{P}}{\to }0$, the right side is $o_{\mathbb{P}}(1)$, and adding the two limits gives $-n^{-1}{\nabla }^2{\ell }_n({\theta }_n^{\ast })\stackrel{\mathbb{P}}{\to }I({\theta }_0)$. $\square$

Proposition 4 (the asymptotic-normality assembly). Under (C1)-(C3), $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0{)}^{-1})$.

Proof. On the probability-tending-to-one event that ${\hat{\theta }}_n$ is an interior root of the score, the mean-value expansion gives $0=s_n({\theta }_0)+{\nabla }^2{\ell }_n({\bar{\theta }}_n)({\hat{\theta }}_n-{\theta }_0)$ with ${\bar{\theta }}_n$ between ${\theta }_0$ and ${\hat{\theta }}_n$, so ${\bar{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$. Thus $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=[-n^{-1}{\nabla }^2{\ell }_n({\bar{\theta }}_n){]}^{-1}{n}^{-1/2}{s}_n({\theta }_0)$. By Proposition 3 the matrix factor converges in probability to $I({\theta }_0)\succ 0$, hence its inverse to $I({\theta }_0{)}^{-1}$ by continuous mapping; by Proposition 2 the vector factor converges in distribution to $N(0,I({\theta }_0))$. Slutsky's lemma combines them: $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }I({\theta }_0{)}^{-1}N(0,I({\theta }_0))=N(0,I({\theta }_0{)}^{-1})$, the last step using $I^{-1}I{I}^{-\mathsf{T}}=I^{-1}$ for symmetric $I$. $\square$

Proposition 5 (efficiency: the MLE attains the asymptotic Cramér-Rao bound). The asymptotic covariance $I({\theta }_0{)}^{-1}$ of the MLE equals the asymptotic Cramér-Rao bound, and no regular estimator has a strictly smaller asymptotic covariance in the Loewner order.

Proof. For an unbiased estimator the finite-sample matrix bound of 45.01.05 reads ${\mathrm{Cov}}_{{\theta }_0}(W)⪰I_n({\theta }_0{)}^{-1}=(nI({\theta }_0){)}^{-1}$, so the normalised covariance satisfies ${\mathrm{Cov}}_{{\theta }_0}(\sqrt{n}W)⪰I({\theta }_0{)}^{-1}$; this lower bound is the asymptotic Cramér-Rao floor, and the MLE meets it by Proposition 4. The unbiasedness restriction is removed asymptotically by the convolution theorem (Theorem 2): any regular limit law factors as $N(0,I({\theta }_0{)}^{-1})\ast M$, so its covariance is $I({\theta }_0{)}^{-1}+\mathrm{Cov}(M)⪰I({\theta }_0{)}^{-1}$, with equality iff $M={\delta }_0$. The MLE realises $M={\delta }_0$, hence is asymptotically efficient and no regular competitor improves on it. $\square$

Connections Master

The Fisher information and Cramér-Rao bound of 45.01.05 supply both the asymptotic variance and the optimality benchmark: the limit covariance $I({\theta }_0{)}^{-1}$ is the inverse of the very information matrix defined there, and the efficiency claim is that the MLE meets, in the limit, the finite-sample floor that unit proves is reachable exactly only in exponential families. The information equality $I=-\mathbb{E}[{\nabla }^2\log f]$ from that unit is the bridge between the observed information used in practice and the expected information appearing in the limit theorem.

Consistency and the modes of stochastic convergence 45.04.01 furnish the entire machinery of the proof — the $O_{\mathbb{P}}/o_{\mathbb{P}}$ calculus that upgrades the deterministic Taylor remainder to a stochastic one, the continuous-mapping theorem that inverts the observed-information matrix, and Slutsky's lemma that fuses the convergent-in-probability matrix factor with the convergent-in-distribution score factor. Wald's consistency argument, previewed there, is the (C1) hypothesis made precise in Proposition 1.

The delta method 45.04.02 is consumed twice: the MLE limit is itself a delta-method-style linearisation of the implicitly defined root of the score equation, and the asymptotic distribution of any smooth function $g({\hat{\theta }}_n)$ of the estimate is the first-order delta method applied to that limit, giving asymptotic variance $\nabla g({\theta }_0)I({\theta }_0{)}^{-1}\nabla g({\theta }_0{)}^{\mathsf{T}}$ and confirming that maximum likelihood is invariant under reparametrisation.

Consistency and asymptotic normality of M-estimators 45.04.04 is the direct generalisation: replacing the score by an arbitrary estimating function $\psi$ turns the inverse-information variance into the sandwich $J^{-1}\Sigma J^{-1}$, with the MLE the optimal special case where the Bartlett identities force $J=\Sigma =I$; the misspecified Huber sandwich (Theorem 5) is the same formula away from the model.

The chi-squared limits of the Wald, score, and likelihood-ratio tests 45.06.01 are built directly on this theorem: the Wald statistic is a quadratic form in the asymptotically normal $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$, the score statistic a quadratic form in $n^{-1/2}{s}_n({\theta }_0)$, and the likelihood-ratio statistic their common limit via a second-order expansion, all yielding ${\chi }^2$ null laws whose degrees of freedom count the restrictions, exactly the second-order delta-method mechanism of 45.04.02.

Historical & philosophical context Master

Maximum likelihood as a general method of estimation, and the claim that it is asymptotically the most precise, are due to Ronald A. Fisher, who introduced the likelihood and the information measure in his 1922 foundations paper and argued through the 1920s that the maximum-likelihood estimate is consistent, asymptotically normal, and asymptotically efficient — extracting in the limit all the information a sample carries ^{[Fisher 1922]}. Fisher's arguments were heuristic by later standards; he asserted the $1/I$ asymptotic variance and the optimality without the regularity bookkeeping that the modern theorem requires.

The first rigorous treatment of the asymptotic normality and efficiency of the MLE under explicit regularity hypotheses — twice differentiability, third-derivative domination, a positive-definite information matrix — was given by Harald Cramér in his 1946 Mathematical Methods of Statistics ^{[Cramér 1946]}, and these are still called the Cramér conditions. Abraham Wald supplied in 1949 the clean consistency argument via identifiability and a uniform law of large numbers on the log-likelihood ^{[Wald 1949]}, which Proposition 1 follows. The efficiency claim was then sharpened and partly overturned by Lucien Le Cam, whose 1953 paper exhibited superefficiency — estimators beating the Fisher bound on a measure-zero set — and showed the phenomenon is confined to such null sets ^{[Le Cam 1953]}; Le Cam's subsequent local-asymptotic-normality and convolution-theorem programme, developed with Jaroslav Hájek, recast asymptotic efficiency as a minimax and convolution statement that the pointwise-variance definition could not support, the framework in which the optimality of maximum likelihood is now stated.

Bibliography Master

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{LehmannCasella1998,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}

@book{CasellaBerger2002,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2},
  publisher = {Duxbury / Thomson Learning},
  year      = {2002}
}

@book{Cramer1946,
  author    = {Cram\'er, Harald},
  title     = {Mathematical Methods of Statistics},
  publisher = {Princeton University Press},
  address   = {Princeton},
  year      = {1946}
}

@article{Fisher1922,
  author  = {Fisher, Ronald A.},
  title   = {On the mathematical foundations of theoretical statistics},
  journal = {Philosophical Transactions of the Royal Society of London, Series A},
  volume  = {222},
  year    = {1922},
  pages   = {309--368}
}

@article{Wald1949,
  author  = {Wald, Abraham},
  title   = {Note on the consistency of the maximum likelihood estimate},
  journal = {Annals of Mathematical Statistics},
  volume  = {20},
  number  = {4},
  year    = {1949},
  pages   = {595--601}
}

@article{LeCam1953,
  author  = {Le Cam, Lucien},
  title   = {On some asymptotic properties of maximum likelihood estimates and related Bayes' estimates},
  journal = {University of California Publications in Statistics},
  volume  = {1},
  year    = {1953},
  pages   = {277--330}
}

@article{Hajek1970,
  author  = {H\'ajek, Jaroslav},
  title   = {A characterization of limiting distributions of regular estimates},
  journal = {Zeitschrift f\"ur Wahrscheinlichkeitstheorie und Verwandte Gebiete},
  volume  = {14},
  year    = {1970},
  pages   = {323--330}
}