Asymptotics of the Likelihood-Ratio, Wald, and Score Tests: The Chi-Squared Trinity

Intuition Beginner

You have three ways to score how badly a claim about your model fits the data — the height drop, the horizontal gap, and the slope, the three readings of one log-likelihood hill from the companion unit. Each gives a number that is small when the claim looks right and large when it looks wrong. The practical question is: how large is large? You need a yardstick, a reference set of "scores you would expect to see by chance if the claim were actually true."

The remarkable fact is that all three scores share the same yardstick once you have a lot of data, and that yardstick has a name: the chi-squared distribution. Picture the leftover error of a well-fit model as a small random wobble shaped like a bell curve. A chi-squared number is what you get when you take such a wobble, square it, and add up several independent copies. The count of independent copies is the count of things the claim pins down — how many separate restrictions it imposes.

So the recipe is simple to state. Compute any of the three scores. Compare it to a chi-squared yardstick whose setting is the number of restrictions. If your score is out in the tail — bigger than chance would usually produce — you reject the claim.

Two more facts round out the picture. When the claim is only slightly wrong, the score drifts upward by a predictable amount, and the chance of catching the error is higher the more wrong the claim is. And the whole story needs the hill to be smooth and the claim to sit in the open interior of what is possible; if the claim sits on a hard edge, the yardstick changes.

Visual Beginner

Picture two curves over the horizontal axis of possible score values. The first is the chi-squared yardstick for a true claim: a hump that rises near small scores and tails off to the right, so most scores land low and only rarely stray into the far tail. The second is the same shape pushed to the right — the yardstick when the claim is slightly wrong. The further the claim is from the truth, the further the second hump shifts right, and the more of it pokes past the cut-off line where you decide to reject.

situation	which yardstick	where the scores land
claim is true	central chi-squared, set by the number of restrictions	mostly low; rarely past the cut-off
claim slightly wrong	the same shape, shifted right	more often past the cut-off
more wrong	shifted further right	caught more often

The takeaway: one yardstick, set by counting restrictions, judges all three scores; a wrong claim shifts the scores rightward, and a bigger error means a bigger shift and a better chance of catching it.

Worked example Beginner

A coin is flipped $400$ times and lands heads $220$ times. Someone claims the coin is fair, so the chance of heads is $0.5$. You want to score this claim against the data using the slope-based recipe and compare it to the chi-squared yardstick.

Step 1. The expected count if the claim is true. A fair coin in $400$ flips is expected to give $400\times 0.5=200$ heads.

Step 2. The gap. You saw $220$, which is $220-200=20$ more heads than expected.

Step 3. The spread to divide by. For counting heads with a fair coin, the spread (variance) of the head count is $400\times 0.5\times 0.5=100$, so the typical wobble is the square root, $10$.

Step 4. The score. Square the standardized gap: take the gap of $20$, divide by the wobble of $10$ to get $2$, and square it to get $2\times 2=4$.

Step 5. The yardstick. The claim pins down one number — the chance of heads — so the reference is the chi-squared yardstick with one restriction. A score past about $3.84$ would happen only $5$ times in a hundred by chance.

What this tells us: the score $4$ is just past the cut-off $3.84$, so the data lean against the fair-coin claim, though not by much. The single restriction set the yardstick, and the score landed in its tail.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_1,\dots ,X_n$ be i.i.d. from a regular family $\{f(\cdot \mid \theta ):\theta \in \Theta \}$, $\Theta \subseteq {\mathbb{R}}^k$ open, with log-likelihood ${\ell }_n(\theta )={\sum }_i\log f(X_i\mid \theta )$, score $s_n(\theta )={\nabla }_{\theta }{\ell }_n(\theta )$, Fisher information $I(\theta )$, and MLE ${\hat{\theta }}_n$, all as in 45.04.03 and 45.01.05. The three statistics are the GLR, Wald, and score (Rao) constructions of 45.02.03. Convergence in distribution $\stackrel{d}{\to }$, in probability $\stackrel{\mathbb{P}}{\to }$, and the $o_{\mathbb{P}}/O_{\mathbb{P}}$ calculus are as in 45.04.01.

Definition (the three statistics for a simple null). For $H_0:\theta ={\theta }_0$ against $H_1:\theta \ne {\theta }_0$, write

$${\Lambda }_n=-2\log {\lambda }_n=2({\ell }_n({\hat{\theta }}_n)-{\ell }_n({\theta }_0)),W_n=n({\hat{\theta }}_n-{\theta }_0{)}^{\top }I({\hat{\theta }}_n)({\hat{\theta }}_n-{\theta }_0),$$ $$R_n=\frac{1}{n}{s}_n({\theta }_0{)}^{\top }I({\theta }_0{)}^{-1}{s}_n({\theta }_0),$$

the likelihood-ratio, Wald, and score statistics. (The normalisations isolate the per-observation information: with $I$ the per-observation matrix, the sample information is $nI$, so $W_n$ and $R_n$ are written with the explicit $n$ and $1/n$.)

Definition (chi-squared and noncentral chi-squared). The chi-squared distribution with $q$ degrees of freedom, ${\chi }_q^2$, is the law of $\Vert Z{\Vert }^2={\sum }_{j=1}^q{Z}_j^2$ for $Z\sim N(0,I_q)$. The noncentral chi-squared ${\chi }_q^2(\delta )$ with noncentrality $\delta \ge 0$ is the law of $\Vert Z+\mu {\Vert }^2$ for $Z\sim N(0,I_q)$ and any fixed $\mu \in {\mathbb{R}}^q$ with $\Vert \mu {\Vert }^2=\delta$; its mean is $q+\delta$. A quadratic form $V^{\top }AV$ for $V\sim N(0,\Sigma )$ with $A={\Sigma }^{-1}$ is distributed as ${\chi }_q^2$ where $q=\dim V$, since ${\Sigma }^{-1/2}V\sim N(0,I_q)$ and $V^{\top }{\Sigma }^{-1}V=\Vert {\Sigma }^{-1/2}V{\Vert }^2$.

Definition (composite null with nuisance parameters). Partition $\theta =(\psi ,\eta )$ with $\psi \in {\mathbb{R}}^q$ the parameter of interest and $\eta \in {\mathbb{R}}^{k-q}$ a nuisance parameter, and test $H_0:\psi ={\psi }_0$ ($\eta$ free) against $H_1:\psi \ne {\psi }_0$. The codimension of the null is $\dim \Theta -\dim {\Theta }_0=k-(k-q)=q$, the number of restrictions. The GLR statistic compares the unrestricted MLE ${\hat{\theta }}_n=({\hat{\psi }}_n,{\hat{\eta }}_n)$ to the restricted MLE ${\hat{\theta }}_{0,n}=({\psi }_0,{\tilde{\eta }}_n)$ that maximises the likelihood over the null; the Wald and score statistics use the profile-information block for $\psi$ after eliminating $\eta$ (the Schur complement $I_{\psi \psi \cdot \eta }=I_{\psi \psi }-I_{\psi \eta }{I}_{\eta \eta }^{-1}{I}_{\eta \psi }$).

Definition (local / contiguous alternatives). A local alternative sequence is ${\theta }_n={\theta }_0+h/\sqrt{n}$ for a fixed direction $h\in {\mathbb{R}}^k$. Under regularity (differentiability in quadratic mean) the laws $P_{{\theta }_n}^n$ and $P_{{\theta }_0}^n$ are mutually contiguous 45.04.07, so a statistic's limit under ${\theta }_n$ can be read off from its joint limit with the log-likelihood-ratio under ${\theta }_0$ via Le Cam's third lemma. The local-power calculation evaluates the trinity's limit along such sequences.

Counterexamples to common slips

• The degrees of freedom count restrictions, not parameters. For a $k$-parameter model with a $q$-dimensional interest parameter and $k-q$ nuisance parameters, $-2\log {\lambda }_n\stackrel{d}{\to }{\chi }_q^2$, not ${\chi }_k^2$. The nuisance parameters are re-estimated under both hypotheses and cancel from the codimension.
• The Wald information is taken at the MLE, the score information at the null; both still give the same chi-squared limit. The two informations $I({\hat{\theta }}_n)$ and $I({\theta }_0)$ differ at finite $n$, so $W_n$ and $R_n$ differ numerically, but both converge in probability to $I({\theta }_0)$, so the limiting laws coincide. Using the wrong information breaks the finite-sample logic without changing the limit.
• Wilks' theorem fails on the boundary. If the null places ${\theta }_0$ on the edge of $\Theta$ (e.g. testing a variance component equal to zero), the MLE is constrained to a cone and $-2\log {\lambda }_n$ has a chi-bar-squared mixture limit, not a pure ${\chi }_q^2$. The interior-point hypothesis is essential.

Key theorem with proof Intermediate+

The signature result is Wilks' theorem: under regularity and a simple interior null, all three statistics converge in null distribution to the same chi-squared law on the codimension of the null. The mechanism is the local quadratic (LAN) expansion of the log-likelihood about the truth — the same score CLT and observed-information LLN that drove the MLE limit of 45.04.03, now read off the trinity's quadratic forms ^{[van der Vaart Ch. 16]}.

Theorem (Wilks; asymptotic equivalence of the trinity under a simple null). Let $X_1,\dots ,X_n$ be i.i.d. from a regular family with true parameter ${\theta }_0$ interior to $\Theta \subseteq {\mathbb{R}}^k$, satisfying the conditions (C1)-(C3) of 45.04.03. Test $H_0:\theta ={\theta }_0$. Then, under $H_0$,

$${\Lambda }_n=-2\log {\lambda }_n\stackrel{d}{\to }{\chi }_k^2,W_n\stackrel{d}{\to }{\chi }_k^2,R_n\stackrel{d}{\to }{\chi }_k^2,$$

and moreover ${\Lambda }_n-W_n\stackrel{\mathbb{P}}{\to }0$ and ${\Lambda }_n-R_n\stackrel{\mathbb{P}}{\to }0$: the three statistics are first-order equivalent.

Proof. Write $I=I({\theta }_0)$ (positive definite by (C3)), the standardized score $Z_n=n^{-1/2}{I}^{-1/2}{s}_n({\theta }_0)$, and the rescaled estimation error $U_n=\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$. From 45.04.03, $n^{-1/2}{s}_n({\theta }_0)\stackrel{d}{\to }N(0,I)$ by the score central limit theorem 37.03.02, so $Z_n\stackrel{d}{\to }N(0,I_k)$; and the asymptotic-normality assembly gives $U_n=I^{-1}{n}^{-1/2}{s}_n({\theta }_0)+o_{\mathbb{P}}(1)$, hence $U_n\stackrel{d}{\to }N(0,I^{-1})$ and the identity

$$I^{1/2}{U}_n=I^{-1/2}{n}^{-1/2}{s}_n({\theta }_0)+o_{\mathbb{P}}(1)=Z_n+o_{\mathbb{P}}(1).\text{\hspace{1em}(∗)}$$

The Wald statistic. Since ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$ and $I(\cdot )$ is continuous, $I({\hat{\theta }}_n)\stackrel{\mathbb{P}}{\to }I$. Then

$$W_n=U_n^{\top }I({\hat{\theta }}_n)U_n=U_n^{\top }I{U}_n+U_n^{\top }(I({\hat{\theta }}_n)-I)U_n=\Vert I^{1/2}{U}_n{\Vert }^2+o_{\mathbb{P}}(1),$$

the error term being $o_{\mathbb{P}}(1)$ because $U_n=O_{\mathbb{P}}(1)$ and $I({\hat{\theta }}_n)-I=o_{\mathbb{P}}(1)$. By $(\ast )$, $W_n=\Vert Z_n{\Vert }^2+o_{\mathbb{P}}(1)\stackrel{d}{\to }\Vert N(0,I_k){\Vert }^2={\chi }_k^2$.

The score statistic. By definition $R_n=(n^{-1/2}{I}^{-1/2}{s}_n({\theta }_0){)}^{\top }(n^{-1/2}{I}^{-1/2}{s}_n({\theta }_0))=\Vert Z_n{\Vert }^2$ exactly (using $I({\theta }_0{)}^{-1}=I^{-1/2}{I}^{-1/2}$), so $R_n=\Vert Z_n{\Vert }^2\stackrel{d}{\to }{\chi }_k^2$, and $R_n-W_n=o_{\mathbb{P}}(1)$ from the line above.

The likelihood-ratio statistic. Taylor-expand ${\ell }_n$ about ${\hat{\theta }}_n$ to second order, using $\nabla {\ell }_n({\hat{\theta }}_n)=0$ (interior maximum) and $-n^{-1}{\nabla }^2{\ell }_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }I$ for ${\bar{\theta }}_n$ between ${\theta }_0$ and ${\hat{\theta }}_n$ (Proposition 3 of 45.04.03):

$${\Lambda }_n=2({\ell }_n({\hat{\theta }}_n)-{\ell }_n({\theta }_0))=-({\theta }_0-{\hat{\theta }}_n{)}^{\top }{\nabla }^2{\ell }_n({\bar{\theta }}_n)({\theta }_0-{\hat{\theta }}_n)=U_n^{\top }(-\frac{1}{n}{\nabla }^2{\ell }_n({\bar{\theta }}_n))U_n.$$

The bracket converges in probability to $I$, so ${\Lambda }_n=U_n^{\top }I{U}_n+o_{\mathbb{P}}(1)=\Vert I^{1/2}{U}_n{\Vert }^2+o_{\mathbb{P}}(1)=\Vert Z_n{\Vert }^2+o_{\mathbb{P}}(1)$ by $(\ast )$. Hence ${\Lambda }_n\stackrel{d}{\to }{\chi }_k^2$ and ${\Lambda }_n-W_n=o_{\mathbb{P}}(1)$, ${\Lambda }_n-R_n=o_{\mathbb{P}}(1)$. All three statistics equal the single quadratic form $\Vert Z_n{\Vert }^2$ to leading order. $\square$

Bridge. This theorem builds toward the entire apparatus of likelihood-based inference — confidence regions inverted from the chi-squared cutoff, analysis-of-deviance tables, and the generalized-linear-model fitting that appears again in applied regression — and it is the foundational reason the same critical value serves three different computations. This is exactly the quadratic-expansion principle of 45.04.03 read at the level of the test rather than the estimate: where that unit linearised the score to get the MLE's Gaussian law, here the same expansion turns each statistic into the squared norm of one standardized Gaussian score, so the trinity collapses to a single object. The result generalises the exact normal-case pivots of 45.02.03 — the ${\chi }_{n-1}^2$ variance pivot and the $t_{n-1}$ mean pivot — whose finite-sample exactness is replaced by an asymptotic chi-squared valid for every regular family. Putting these together, the limiting law $\Vert Z_n{\Vert }^2$ with $Z_n\stackrel{d}{\to }N(0,I_k)$ is the central fact, and the bridge is the Fisher information 45.01.05, the metric whose square root standardizes the score into a standard Gaussian and whose role as the common denominator of all three statistics is what forces them to share one chi-squared reference.

Exercises Intermediate+

Advanced results Master

One quadratic form, three readings, and the local-asymptotic-normality expansion

The chi-squared limit of all three statistics is a corollary of a single approximation: under regularity the log-likelihood ratio between ${\theta }_0+h/\sqrt{n}$ and ${\theta }_0$ admits the local asymptotic normality (LAN) expansion

$$\log \frac{d{P}_{{\theta }_0+h/\sqrt{n}}^n}{d{P}_{{\theta }_0}^n}=h^{\top }{\Delta }_n-\frac{1}{2}{h}^{\top }I({\theta }_0)h+o_{P_{{\theta }_0}}(1),{\Delta }_n=\frac{1}{\sqrt{n}}{s}_n({\theta }_0)\stackrel{d}{\to }N(0,I({\theta }_0)),$$

the central object of 45.04.06. The experiment is asymptotically a single Gaussian shift experiment with shift $h$ and precision $I({\theta }_0)$. Maximising the right side over $h$ at $h=I({\theta }_0{)}^{-1}{\Delta }_n$ gives the local MLE and the value $\frac{1}{2}{\Delta }_n^{\top }I({\theta }_0{)}^{-1}{\Delta }_n=\frac{1}{2}{R}_n$; the same maximised quadratic is what the GLR statistic computes, so ${\Lambda }_n={\Delta }_n^{\top }I({\theta }_0{)}^{-1}{\Delta }_n+o_{\mathbb{P}}(1)=\Vert Z_n{\Vert }^2+o_{\mathbb{P}}(1)$ recovers the Key theorem. The Wald and score statistics read the same quadratic from the two ends — the optimum $h$ and the gradient ${\Delta }_n$ — which is why their difference is $o_{\mathbb{P}}(1)$. In the composite case the quadratic is profiled over the nuisance directions, and Cochran's theorem splits $\Vert Z_n{\Vert }^2$ across the $q$-dimensional interest subspace and the $(k-q)$-dimensional nuisance subspace, leaving a ${\chi }_q^2$ on the interest block ^{[van der Vaart Ch. 16]}.

Local power and the noncentral chi-squared

Under the contiguous alternative ${\theta }_n={\theta }_0+h/\sqrt{n}$, contiguity 45.04.07 and Le Cam's third lemma shift the limit of ${\Delta }_n$ from $N(0,I)$ to $N(Ih,I)$, so each of ${\Lambda }_n,W_n,R_n$ converges to ${\chi }_q^2(\delta )$ with noncentrality $\delta =h_{\psi }^{\top }{I}_{\psi \psi \cdot \eta }{h}_{\psi }$ — the squared length of the interest component of the local shift in the effective (Schur-complement) information metric, the nuisance directions contributing nothing. The asymptotic power of the level-$\alpha$ test is $\mathbb{P}({\chi }_q^2(\delta )>{\chi }_{q,1-\alpha }^2)$, a strictly increasing function of $\delta$; this is the asymptotic basis for sample-size and power calculations and for the Pitman asymptotic relative efficiency of competing tests, which compares the noncentralities they accumulate per unit of local shift. Because the three statistics share both the central and the noncentral limit along $\sqrt{n}$-local sequences, they are asymptotically equally powerful to first order; their differences are second-order, governed by the curvature of ${\ell }_n$ away from the LAN parabola, and surface only under non-local (Bahadur) alternatives or in finite samples.

Composite nulls, the degrees-of-freedom rule, and estimated parameters in goodness of fit

For a nested null ${\Theta }_0\subset \Theta$ of codimension $q$, the deviance difference $-2\log {\lambda }_n$ converges to ${\chi }_q^2$, the codimension counting the constraints removed in passing from ${\Theta }_0$ to $\Theta$. The goodness-of-fit case sharpens the bookkeeping: testing a fully specified multinomial gives ${\chi }_{m-1}^2$ (one constraint, $\sum O_j=n$), but if $p$ parameters of the cell probabilities are estimated under the null — fitting a parametric family to the cells — each efficiently estimated parameter removes one further degree of freedom, giving ${\chi }_{m-1-p}^2$. The reduction is the asymptotic image of the projection in Exercise 8 acting now on a $(m-1-p)$-dimensional residual subspace; it is the Fisher-vs-Pearson degrees-of-freedom dispute of 1924, resolved in Fisher's favour by exactly this projection count. The deviance $G^2=2{\sum }_j{O}_j\log (O_j/E_j)$ and Pearson's $X^2={\sum }_j(O_j-E_j{)}^2/E_j$ are the LR and score members; a second-order expansion of $u\log (u/v)$ about $u=v$ gives $G^2=X^2+o_{\mathbb{P}}(1)$, so they share the ${\chi }_{m-1-p}^2$ limit, agreeing to first order and diverging only in cells with large relative residuals ^{[Casella & Berger §10.3]}.

Boundary and non-regular failures (Chernoff)

The chi-squared limit rests on the null sitting in the open interior of $\Theta$, so that the unrestricted MLE is asymptotically unconstrained. When ${\theta }_0$ lies on the boundary — testing whether a variance component, a mixing proportion, or a nonnegative effect equals its smallest admissible value — the MLE is projected onto a closed convex cone $C$ (the local approximation to $\Theta$ at ${\theta }_0$), and Chernoff's theorem replaces the chi-squared by a chi-bar-squared mixture ${\bar{\chi }}^2={\sum }_j{w}_j{\chi }_j^2$, whose weights $w_j$ are the probabilities that the Gaussian shift projects onto the $j$-dimensional faces of $C$ ^{[Chernoff 1954]}. The canonical instance — one nonnegative scalar parameter on the boundary ${\theta }_0=0$ — gives $-2\log {\lambda }_n\stackrel{d}{\to }\frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$, an even mixture of a point mass at zero (the MLE lands at the boundary half the time) and a ${\chi }_1^2$. Using the naive ${\chi }_1^2$ cutoff is then conservative; the correct mixture restores the nominal level. Related non-regular failures — loss of identifiability under the null (testing the number of mixture components), nuisance parameters present only under the alternative (the Davies problem), or a singular information matrix — break the LAN expansion itself and require bespoke limit theory rather than a degrees-of-freedom adjustment.

Synthesis. The central insight is that under local asymptotic normality every regular testing problem is asymptotically a Gaussian shift experiment, and the likelihood-ratio, Wald, and score statistics are three readings of one quadratic form $\Vert Z_n{\Vert }^2$, so they coincide to first order and share the chi-squared null law of 45.02.03. The foundational reason the degrees of freedom equal the codimension is Cochran's projection: the standardized score is asymptotically standard Gaussian, and restricting to the $q$-dimensional interest subspace projects it onto a ${\chi }_q^2$, the nuisance directions cancelling because they are re-estimated under both hypotheses. This is exactly the local-power mechanism in reverse — a contiguous alternative ${\theta }_0+h/\sqrt{n}$ shifts the Gaussian's mean by $Ih$ 45.04.07, turning the central ${\chi }_q^2$ into a noncentral ${\chi }_q^2(\delta )$ with $\delta =h_{\psi }^{\top }{I}_{\psi \psi \cdot \eta }{h}_{\psi }$, the power monotone in this single noncentrality. The chi-squared regime is dual to the Chernoff boundary regime: an interior null leaves the MLE asymptotically free and yields a clean ${\chi }_q^2$, a boundary null projects it onto a cone and yields a chi-bar-squared mixture. Putting these together, the finite-sample exact pivots of 45.02.03 generalise to a universal asymptotic chi-squared, the LAN expansion of 45.04.06 supplies the quadratic, contiguity 45.04.07 supplies the local power, and the bridge is the Fisher information 45.01.05, whose square root standardizes the score into the standard Gaussian whose squared projected norm is the chi-squared all three statistics inherit.

Full proof set Master

Proposition 1 (a Gaussian quadratic form with inverse-covariance weight is chi-squared). If $V\sim N(0,\Sigma )$ in ${\mathbb{R}}^q$ with $\Sigma \succ 0$, then $V^{\top }{\Sigma }^{-1}V\sim {\chi }_q^2$.

Proof. Set $Z={\Sigma }^{-1/2}V$, well-defined since $\Sigma \succ 0$ has a symmetric positive-definite square root. Then $Z$ is Gaussian with mean $0$ and covariance ${\Sigma }^{-1/2}\Sigma {\Sigma }^{-1/2}=I_q$, so $Z\sim N(0,I_q)$ with independent standard-normal coordinates. Hence $V^{\top }{\Sigma }^{-1}V=V^{\top }{\Sigma }^{-1/2}{\Sigma }^{-1/2}V=\Vert Z{\Vert }^2={\sum }_{j=1}^q{Z}_j^2\sim {\chi }_q^2$ by the definition of the chi-squared law. $\square$

Proposition 2 (Cochran: a Gaussian quadratic form on an idempotent covariance). If $V\sim N(0,\Pi )$ with $\Pi$ a symmetric idempotent matrix (${\Pi }^2=\Pi ={\Pi }^{\top }$) of rank $r$, then $\Vert V{\Vert }^2=V^{\top }V\sim {\chi }_r^2$.

Proof. As an orthogonal projection, $\Pi$ has eigenvalues $1$ (multiplicity $r$) and $0$ (multiplicity $m-r$); diagonalise $\Pi =Q\mathrm{diag}(I_r,0)Q^{\top }$ with $Q$ orthogonal. Then $W:=Q^{\top }V\sim N(0,Q^{\top }\Pi Q)=N(0,\mathrm{diag}(I_r,0))$, so the first $r$ coordinates of $W$ are i.i.d. $N(0,1)$ and the remaining $m-r$ are degenerate at $0$. Since $\Vert V{\Vert }^2=\Vert W{\Vert }^2={\sum }_{j=1}^r{W}_j^2$, the sum is ${\chi }_r^2$. (This is the goodness-of-fit case: $\Pi =I_m-u{u}^{\top }$ has rank $m-1$, giving $X^2\stackrel{d}{\to }{\chi }_{m-1}^2$ in Exercise 8.) $\square$

Proposition 3 (the LAN quadratic and the GLR limit). Under (C1)-(C3) of 45.04.03 with simple interior null ${\theta }_0$, ${\Lambda }_n={\Delta }_n^{\top }I({\theta }_0{)}^{-1}{\Delta }_n+o_{\mathbb{P}}(1)$ where ${\Delta }_n=n^{-1/2}{s}_n({\theta }_0)$, hence ${\Lambda }_n\stackrel{d}{\to }{\chi }_k^2$.

Proof. Let $I=I({\theta }_0)$ and $U_n=\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$. The asymptotic-linearity of the MLE (Proposition 4 of 45.04.03) gives $U_n=I^{-1}{\Delta }_n+o_{\mathbb{P}}(1)$. A second-order Taylor expansion of ${\ell }_n$ about ${\hat{\theta }}_n$, with $\nabla {\ell }_n({\hat{\theta }}_n)=0$ at the interior maximum and remainder at ${\bar{\theta }}_n$ between ${\theta }_0$ and ${\hat{\theta }}_n$, gives ${\Lambda }_n=U_n^{\top }{A}_n{U}_n$ with $A_n=-n^{-1}{\nabla }^2{\ell }_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }I$ (Proposition 3 of 45.04.03). Substituting and using $U_n=O_{\mathbb{P}}(1)$,

$${\Lambda }_n=(I^{-1}{\Delta }_n{)}^{\top }I(I^{-1}{\Delta }_n)+o_{\mathbb{P}}(1)={\Delta }_n^{\top }{I}^{-1}{\Delta }_n+o_{\mathbb{P}}(1).$$

Since ${\Delta }_n\stackrel{d}{\to }N(0,I)$ by the score CLT (Proposition 2 of 45.04.03), Proposition 1 gives ${\Delta }_n^{\top }{I}^{-1}{\Delta }_n\stackrel{d}{\to }{\chi }_k^2$, and the continuous-mapping/Slutsky argument transfers the limit through the $o_{\mathbb{P}}(1)$. $\square$

Proposition 4 (first-order equivalence of the trinity). Under the same hypotheses, ${\Lambda }_n-W_n=o_{\mathbb{P}}(1)$ and ${\Lambda }_n-R_n=o_{\mathbb{P}}(1)$, so all three converge to ${\chi }_k^2$.

Proof. With $U_n=I^{-1}{\Delta }_n+o_{\mathbb{P}}(1)$: the score statistic is $R_n={\Delta }_n^{\top }{I}^{-1}{\Delta }_n$ identically (its definition with $I({\theta }_0{)}^{-1}$), matching ${\Lambda }_n$ up to $o_{\mathbb{P}}(1)$ by Proposition 3. The Wald statistic is $W_n=U_n^{\top }I({\hat{\theta }}_n)U_n$; since $I({\hat{\theta }}_n)=I+o_{\mathbb{P}}(1)$ by continuity and consistency, and $U_n=O_{\mathbb{P}}(1)$,

$$W_n=U_n^{\top }I{U}_n+o_{\mathbb{P}}(1)=(I^{-1}{\Delta }_n{)}^{\top }I(I^{-1}{\Delta }_n)+o_{\mathbb{P}}(1)={\Delta }_n^{\top }{I}^{-1}{\Delta }_n+o_{\mathbb{P}}(1),$$

again matching ${\Lambda }_n$ and $R_n$ up to $o_{\mathbb{P}}(1)$. All three equal the single form ${\Delta }_n^{\top }{I}^{-1}{\Delta }_n+o_{\mathbb{P}}(1)$, hence share the ${\chi }_k^2$ limit and differ by $o_{\mathbb{P}}(1)$. $\square$

Proposition 5 (noncentral chi-squared local power). Under ${\theta }_n={\theta }_0+h/\sqrt{n}$, each of ${\Lambda }_n,W_n,R_n\stackrel{d}{\to }{\chi }_k^2(\delta )$ with $\delta =h^{\top }I({\theta }_0)h$ (simple null), and the asymptotic power of the level-$\alpha$ test is $\mathbb{P}({\chi }_k^2(\delta )>{\chi }_{k,1-\alpha }^2)$, strictly increasing in $\delta$.

Proof. By contiguity of $P_{{\theta }_n}^n$ to $P_{{\theta }_0}^n$ 45.04.07 (differentiability in quadratic mean gives the LAN expansion, and Le Cam's first lemma yields contiguity), the joint limit of $({\Delta }_n,\log (d{P}_{{\theta }_n}^n/d{P}_{{\theta }_0}^n))$ under ${\theta }_0$ is bivariate Gaussian with the LAN covariance, so Le Cam's third lemma gives, under ${\theta }_n$, ${\Delta }_n\stackrel{d}{\to }N(Ih,I)$ with $I=I({\theta }_0)$. Standardizing, $Z_n=I^{-1/2}{\Delta }_n\stackrel{d}{\to }N(I^{1/2}h,I_k)$, so $R_n=\Vert Z_n{\Vert }^2\stackrel{d}{\to }\Vert N(I^{1/2}h,I_k){\Vert }^2={\chi }_k^2(\delta )$ with $\delta =\Vert I^{1/2}h{\Vert }^2=h^{\top }Ih$. Propositions 3-4 give ${\Lambda }_n=W_n=R_n+o_{\mathbb{P}}(1)$, and contiguity preserves $o_{\mathbb{P}}(1)$ statements under ${\theta }_n$, so all three share the ${\chi }_k^2(\delta )$ limit. The power is $\mathbb{P}({\chi }_k^2(\delta )>c_{\alpha })$ with $c_{\alpha }={\chi }_{k,1-\alpha }^2$; the family ${\chi }_k^2(\delta )$ is stochastically increasing in $\delta$ (its representation $\Vert Z+\mu {\Vert }^2$ with $\Vert \mu {\Vert }^2=\delta$ is monotone in $\Vert \mu \Vert$), so the power strictly increases in $\delta$. $\square$

Proposition 6 (boundary null: the even chi-bar-squared mixture). For a single nonnegative parameter $\theta \ge 0$ with regular interior behaviour and true value ${\theta }_0=0$ on the boundary, $-2\log {\lambda }_n\stackrel{d}{\to }\frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$.

Proof. Localise: $U_n=\sqrt{n}{\hat{\theta }}_n$ where ${\hat{\theta }}_n=\max ({\check{\theta }}_n,0)$ is the unrestricted MLE projected onto $[0,\infty )$, and ${\check{\theta }}_n$ is the unconstrained root with $\sqrt{n}{\check{\theta }}_n\stackrel{d}{\to }Z\sim N(0,I^{-1})$, $I=I(0)$. The LAN expansion gives $-2\log {\lambda }_n=(\max (\sqrt{n}{\check{\theta }}_n,0){)}^{2}I+o_{\mathbb{P}}(1)=(\max (Z,0){)}^{2}I+o_{\mathbb{P}}(1)$ in the limit, since the constrained maximiser equals $0$ when ${\check{\theta }}_n\le 0$ and ${\check{\theta }}_n$ otherwise. Now $W:=(\max (Z,0){)}^{2}I$: with probability $\mathbb{P}(Z\le 0)=\frac{1}{2}$, $W=0$ (a point mass ${\delta }_0$); with probability $\frac{1}{2}$, $W=Z^{2}I$ conditioned on $Z>0$, and $Z^{2}I=(Z/\sqrt{I^{-1}}{)}^2\sim {\chi }_1^2$ with the conditioning on $\{Z>0\}$ acting on a symmetric law so that the conditional law of $Z^{2}I$ is again ${\chi }_1^2$. Hence $W\sim \frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$ ^{[Chernoff 1954]}. The naive ${\chi }_1^2$ test that ignores the boundary over-states the critical value's tail mass and is conservative; the correct mixture uses ${\chi }_{1,1-2\alpha }^2$ for level $\alpha$. $\square$

Connections Master

• The asymptotic normality of the MLE 45.04.03 is the engine of every limit here: the score central limit theorem $n^{-1/2}{s}_n({\theta }_0)\stackrel{d}{\to }N(0,I)$ and the observed-information law of large numbers $-n^{-1}{\nabla }^2{\ell }_n\stackrel{\mathbb{P}}{\to }I$, proved there, are exactly the two ingredients that turn each test statistic into the squared norm of a standardized Gaussian score. The MLE's $N(0,I^{-1})$ law is the Wald statistic's input; the score's $N(0,I)$ law is the score statistic's.

• The finite-sample trinity 45.02.03 supplies the constructions whose asymptotic null law this unit derives: the GLR statistic $-2\log \lambda$, the Wald quadratic form, and the score/Rao statistic are defined there with their exact normal-case pivots, and this unit shows those exact ${\chi }_{n-1}^2$ and $t_{n-1}$ pivots are the special cases of a universal chi-squared limit valid for every regular family, while Pearson's $X^2$ is the multinomial score statistic attaining ${\chi }_{m-1-p}^2$.

• Local asymptotic normality 45.04.06 provides the quadratic expansion on which Wilks' theorem rests: the LAN representation $\log (d{P}_{{\theta }_0+h/\sqrt{n}}^n/d{P}_{{\theta }_0}^n)=h^{\top }{\Delta }_n-\frac{1}{2}{h}^{\top }Ih+o_{\mathbb{P}}(1)$ is the statement that the experiment is asymptotically Gaussian-shift, and maximising it over $h$ yields the same quadratic form ${\Delta }_n^{\top }{I}^{-1}{\Delta }_n$ that all three statistics compute, making the chi-squared limit a corollary of the Gaussian-shift structure.

• Contiguity and Le Cam's lemmas 45.04.07 deliver the local-power theory: contiguity of $P_{{\theta }_0+h/\sqrt{n}}^n$ to $P_{{\theta }_0}^n$ lets the third lemma transport the null limit $N(0,I)$ of ${\Delta }_n$ to the alternative limit $N(Ih,I)$, converting the central ${\chi }_q^2$ into the noncentral ${\chi }_q^2(\delta )$ with $\delta =h^{\top }Ih$, and thereby furnishing the asymptotic power, Pitman efficiency, and sample-size machinery for the trinity.

• Fisher information and the Cramér-Rao bound 45.01.05 is the standardizing metric throughout: the chi-squared limit appears because $I^{-1/2}$ converts the asymptotically normal score into a standard Gaussian whose squared norm is chi-squared, the codimension counts the dimension of the projected Gaussian, and the noncentrality is the squared local shift measured in the information metric, with the nuisance-adjusted Schur complement $I_{\psi \psi \cdot \eta }$ playing the role of the effective information for the parameter of interest.

Historical & philosophical context Master

The asymptotic chi-squared distribution of minus twice the log-likelihood ratio was established by Samuel S. Wilks in 1938 ^{[Wilks 1938]} (Ann. Math. Statist. 9, 60-62), extending the likelihood-ratio principle of Neyman and Pearson to a general large-sample reference distribution and fixing the degrees of freedom as the number of restrictions. The Wald (1943) and Rao (1948) statistics were shown to share this limit and to be first-order equivalent to the likelihood-ratio statistic in the post-war large-sample theory, the equivalence resting on the common quadratic approximation of the log-likelihood that all three statistics evaluate at different points. Pearson's chi-squared statistic of 1900 predates the trinity by decades; its identification as the multinomial score statistic, and the correct degrees-of-freedom count under estimated parameters, were settled by R. A. Fisher in 1924 against Karl Pearson's original count, the dispute resolved by the projection that removes one degree of freedom per efficiently estimated parameter.

The boundary failure was charted by Herman Chernoff in 1954 ^{[Chernoff 1954]} (Ann. Math. Statist. 25, 573-578), who showed that when the true parameter lies on the edge of the parameter space the likelihood-ratio statistic follows a chi-bar-squared mixture determined by the local cone, the half-and-half mixture $\frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$ being the canonical one-sided case. The deeper synthesis — that the chi-squared limit, its noncentral local-power version, and the boundary mixtures are all consequences of the local asymptotic normality of the experiment — is due to Lucien Le Cam and Jaroslav Hájek, whose contiguity calculus and convolution theory recast likelihood asymptotics as the study of a limiting Gaussian shift experiment, the framework in which van der Vaart's modern treatment (Asymptotic Statistics, 1998, Ch. 16) presents the trinity.

Bibliography Master

@book{vanderVaart1998asymptotic,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{lehmannromano2005testing,
  author    = {Lehmann, Erich L. and Romano, Joseph P.},
  title     = {Testing Statistical Hypotheses},
  edition   = {3rd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {2005}
}

@book{casellaberger2002statistical,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2nd},
  publisher = {Duxbury / Thomson Learning},
  year      = {2002}
}

@article{wilks1938large,
  author  = {Wilks, Samuel S.},
  title   = {The large-sample distribution of the likelihood ratio for testing composite hypotheses},
  journal = {The Annals of Mathematical Statistics},
  volume  = {9},
  number  = {1},
  pages   = {60--62},
  year    = {1938}
}

@article{chernoff1954distribution,
  author  = {Chernoff, Herman},
  title   = {On the distribution of the likelihood ratio},
  journal = {The Annals of Mathematical Statistics},
  volume  = {25},
  number  = {3},
  pages   = {573--578},
  year    = {1954}
}

@article{wald1943tests,
  author  = {Wald, Abraham},
  title   = {Tests of statistical hypotheses concerning several parameters when the number of observations is large},
  journal = {Transactions of the American Mathematical Society},
  volume  = {54},
  number  = {3},
  pages   = {426--482},
  year    = {1943}
}

@article{rao1948large,
  author  = {Rao, C. Radhakrishna},
  title   = {Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation},
  journal = {Proceedings of the Cambridge Philosophical Society},
  volume  = {44},
  number  = {1},
  pages   = {50--57},
  year    = {1948}
}

@article{fisher1924conditions,
  author  = {Fisher, Ronald A.},
  title   = {The conditions under which chi-square measures the discrepancy between observation and hypothesis},
  journal = {Journal of the Royal Statistical Society},
  volume  = {87},
  number  = {3},
  pages   = {442--450},
  year    = {1924}
}