Likelihood-Ratio, Wald, and Score Tests

Intuition Beginner

You have fit a model to data and you want to ask a yes-or-no question about it: is the true value of some quantity equal to a specific number, or not? You need a single number — a test statistic — that is small when the data agree with the claim and large when they argue against it. This unit is about three general recipes for building that number, recipes that work for almost any model.

Picture the log-likelihood as a hill. Its peak sits over the best-fit value, the one the data prefer most. The claim you are testing points to one particular spot on the horizontal axis. The three recipes all measure, in different ways, how far that claimed spot is from the peak.

The first recipe compares heights: how much lower is the hill at the claimed spot than at its peak? A big drop means the claim fits much worse than the best fit, so you reject it. The second recipe measures horizontal distance: how far is the claimed spot from the peak, scaled by how sharply the hill curves? The third recipe measures slope: how steep is the hill at the claimed spot? At the true peak the slope is flat, so a steep slope at the claim is evidence against it.

These three views — the drop in height, the horizontal distance, and the slope — are three windows onto the same hill, and they usually agree.

Visual Beginner

Figure: a single hill-shaped curve, the log-likelihood, plotted against candidate parameter values. Its peak sits above the best-fit value. A dashed line marks the claimed value being tested. Three measurements are drawn. A vertical arrow shows the drop in height from the peak down to the curve at the claim: the likelihood-ratio view. A horizontal arrow shows the gap from the claim to the peak: the Wald view. A short tangent at the claim shows the slope there: the score view. The flatter the hill, the smaller all three; the sharper the hill, the larger they grow for the same gap.

 log-likelihood
   |                 . peak (best fit)
   |               /   \
   |              /  ^  \        height drop = likelihood-ratio view
   |   slope --> /   |   \
   |   (tangent)/    |    \
   |          ./     |     \
   |         /|<-----+      \      horizontal gap = Wald view
   |        / |   claimed    \
   +-------------------------------> parameter value
            ^             ^
        claimed         peak
        value          (MLE)

Worked example Beginner

A bakery claims its loaves average $500$ grams. You weigh $10$ loaves and the model is a bell-shaped spread around an unknown true average, with the spread estimated from the data. Your data give a sample average of $508$ grams and a sample standard deviation of $12$ grams. You want to test the claim "the true average is $500$."

Step 1. Find the best fit. The best-fit average is simply the sample average, $508$ grams. The hill peaks there.

Step 2. Measure the horizontal gap (the Wald view). The claimed value $500$ sits $508-500=8$ grams below the peak. To make this a fair, unit-free number you divide by the standard error, which is the standard deviation divided by the square root of the count: $12/\sqrt{10}=12/3.162=3.795$ grams. The scaled gap is $8/3.795=2.11$.

Step 3. Read off the slope view. Here the slope at the claimed value points the same way: the steeper the hill is at $500$, the stronger the push away from it. For this bell-shaped model the slope measurement, scaled the same way, also comes out near $2.11$, because the hill is close to a clean parabola.

Step 4. Read off the height-drop view. Comparing the height at the peak to the height at $500$ gives a drop whose scaled size again lands near $2.11$ for a sample this clean.

What this tells us. A scaled gap of about $2.1$ is on the large side — it would happen by chance only a few times in a hundred if the claim were true — so the data lean against the $500$-gram claim. The lesson is that all three recipes converted the same $8$-gram discrepancy into nearly the same verdict. They are three readings of one hill.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_1,\dots ,X_n$ be i.i.d. from $\{f(x\mid \theta ):\theta \in \Theta \}$, $\Theta \subseteq {\mathbb{R}}^k$, with likelihood $L(\theta \mid x)={\prod }_{i}f(x_i\mid \theta )$, log-likelihood $\ell (\theta )$, score $s(\theta )={\nabla }_{\theta }\ell (\theta )$, and (when it exists) MLE $\hat{\theta }=\arg {\max }_{\theta }\ell (\theta )$, all as in 45.01.04. Test $H_0:\theta \in {\Theta }_0$ against $H_1:\theta \in \Theta \setminus {\Theta }_0$, where ${\Theta }_0\subseteq \Theta$.

Definition (generalized likelihood-ratio statistic). The generalized likelihood-ratio (GLR) statistic is

$$\lambda (x)=\frac{{\sup }_{\theta \in {\Theta }_0}L(\theta \mid x)}{{\sup }_{\theta \in \Theta }L(\theta \mid x)}=\frac{L({\hat{\theta }}_0\mid x)}{L(\hat{\theta }\mid x)},$$

where ${\hat{\theta }}_0$ is the MLE constrained to ${\Theta }_0$ (the restricted MLE) and $\hat{\theta }$ is the unrestricted MLE. By construction $0\le \lambda (x)\le 1$, and small $\lambda$ — the null fit far worse than the best fit — is evidence against $H_0$. The generalized likelihood-ratio test (GLRT) rejects $H_0$ when $\lambda (x)\le c$ for a constant $c$ chosen to give the desired level. The equivalent log form is the statistic $-2\log \lambda (x)=2(\ell (\hat{\theta })-\ell ({\hat{\theta }}_0))\ge 0$, on which the test rejects for large values. When $H_0:\theta ={\theta }_0$ is simple, $\lambda (x)=L({\theta }_0)/L(\hat{\theta })$, and the GLRT is the Neyman-Pearson likelihood-ratio test of 45.02.01 with the alternative density replaced by its maximized version.

Definition (Wald statistic). Suppose $H_0:\theta ={\theta }_0$ (scalar, for definiteness; the vector case replaces the square by a quadratic form). The Wald statistic standardizes the gap between the MLE and the null by an estimate of the MLE's standard error:

$$W=(\hat{\theta }-{\theta }_0{)}^{2}I(\hat{\theta }),\text{or in the vector case}W=(\hat{\theta }-{\theta }_0{)}^{\top }I(\hat{\theta })(\hat{\theta }-{\theta }_0),$$

where $I(\cdot )$ is the Fisher (or observed) information 45.01.05, the curvature of $\ell$ that fixes the scale. The test rejects for large $W$. Only the unrestricted MLE $\hat{\theta }$ and the information at it are needed; the model is fit once, under $H_1$.

Definition (score / Rao / Lagrange-multiplier statistic). With the same $H_0:\theta ={\theta }_0$, the score statistic evaluates the score at the null and standardizes it by the information at the null:

$$R=s({\theta }_0{)}^{\top }I({\theta }_0{)}^{-1}s({\theta }_0),s({\theta }_0)={\nabla }_{\theta }\ell (\theta ){|}_{\theta ={\theta }_0}.$$

The test rejects for large $R$. Because $s(\hat{\theta })=0$ at an interior unrestricted maximum, $R$ measures how far the null is from satisfying the likelihood equations; it needs the fit only under $H_0$ (no unrestricted maximization), which is its computational advantage. In a constrained problem the score at the restricted MLE is proportional to the Lagrange multiplier of the constraint, giving the test its "Lagrange-multiplier" name.

Counterexamples to common slips

• The GLRT is not the Neyman-Pearson test for composite alternatives. When ${\Theta }_1$ is composite the GLRT replaces ${\sup }_{{\theta }_1}L$ by a maximized likelihood; this need not be most powerful at any single alternative, and the Neyman-Pearson optimality of 45.02.01 does not transfer. The GLRT is a general-purpose recipe, not an optimum.
• Wald is not invariant to reparametrization; the GLRT is. Testing $\theta ={\theta }_0$ by Wald can give a different answer than testing $g(\theta )=g({\theta }_0)$ for a nonlinear $g$, because the quadratic form changes. The GLR statistic $\lambda$ is unchanged by any reparametrization, since suprema of $L$ are coordinate-free.
• The score test uses information at the null, the Wald test at the MLE. Swapping them is a common error. Using $I(\hat{\theta })$ inside $R$, or $I({\theta }_0)$ inside $W$, breaks the construction's logic (slope-at-null versus distance-from-null) even though the two informations are close when the hypotheses are close.

Key theorem with proof Intermediate+

The signature finite-sample fact is that for the normal location and scale problems the GLRT is an exact, computable test: the GLR statistic is a monotone function of a familiar pivot, so its exact null distribution is known without any large-sample approximation. This is what makes the trinity constructional before it is asymptotic ^{[Casella & Berger §8.2.1]}.

Theorem (exact GLRT for a normal variance). Let $X_1,\dots ,X_n$ be i.i.d. $N(\mu ,{\sigma }^2)$ with both parameters unknown, and test $H_0:{\sigma }^2={\sigma }_0^2$ against $H_1:{\sigma }^2\ne {\sigma }_0^2$. Write ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(X_i-\bar{X}{)}^2$ for the unrestricted variance MLE and $V={\sum }_i(X_i-\bar{X}{)}^2=n{\hat{\sigma }}^2$. Then the GLR statistic is

$$\lambda (x)={\left(\frac{{\hat{\sigma }}^2}{{\sigma }_0^2}\right)}^{n/2}\exp \left(-\frac{n}{2}(\frac{{\hat{\sigma }}^2}{{\sigma }_0^2}-1)\right),$$

a function of the single quantity $t=V/{\sigma }_0^2=(n-1)S^2/{\sigma }_0^2$ alone, where $S^2=V/(n-1)$. Under $H_0$, $t\sim {\chi }_{n-1}^2$, so the GLRT — reject when $\lambda \le c$ — has an exact null distribution, and the rejection region is $\{t\le a\}\cup \{t\ge b\}$ for constants determined by the level.

Proof. Under $H_1$ the parameter space is $\Theta =\{(\mu ,{\sigma }^2):\mu \in \mathbb{R},{\sigma }^2>0\}$, and the unrestricted MLE is $(\hat{\mu },{\hat{\sigma }}^2)=(\bar{X},{\hat{\sigma }}^2)$ by 45.01.04, giving

$$L(\hat{\theta })=(2\pi {\hat{\sigma }}^2{)}^{-n/2}\exp (-\frac{1}{2{\hat{\sigma }}^2}\sum _i(x_i-\bar{x}{)}^2)=(2\pi {\hat{\sigma }}^2{)}^{-n/2}{e}^{-n/2}.$$

Under $H_0$, ${\sigma }^2={\sigma }_0^2$ is fixed and only $\mu$ is free; the restricted MLE is ${\hat{\mu }}_0=\bar{X}$ (the mean MLE does not depend on ${\sigma }^2$), so

$$L({\hat{\theta }}_0)=(2\pi {\sigma }_0^2{)}^{-n/2}\exp (-\frac{1}{2{\sigma }_0^2}\sum _i(x_i-\bar{x}{)}^2)=(2\pi {\sigma }_0^2{)}^{-n/2}\exp (-\frac{n{\hat{\sigma }}^2}{2{\sigma }_0^2}).$$

Forming the ratio, the $(2\pi {)}^{-n/2}$ factors cancel and

$$\lambda =\frac{L({\hat{\theta }}_0)}{L(\hat{\theta })}=\frac{({\sigma }_0^2{)}^{-n/2}\exp (-n{\hat{\sigma }}^2/2{\sigma }_0^2)}{({\hat{\sigma }}^2{)}^{-n/2}{e}^{-n/2}}={\left(\frac{{\hat{\sigma }}^2}{{\sigma }_0^2}\right)}^{n/2}\exp \left(-\frac{n}{2}(\frac{{\hat{\sigma }}^2}{{\sigma }_0^2}-1)\right),$$

the displayed expression. Setting $r={\hat{\sigma }}^2/{\sigma }_0^2=t/n$, the map $r\mapsto r^{n/2}{e}^{-n(r-1)/2}$ rises on $r\in (0,1)$ and falls on $r\in (1,\infty )$, peaking at $r=1$ with value $1$; it is therefore a strictly unimodal function of $t$. Hence $\{\lambda \le c\}$ is exactly a two-sided region $\{t\le a\}\cup \{t\ge b\}$ in the statistic $t=V/{\sigma }_0^2$. By Cochran's theorem $V/{\sigma }^2={\sum }_i(X_i-\bar{X}{)}^2/{\sigma }^2\sim {\chi }_{n-1}^2$, so under $H_0$ (${\sigma }^2={\sigma }_0^2$) the statistic $t=V/{\sigma }_0^2\sim {\chi }_{n-1}^2$ exactly, and $a,b$ are chosen from the ${\chi }_{n-1}^2$ quantiles to give size $\alpha$ (for instance $a={\chi }_{n-1,\alpha /2}^2$, $b={\chi }_{n-1,1-\alpha /2}^2$ for the equal-tail version). $\square$

Bridge. This exact reduction builds toward the asymptotic theory of 45.04.09: the same statistic $-2\log \lambda$, which is finite-sample-exact here through the ${\chi }_{n-1}^2$ pivot, has under regularity a common ${\chi }^2$ limiting null distribution (Wilks) that appears again in every regular testing problem, and the equal-tail ${\chi }_{n-1}^2$ cutoffs are the finite-sample ancestor of that limit. This is exactly the Neyman-Pearson logic of 45.02.01 lifted to a composite problem: where the simple-vs-simple lemma thresholds $p_1/p_0$, the GLRT thresholds the maximized ratio ${\sup }_{{\Theta }_0}L/{\sup }_{\Theta }L$, so the likelihood ratio remains the privileged object and the GLRT generalises the most-powerful test to nested composite hypotheses. The central insight is that the GLR statistic collapses onto a low-dimensional pivot — here the chi-squared variable — whenever the model has the exponential/normal structure, which is the foundational reason the trinity is computable in closed form before any limit is taken. Putting these together, the Wald and score statistics, which curve-fit the same hill at the MLE and at the null respectively, agree with $-2\log \lambda$ to leading order, and the bridge is that all three are quadratic forms in the score standardized by the information, an identity made exact in the normal case and asymptotic in general.

Exercises Intermediate+

Advanced results Master

The geometry of the trinity: distance, slope, and ratio of one log-likelihood

Three measurements of the gap between the null and the maximum read off the same log-likelihood surface. The likelihood-ratio statistic $-2\log \lambda =2(\ell (\hat{\theta })-\ell ({\hat{\theta }}_0))$ is the vertical drop in $\ell$ from the unrestricted maximum to the constrained maximum. The Wald statistic $W=(\hat{\theta }-{\theta }_0{)}^{\top }I(\hat{\theta })(\hat{\theta }-{\theta }_0)$ is the squared horizontal distance from $\hat{\theta }$ to ${\theta }_0$ measured in the Riemannian metric supplied by the information 45.01.05. The score statistic $R=s({\theta }_0{)}^{\top }I({\theta }_0{)}^{-1}s({\theta }_0)$ is the squared length of the gradient of $\ell$ at ${\theta }_0$, again in the information metric. When $\ell$ is exactly quadratic the three agree identically (Exercise 7); in general they are the same number computed by three approximations to the curve — secant drop, tangent-at-the-top, tangent-at-the-null. The information metric is the common standardizer that makes each scale-free and removes the dependence on units of $\theta$, and it is the reason the three statistics share an asymptotic distribution once the curvature is estimated consistently.

Nested hypotheses, degrees of freedom, and the deviance

For nested models ${\Theta }_0\subset \Theta$ with $\dim \Theta -\dim {\Theta }_0=q$, the GLR statistic compares two fits differing in $q$ free parameters, and $-2\log \lambda$ is the deviance difference between the constrained and the full model ^{[Casella & Berger §8.2.1]}. The exact normal-variance and normal-mean cases above reduce $-2\log \lambda$ to functions of ${\chi }^2$ and $t$ pivots, but the deviance organizes the general nested comparison: in exponential-family generalized linear models, $-2\log \lambda =D(\text{null})-D(\text{full})$ where $D$ is the model deviance, and the analysis-of-deviance table is the likelihood analogue of analysis of variance. The integer $q$ — the codimension of ${\Theta }_0$ — is the number that becomes the degrees of freedom of the common limiting null distribution; the finite-sample constructions of this unit are the cases where that limit is attained exactly.

Pearson versus deviance: two members of the trinity for goodness of fit

The multinomial goodness-of-fit problem exhibits two of the three constructions side by side. The likelihood-ratio statistic is the deviance $G^2=2{\sum }_j{O}_j\log (O_j/E_j)$ (Exercise 5); the score statistic is Pearson's $X^2={\sum }_j(O_j-E_j{)}^2/E_j$, obtained by standardizing each cell residual by its null variance and ignoring the off-diagonal correlations, which the multinomial information accounts for through the constraint ${\sum }_j{O}_j=n$. A Taylor expansion of $u\log (u/v)$ about $u=v$ gives $G^2=X^2+O({\sum }_j(O_j-E_j{)}^3/E_j^2)$, so the two agree to second order and diverge only in cells with large relative residuals — $G^2$ is more sensitive to cells where $O_j\gg E_j$, $X^2$ to cells where $O_j\ll E_j$. Both are referred to the same ${\chi }_{m-1-p}^2$ reference (with $p$ estimated parameters), the asymptotic content of 45.04.09, but their finite-sample behaviour differs, and the choice between them is a recurring applied question.

The score test's structural advantage and the union-intersection bridge

The score statistic requires only the restricted fit ${\hat{\theta }}_0$, never the unrestricted MLE, which is decisive when the alternative model is hard to fit but the null is simple — adding a parameter to a fitted model and asking whether it is needed, the Lagrange-multiplier test of econometrics ^{[Rao 1948]}. Dually, the Wald test requires only the unrestricted fit, convenient when many nulls are to be tested against one fitted model. The GLRT requires both fits and is the most computationally expensive but the only reparametrization-invariant member (Exercise 8). The GLRT also connects to the union-intersection construction: a composite null written as an intersection ${\Theta }_0={\bigcap }_{\gamma }{\Theta }_{0\gamma }$ yields a test rejecting when any component likelihood ratio is extreme, which for the normal linear model recovers the $F$- and $t$-based GLRTs and ties the trinity to the confidence-set duality that organizes the chapter.

Synthesis. The central insight is that the likelihood-ratio, Wald, and score statistics are three readings of a single log-likelihood surface — the drop in height, the squared distance to the maximum, and the squared slope at the null — each standardized by the information metric, and this single geometric picture is exactly what makes them coincide for a quadratic log-likelihood and agree to first order in general. The foundational reason the likelihood ratio is privileged is the same one that drives 45.02.01: thresholding a maximized ratio ${\sup }_{{\Theta }_0}L/{\sup }_{\Theta }L$ generalises the Neyman-Pearson most-powerful test from simple-versus-simple to nested composite hypotheses, and the GLRT inherits the coordinate-freeness that the ratio enjoys while the Wald form does not. Putting these together, the normal-variance and normal-mean GLRTs reduce $-2\log \lambda$ to exact ${\chi }^2$ and $t$ pivots, the multinomial deviance $G^2$ and Pearson's $X^2$ realize the LR and score members for goodness of fit, and all of these appear again in the asymptotic theory of 45.04.09, where the codimension $q$ becomes the degrees of freedom of the common chi-squared limit. The score test's null-only fit is dual to the Wald test's alternative-only fit, and the bridge is the information matrix 45.01.05, the Riemannian metric whose role as the common standardizer is what turns three different finite-sample constructions into one asymptotically pivotal family and what carries this optimality into the dual confidence sets of the chapter.

Full proof set Master

Proposition 1 (GLR statistic bounds and the log form). For any ${\Theta }_0\subseteq \Theta$, $0\le \lambda (x)\le 1$, and $-2\log \lambda (x)=2(\ell (\hat{\theta })-\ell ({\hat{\theta }}_0))\ge 0$, with $\lambda (x)=1$ iff a constrained maximizer is also an unrestricted maximizer.

Proof. Since ${\Theta }_0\subseteq \Theta$, ${\sup }_{\theta \in {\Theta }_0}L(\theta )\le {\sup }_{\theta \in \Theta }L(\theta )$, and both suprema are nonnegative as suprema of a nonnegative function, so the ratio lies in $[0,1]$ (the denominator is positive whenever the data have positive likelihood somewhere). Taking logarithms, $\log \lambda =\ell ({\hat{\theta }}_0)-\ell (\hat{\theta })\le 0$, hence $-2\log \lambda =2(\ell (\hat{\theta })-\ell ({\hat{\theta }}_0))\ge 0$. Equality $\lambda =1$ holds iff $\ell ({\hat{\theta }}_0)=\ell (\hat{\theta })$, i.e. the constrained maximum equals the global maximum, which happens iff some ${\theta }^{\ast }\in {\Theta }_0$ attains ${\sup }_{\Theta }L$. $\square$

Proposition 2 (exact GLRT for a normal mean, variance unknown, reduces to the $t$-statistic). Let $X_1,\dots ,X_n$ be i.i.d. $N(\mu ,{\sigma }^2)$ with both unknown, and test $H_0:\mu ={\mu }_0$. Then $\lambda (x)$ is a strictly decreasing function of $|T|$ where $T=\sqrt{n}(\bar{X}-{\mu }_0)/S$ and $S^2=\frac{1}{n-1}{\sum }_i(X_i-\bar{X}{)}^2$, and under $H_0$, $T\sim t_{n-1}$, so the GLRT is the two-sided $t$-test.

Proof. The unrestricted MLE is $(\bar{x},{\hat{\sigma }}^2)$ with ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(x_i-\bar{x}{)}^2$, giving $L(\hat{\theta })=(2\pi {\hat{\sigma }}^2{)}^{-n/2}{e}^{-n/2}$. Under $H_0$ the mean is fixed at ${\mu }_0$ and the variance MLE is ${\hat{\sigma }}_0^2=\frac{1}{n}{\sum }_i(x_i-{\mu }_0{)}^2$, giving $L({\hat{\theta }}_0)=(2\pi {\hat{\sigma }}_0^2{)}^{-n/2}{e}^{-n/2}$. The ratio is

$$\lambda ={\left(\frac{{\hat{\sigma }}^2}{{\hat{\sigma }}_0^2}\right)}^{n/2}={\left(\frac{{\sum }_i(x_i-\bar{x}{)}^2}{{\sum }_i(x_i-{\mu }_0{)}^2}\right)}^{n/2}.$$

Using ${\sum }_i(x_i-{\mu }_0{)}^2={\sum }_i(x_i-\bar{x}{)}^2+n(\bar{x}-{\mu }_0{)}^2$, write ${\lambda }^{2/n}=(1+n(\bar{x}-{\mu }_0{)}^2/{\sum }_i(x_i-\bar{x}{)}^2{)}^{-1}$. Since ${\sum }_i(x_i-\bar{x}{)}^2=(n-1)S^2$ and $T^2=n(\bar{x}-{\mu }_0{)}^2/S^2$, we get ${\lambda }^{2/n}=(1+T^2/(n-1){)}^{-1}$, a strictly decreasing function of $T^2$, hence of $|T|$. Therefore $\{\lambda \le c\}=\{|T|\ge t^{\ast }\}$. Under $H_0$, $\bar{X}\sim N({\mu }_0,{\sigma }^2/n)$ independent of $(n-1)S^2/{\sigma }^2\sim {\chi }_{n-1}^2$ (Cochran), so $T=\frac{\sqrt{n}(\bar{X}-{\mu }_0)/\sigma }{\sqrt{(n-1)S^2/({\sigma }^2(n-1))}}\sim t_{n-1}$. The GLRT is the exact two-sided $t$-test. $\square$

Proposition 3 (the score is centered and information-scaled under the null). Under the usual regularity conditions permitting differentiation under the integral, ${\mathbb{E}}_{{\theta }_0}[s({\theta }_0)]=0$ and ${\mathrm{Cov}}_{{\theta }_0}(s({\theta }_0))=I({\theta }_0)$, so the score statistic $R=s({\theta }_0{)}^{\top }I({\theta }_0{)}^{-1}s({\theta }_0)$ is a squared standardized score with null mean equal to its dimension.

Proof. For a single observation, $\int f(x\mid \theta )dx=1$ for all $\theta$; differentiating under the integral, $\int {\nabla }_{\theta }fdx=\int ({\nabla }_{\theta }\log f)fdx={\mathbb{E}}_{\theta }[{\nabla }_{\theta }\log f]=0$, the mean-zero score identity. Summing over the i.i.d. sample, ${\mathbb{E}}_{{\theta }_0}[s({\theta }_0)]=0$. Differentiating the identity ${\mathbb{E}}_{\theta }[\nabla \log f]=0$ once more gives $\int ({\nabla }^2\log f)f+(\nabla \log f)(\nabla \log f{)}^{\top }fdx=0$, i.e. ${\mathbb{E}}_{\theta }[(\nabla \log f)(\nabla \log f{)}^{\top }]=-{\mathbb{E}}_{\theta }[{\nabla }^2\log f]$, the two equivalent forms of the per-observation information; over the sample ${\mathrm{Cov}}_{{\theta }_0}(s({\theta }_0))=I({\theta }_0)$ 45.01.05. Thus $I({\theta }_0{)}^{-1/2}s({\theta }_0)$ has null mean $0$ and identity covariance, so $R=\Vert I({\theta }_0{)}^{-1/2}s({\theta }_0){\Vert }^2$ has ${\mathbb{E}}_{{\theta }_0}[R]=\mathrm{tr}(I({\theta }_0{)}^{-1}{\mathrm{Cov}}_{{\theta }_0}(s))=\mathrm{tr}(I_k)=k$, its dimension. $\square$

Proposition 4 (Pearson's $X^2$ is the multinomial score statistic). For the multinomial goodness-of-fit problem with null $\pi ={\pi }^0$ fully specified, the score statistic for $H_0:\pi ={\pi }^0$ equals Pearson's $X^2={\sum }_j(O_j-E_j{)}^2/E_j$ with $E_j=n{\pi }_j^0$.

Proof. Parametrize by $({\pi }_1,\dots ,{\pi }_{m-1})$ with ${\pi }_m=1-{\sum }_{j<m}{\pi }_j$. The log-likelihood is $\ell ={\sum }_j{O}_j\log {\pi }_j$, with score components $s_j=O_j/{\pi }_j-O_m/{\pi }_m$ for $j<m$. The Fisher information for the multinomial is $I_{jl}=n({\delta }_{jl}/{\pi }_j+1/{\pi }_m)$, with inverse $(I^{-1}{)}_{jl}=({\delta }_{jl}{\pi }_j-{\pi }_j{\pi }_l)/n$. A direct computation of $R=s^{\top }{I}^{-1}s$ at $\pi ={\pi }^0$ — substituting $s_j=O_j/{\pi }_j^0-O_m/{\pi }_m^0$ and the inverse — collapses, after using ${\sum }_j{O}_j=n$, to ${\sum }_{j=1}^m(O_j-n{\pi }_j^0{)}^2/(n{\pi }_j^0)$. Writing $E_j=n{\pi }_j^0$ gives $R={\sum }_j(O_j-E_j{)}^2/E_j=X^2$. The Lagrange-multiplier form makes the same computation transparent: imposing $\pi ={\pi }^0$ as a constraint, the multiplier is proportional to the standardized residual vector, whose squared information-norm is $X^2$. $\square$

Proposition 5 (exact unimodality of the variance GLRT region). The map $\varphi (r)=r^{n/2}{e}^{-n(r-1)/2}$ on $r>0$ is strictly increasing on $(0,1)$, strictly decreasing on $(1,\infty )$, with maximum $\varphi (1)=1$; consequently $\{\lambda \le c\}$ in the normal-variance problem is an interval complement $\{r\le r_1\}\cup \{r\ge r_2\}$ with $0<r_1<1<r_2$.

Proof. Take logarithms: $\log \varphi (r)=\frac{n}{2}\log r-\frac{n}{2}(r-1)$, so $\frac{d}{dr}\log \varphi =\frac{n}{2}(1/r-1)=\frac{n(1-r)}{2r}$, which is positive for $r<1$, zero at $r=1$, negative for $r>1$. Hence $\varphi$ strictly increases then strictly decreases, with a unique maximum $\varphi (1)=1$. For $c\in (0,1)$ the superlevel set $\{\varphi (r)>c\}$ is an open interval $(r_1,r_2)$ straddling $1$ (by strict unimodality and $\varphi (0^{+})=\varphi (\infty )=0$), so its complement $\{\varphi \le c\}=\{r\le r_1\}\cup \{r\ge r_2\}$. Since $r=t/n$ with $t=V/{\sigma }_0^2\sim {\chi }_{n-1}^2$ under $H_0$, the cutoffs translate to two-tailed ${\chi }_{n-1}^2$ critical values. $\square$

Connections Master

• The asymptotic theory of these three statistics 45.04.09 is the direct continuation: under regularity and an interior null of codimension $q$, $-2\log \lambda$, $W$, and $R$ each converge in null distribution to ${\chi }_q^2$ (Wilks' theorem) and are first-order equivalent, with the finite-sample ${\chi }_{n-1}^2$ and $t_{n-1}$ pivots of this unit's normal cases as the exact instances. That unit supplies the limit; this unit supplies the constructions and the exact special cases it specializes.

• The maximum-likelihood machinery 45.01.04 is the substrate every statistic is built on: the GLRT needs both the restricted and unrestricted MLEs, the Wald test needs the unrestricted MLE and the information at it, and the score test needs only the score at the null, which vanishes at the unrestricted MLE by the likelihood equations. The exponential-family moment-matching identity of that unit is what makes the normal, Poisson, and multinomial statistics here collapse to closed form.

• The Neyman-Pearson lemma 45.02.01 is the optimality benchmark the GLRT generalizes: the simple-vs-simple lemma thresholds the likelihood ratio $p_1/p_0$ and is most powerful, while the GLRT thresholds the maximized ratio ${\sup }_{{\Theta }_0}L/{\sup }_{\Theta }L$ for composite nested hypotheses, retaining the likelihood ratio as the privileged object but trading exact optimality for general applicability.

• Fisher information and the Cramér-Rao bound 45.01.05 provide the standardizing metric: the Wald statistic uses $I(\hat{\theta })$ and the score statistic uses $I({\theta }_0)$, the two information evaluations whose equivalence under the null (the mean-zero, information-covariance score identity) makes all three statistics asymptotically pivotal. The information's role as the Riemannian metric on the parameter space is what gives the trinity its distance-slope-ratio geometry.

Historical & philosophical context Master

The three constructions entered the literature within fifteen years of one another, each from a different motivation. The generalized likelihood-ratio principle was set out by Jerzy Neyman and Egon Pearson in 1928 as the natural extension of their most-powerful-test program to composite hypotheses, and its large-sample chi-squared null distribution was established by Samuel S. Wilks in 1938 ^{[Wilks 1938]} (Ann. Math. Statist. 9, 60-62), the result now bearing his name. Abraham Wald introduced the test based on the maximum-likelihood estimator and its asymptotic variance in 1943 ^{[Wald 1943]} (Trans. Amer. Math. Soc. 54, 426-482), as part of his broader decision-theoretic and large-sample program. C. R. Rao introduced the score test in 1948 ^{[Rao 1948]} (Proc. Cambridge Phil. Soc. 44, 50-57), motivated by the wish to test a hypothesis using only the fit under the null; the same statistic was rediscovered in econometrics as the Lagrange-multiplier test, where the score at the restricted estimate is read as the multiplier of the constraint.

The recognition that the three are first-order equivalent — distinct finite-sample statistics sharing one asymptotic null law — is due to the post-war large-sample theory codified by Erich Lehmann and later by Lehmann and Joseph Romano, whose Testing Statistical Hypotheses gives the modern treatment of the trinity and its equivalences ^{[Lehmann & Romano §12.4]}. The persistence of all three in practice reflects genuine differences: the likelihood-ratio statistic is reparametrization-invariant but needs two fits, the Wald statistic needs one fit at the alternative but is coordinate-dependent, and the score statistic needs one fit at the null and is the tool of choice when the alternative is hard to estimate. Pearson's chi-squared statistic of 1900, predating all three, turns out to be the multinomial score statistic, a historical convergence that Rao's 1948 framework made explicit.

Bibliography Master

@book{casellaberger2002statistical,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2nd},
  publisher = {Duxbury},
  year      = {2002}
}

@book{lehmannromano2005testing,
  author    = {Lehmann, Erich L. and Romano, Joseph P.},
  title     = {Testing Statistical Hypotheses},
  edition   = {3rd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {2005}
}

@article{wilks1938large,
  author  = {Wilks, Samuel S.},
  title   = {The large-sample distribution of the likelihood ratio for testing composite hypotheses},
  journal = {The Annals of Mathematical Statistics},
  volume  = {9},
  number  = {1},
  pages   = {60--62},
  year    = {1938}
}

@article{wald1943tests,
  author  = {Wald, Abraham},
  title   = {Tests of statistical hypotheses concerning several parameters when the number of observations is large},
  journal = {Transactions of the American Mathematical Society},
  volume  = {54},
  number  = {3},
  pages   = {426--482},
  year    = {1943}
}

@article{rao1948large,
  author  = {Rao, C. Radhakrishna},
  title   = {Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation},
  journal = {Proceedings of the Cambridge Philosophical Society},
  volume  = {44},
  number  = {1},
  pages   = {50--57},
  year    = {1948}
}

@article{neymanpearson1928use,
  author  = {Neyman, Jerzy and Pearson, Egon S.},
  title   = {On the use and interpretation of certain test criteria for purposes of statistical inference},
  journal = {Biometrika},
  volume  = {20A},
  pages   = {175--240, 263--294},
  year    = {1928}
}