Rao-Blackwell and Lehmann-Scheffé: UMVU Estimation

Intuition Beginner

You want the best possible unbiased guess at some unknown quantity — say the true rate of typos per page in a long manuscript. "Unbiased" means your recipe is right on average: across many manuscripts it neither systematically over- nor under-shoots. Among all such honest recipes, you want the one whose guesses scatter least around the truth. That is the best unbiased estimator.

Here is the surprising machine that builds it. Suppose you already have some unbiased guess — maybe a clumsy one, like "just use the count on the first page." Now hand all your data to a faithful summary that has squeezed out every drop of information about the rate, and ask: given only that summary, what would my clumsy guess have averaged to? Replacing the clumsy guess by this average can only help. It stays unbiased, and its scatter shrinks. This is the Rao-Blackwell idea: smoothing a rough estimator over a faithful summary never makes it worse, and usually makes it better.

The second step pins down the winner. If the faithful summary is also non-redundant — the completeness idea from the previous unit, meaning no function of it secretly averages to zero — then there is exactly one unbiased recipe built from it. One, not many. So the smoothed estimator is not merely improved; it is the unique best unbiased estimator. This is the Lehmann-Scheffé theorem. Together the two results give a clean recipe: take any honest guess, smooth it over a complete faithful summary, and you have the single best unbiased estimator there is.

Visual Beginner

Picture a dartboard whose centre is the true value. A rough unbiased estimator throws darts that land all around the board but average to the centre — honest, but widely scattered. Rao-Blackwellisation gathers each wild throw and slides it toward the centre by averaging over what the faithful summary already knows, tightening the cluster without shifting its centre off the bullseye.

Estimator	On target on average?	Scatter around the truth
Rough unbiased guess	yes	wide
Smoothed over the faithful summary	yes (unchanged)	narrower
Smoothed over a complete faithful summary	yes	narrowest possible — the unique best

The picture is the whole lesson: conditioning on a faithful summary pulls a scattered honest estimator inward without moving its centre, and when the summary is also non-redundant, the result is the single tightest honest estimator possible.

Worked example Beginner

A switchboard receives calls following a pattern with a single unknown average rate per minute. You record $n=4$ separate minutes and get the counts $3,1,4,2$. You want an unbiased estimate of the chance that a given minute has zero calls.

Step 1. A rough unbiased guess. Look only at the first minute: it had $3$ calls, which is not zero, so guess the zero-call chance is $0$. If the first minute had shown $0$ calls, you would have guessed $1$. This crude indicator — "was the first minute empty?" — is unbiased for the zero-call probability, because on average it equals exactly that probability. But it is wasteful: it ignores three minutes of data and only ever reports $0$ or $1$.

Step 2. The faithful summary. For this counting pattern the total count $T=3+1+4+2=10$ across all four minutes captures everything the data say about the rate.

Step 3. Smooth the rough guess over the total. Ask: given that the four minutes totalled $10$ calls, what is the chance the first minute was empty? A standard fact about this counting pattern is that, conditioned on the total $T$, each call independently landed in one of the four minutes with equal chance $1/4$. So the chance the first minute got none of the $10$ calls is $(3/4{)}^{10}$.

Step 4. Read off the improved estimate. Compute $(3/4{)}^{10}=0.0563$ to four places. That number, not the crude $0$, is the Rao-Blackwellised estimate of the zero-call probability.

What this tells us: the crude estimator only ever said $0$ or $1$; smoothing it over the faithful total turns it into a sensible fractional probability $0.0563$ that uses all four minutes, and it is provably no more scattered than the crude one. Because the total is also non-redundant for this pattern, $0.0563$ is in fact the single best unbiased estimate.

Check your understanding Beginner

Formal definition Intermediate+

Fix a measurable space $(\mathcal{X},\mathcal{A})$ and a statistical model $\mathcal{P}=\{P_{\theta }:\theta \in \Theta \}$ as in 45.01.02, with expectation ${\mathbb{E}}_{\theta }[\cdot ]$ the integral against $P_{\theta }$. An estimand is a function $\psi :\Theta \to \mathbb{R}$ (or ${\mathbb{R}}^p$); an estimator $W=W(X)$ is a statistic targeting it.

Definition (unbiasedness). An estimator $W$ is unbiased for $\psi$ if ${\mathbb{E}}_{\theta }[W]=\psi (\theta )$ for every $\theta \in \Theta$. The class of unbiased estimators of $\psi$ is denoted ${\mathcal{U}}_{\psi }$; the estimand $\psi$ is U-estimable if ${\mathcal{U}}_{\psi }\ne \varnothing$.

Definition (UMVU estimator). An unbiased estimator $W^{\star }\in {\mathcal{U}}_{\psi }$ is uniformly minimum-variance unbiased (UMVU) if, for every other $W\in {\mathcal{U}}_{\psi }$,

$${\mathrm{Var}}_{\theta }(W^{\star })\le {\mathrm{Var}}_{\theta }(W)\text{for all }\theta \in \Theta .$$

The word uniformly records that one estimator must dominate at every parameter value simultaneously — a strong demand, since in general no single estimator minimises variance pointwise across $\Theta$.

Definition (Rao-Blackwellisation). Let $T$ be a sufficient statistic for $\mathcal{P}$ in the sense of 45.01.02 and let $W$ be any estimator with ${\mathbb{E}}_{\theta }|W|<\infty$. The Rao-Blackwellisation of $W$ by $T$ is the conditional expectation

$$\eta (T):=\mathbb{E}[W\mid T].$$

Sufficiency is what makes this a genuine estimator: the conditional distribution of the data given $T$ does not depend on $\theta$, so a single version of $\mathbb{E}[W\mid T]$ may be chosen that is free of $\theta$ and hence computable without knowing the parameter. For a non-sufficient $T$ the conditional expectation would in general depend on $\theta$ and could not be used as an estimator.

The conditional variance is governed by the law of total variance: for any $W$ with finite second moment,

$${\mathrm{Var}}_{\theta }(W)={\mathrm{Var}}_{\theta }(\mathbb{E}[W\mid T])+{\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)],$$

with both summands non-negative — the identity from 37.01.01 and 26.03.01 that the Rao-Blackwell theorem turns into a variance inequality. Completeness of $T$ is taken from 45.01.03: ${\mathbb{E}}_{\theta }[g(T)]=0$ for all $\theta$ forces $g(T)=0$ a.s.

Counterexamples to common slips Intermediate+

• Conditioning on a non-sufficient statistic is not allowed. Rao-Blackwell requires $T$ sufficient. If $T$ is not sufficient, $\mathbb{E}[W\mid T]$ generally depends on $\theta$ and is not an estimator; conditioning on, say, $X_1$ alone in an i.i.d. problem does not produce a usable rule.

• Unbiasedness is not always achievable, and the UMVUE can be absurd. For a single $X\sim \mathrm{Poisson}(\lambda )$ the only unbiased estimator of $e^{-2\lambda }=P_{\lambda }(X=0{)}^2$ is $(-1{)}^X$, which is $-1$ on odd counts — an estimate of a probability that is negative. Unbiasedness alone does not guarantee sensibility.

• UMVU does not mean Cramér-Rao-attaining. A UMVU estimator is best among unbiased estimators, but it need not meet the information bound of 45.01.05; the bound is attained only by estimators affine in the score, a strictly smaller class.

• A function of a complete sufficient statistic that is unbiased is automatically UMVU. The temptation is to also check it against the Cramér-Rao bound; that is unnecessary. Lehmann-Scheffé certifies optimality directly from unbiasedness plus completeness, with no variance computation at all.

Key theorem with proof Intermediate+

The two organising results are the Rao-Blackwell theorem (conditioning on a sufficient statistic reduces variance) and the Lehmann-Scheffé theorem (a complete sufficient statistic makes the resulting unbiased estimator the unique UMVUE). We state both and prove them; the proofs are short and turn on the conditional-variance identity and on completeness respectively ^{[Casella, G. & Berger, R. L. — Statistical Inference, 2nd ed. (Duxbury, 2002)]}.

Theorem (Rao-Blackwell). Let $T$ be sufficient for $\mathcal{P}$ and let $W$ be an unbiased estimator of $\psi$ with ${\mathbb{E}}_{\theta }[W^2]<\infty$. Set $\eta (T)=\mathbb{E}[W\mid T]$. Then $\eta (T)$ is an estimator (free of $\theta$), it is unbiased for $\psi$, and

$${\mathrm{Var}}_{\theta }(\eta (T))\le {\mathrm{Var}}_{\theta }(W)\text{for all }\theta ,$$

with equality at a given $\theta$ if and only if $W=\eta (T)$ $P_{\theta }$-a.s.

Proof. Because $T$ is sufficient, the conditional law of the data given $T$ is parameter-free, so the version $\eta (T)=\mathbb{E}[W\mid T]$ may be taken free of $\theta$: it is a bona fide statistic. Unbiasedness is the tower property,

$${\mathbb{E}}_{\theta }[\eta (T)]={\mathbb{E}}_{\theta }[\mathbb{E}[W\mid T]]={\mathbb{E}}_{\theta }[W]=\psi (\theta )\text{for all }\theta ,$$

so $\eta (T)\in {\mathcal{U}}_{\psi }$. For the variance, apply the law of total variance to $W$:

$${\mathrm{Var}}_{\theta }(W)={\mathrm{Var}}_{\theta }(\mathbb{E}[W\mid T])+{\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)]={\mathrm{Var}}_{\theta }(\eta (T))+{\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)].$$

The remainder term ${\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)]$ is an expectation of a non-negative quantity, so ${\mathrm{Var}}_{\theta }(W)\ge {\mathrm{Var}}_{\theta }(\eta (T))$. Equality at $\theta$ holds exactly when ${\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)]=0$, i.e. $\mathrm{Var}(W\mid T)=0$ $P_{\theta }$-a.s., which says $W$ is a function of $T$, namely $W=\mathbb{E}[W\mid T]=\eta (T)$ a.s. $\square$

Theorem (Lehmann-Scheffé). Let $T$ be a sufficient statistic that is also complete for $\mathcal{P}$. If $\psi$ is U-estimable, there is a unique unbiased estimator of $\psi$ that is a function of $T$, and it is the UMVU estimator of $\psi$. It is obtained by Rao-Blackwellising any unbiased estimator: $\eta (T)=\mathbb{E}[W\mid T]$.

Proof. Existence: take any $W\in {\mathcal{U}}_{\psi }$ and form $\eta (T)=\mathbb{E}[W\mid T]$; by Rao-Blackwell it is an unbiased function of $T$. Uniqueness among functions of $T$: if ${\eta }_1(T)$ and ${\eta }_2(T)$ are both unbiased for $\psi$, then $g(T):={\eta }_1(T)-{\eta }_2(T)$ satisfies ${\mathbb{E}}_{\theta }[g(T)]=\psi (\theta )-\psi (\theta )=0$ for all $\theta$; completeness of $T$ forces $g(T)=0$ a.s., so ${\eta }_1(T)={\eta }_2(T)$ a.s. Optimality: let $W$ be any unbiased estimator of $\psi$. Its Rao-Blackwellisation $\mathbb{E}[W\mid T]$ is an unbiased function of $T$, hence — by the uniqueness just shown — equals $\eta (T)$ a.s. By the Rao-Blackwell inequality, ${\mathrm{Var}}_{\theta }(\eta (T))={\mathrm{Var}}_{\theta }(\mathbb{E}[W\mid T])\le {\mathrm{Var}}_{\theta }(W)$ for every $\theta$. Since $W$ was arbitrary in ${\mathcal{U}}_{\psi }$, $\eta (T)$ has minimum variance uniformly: it is the UMVU estimator. $\square$

Bridge. These two theorems build toward every concrete construction of an optimal unbiased estimator, and they appear again in the linear-model theory where the Gauss-Markov BLUE is the UMVUE under normality 45.01.07. The foundational reason the pair works is that sufficiency lets conditioning collapse a rough estimator onto a $\theta$-free function of $T$ while never inflating variance, and completeness is exactly the hypothesis that there is only one such function to collapse onto — so Rao-Blackwell supplies a candidate and completeness certifies it is the candidate. This is exactly the same completeness that powers Basu's theorem in 45.01.03: there it annihilates the difference $\psi (T)-p_C$ to force independence, here it annihilates the difference ${\eta }_1(T)-{\eta }_2(T)$ to force uniqueness, so Lehmann-Scheffé is dual to Basu, the two canonical uses of one property. The argument generalises from variance to any convex loss, because conditional Jensen replaces the conditional-variance identity, and putting these together, the central insight is that optimal unbiased estimation reduces to a single computation — solve ${\mathbb{E}}_{\theta }[\eta (T)]=\psi (\theta )$ in the complete sufficient statistic — with the bridge being that the natural statistic of a full-rank exponential family from 45.01.02 is the universal complete sufficient statistic, so the UMVUE always exists and is computable there.

Exercises Intermediate+

Advanced results Master

The Rao-Blackwell theorem holds for any convex loss, not merely squared error, and the Lehmann-Scheffé uniqueness rests on completeness alone; together they reduce optimal unbiased estimation to solving a single moment equation in a complete sufficient statistic, while the limitations of unbiasedness — non-existence, absurdity, and inadmissibility — mark the boundary where the programme yields to shrinkage.

The convex-loss Rao-Blackwell theorem. Let $L(\theta ,\cdot )$ be convex in its second argument and let $T$ be sufficient. For any estimator $W$ with finite risk, $\eta (T)=\mathbb{E}[W\mid T]$ satisfies $R(\theta ,\eta (T))\le R(\theta ,W)$ for every $\theta$, where $R(\theta ,\delta )={\mathbb{E}}_{\theta }[L(\theta ,\delta )]$ is the risk of 45.01.01. The proof replaces the conditional-variance identity by conditional Jensen: convexity gives $L(\theta ,\mathbb{E}[W\mid T])\le \mathbb{E}[L(\theta ,W)\mid T]$ pointwise, and taking ${\mathbb{E}}_{\theta }$ of both sides with the tower property yields $R(\theta ,\eta (T))\le R(\theta ,W)$. Squared error is the special case recovering the variance inequality. The theorem says sufficiency-conditioning is risk-improving for the entire convex-loss class, so any admissible estimator under convex loss may be taken to be a function of a sufficient statistic — the sufficiency reduction principle for decision rules ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation, 2nd ed. (Springer, 1998)]}.

The UMVUE as the projection of unbiased estimators of zero. Fix a complete sufficient $T$ and the Hilbert space $L^2(P_{\theta })$. The unbiased estimators of $\psi$ form an affine subspace $\eta (T)+{\mathcal{U}}_0$, where ${\mathcal{U}}_0=\{U:{\mathbb{E}}_{\theta }[U]=0\forall \theta \}$ is the linear space of unbiased estimators of zero. An unbiased $W$ is UMVU if and only if ${\mathrm{Cov}}_{\theta }(W,U)=0$ for every $U\in {\mathcal{U}}_0$ and every $\theta$ — geometrically, $W$ is the component orthogonal to all unbiased estimators of zero. Completeness of $T$ is precisely the statement that the only unbiased estimator of zero that is a function of $T$ is the zero function, so a function of a complete sufficient statistic is automatically orthogonal to ${\mathcal{U}}_0$, recovering Lehmann-Scheffé from the orthogonality criterion. This characterisation does not require sufficiency and gives UMVU estimators even in some models lacking a complete sufficient statistic ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation, 2nd ed. (Springer, 1998)]}.

Relation to the Cramér-Rao bound. A UMVUE is best among unbiased estimators but generally does not attain the information bound of 45.01.05. Equality in Cramér-Rao requires the estimator to be affine in the score, which by the attainment theorem holds only for the natural sufficient statistic of a one-parameter exponential family estimating its own mean. For other estimands the UMVUE exists (via Lehmann-Scheffé) but sits strictly above the Cramér-Rao floor. The UMVUE of $e^{-\lambda }$ in the Poisson model, $((n-1)/n{)}^{{\sum }_i{X}_i}$, is the unique best unbiased estimator yet its variance exceeds the information bound for that estimand; UMVU is the sharp notion of optimality, Cramér-Rao a frequently-loose lower bound for it. Where both apply and coincide, the UMVUE is also efficient, but the implication runs UMVU-then-possibly-efficient, never the reverse ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation, 2nd ed. (Springer, 1998)]}.

Limitations: non-existence, absurdity, inadmissibility. Three failure modes circumscribe unbiased estimation. First, U-estimability can fail: no unbiased estimator exists for, say, $1/\lambda$ in a Poisson model (any ${\mathbb{E}}_{\lambda }[W]={\sum }_{x}W(x){\lambda }^x{e}^{-\lambda }/x!$ is real-analytic in $\lambda$ at $0$, but $1/\lambda$ is not). Second, the UMVUE can be absurd, as $(-1{)}^X$ estimating a probability in Exercise 7. Third, and most consequential, the UMVUE can be inadmissible under squared-error loss. Stein's phenomenon: for $X\sim {\mathcal{N}}_p(\theta ,I_p)$ with $p\ge 3$, the sample mean $X$ — the UMVU (and maximum-likelihood) estimator of $\theta$ — is dominated in total mean-squared error by the James-Stein shrinkage estimator ${\hat{\theta }}_{JS}=(1-(p-2)/\Vert X{\Vert }^2)X$, which is biased but has uniformly smaller risk. Unbiasedness, the constraint that produced the UMVUE, is what makes it inadmissible: relaxing it buys a uniform risk reduction ^{[Stein, C. — Inadmissibility of the usual estimator for the mean of a multivariate normal distribution (Proc. 3rd Berkeley Symp., 1956)]}.

Synthesis. The foundational reason Rao-Blackwell and Lehmann-Scheffé cohere is that the first uses sufficiency to push any unbiased estimator down onto a $\theta$-free function of the sufficient statistic without raising risk, and the second uses completeness to certify that this function is unique — so existence and optimality are supplied by conditioning and uniqueness by completeness. This is exactly the content of the orthogonal decomposition $L^2(P_{\theta })=(\text{functions of }T)\oplus {\mathcal{U}}_0$ at a complete sufficient $T$: Rao-Blackwell is the projection, completeness is the statement that the function-of-$T$ summand meets ${\mathcal{U}}_0$ only at zero, and the UMVUE is the resulting orthogonal component. Lehmann-Scheffé is dual to Basu's theorem from 45.01.03 — both annihilate a difference of functions of $T$ by completeness, Basu the difference $\psi (T)-p_C$ to force independence, Lehmann-Scheffé the difference ${\eta }_1(T)-{\eta }_2(T)$ to force uniqueness — so the completeness of the full-rank exponential family of 45.01.02 is the single fact that makes both work in every standard model. The central insight is that optimal unbiased estimation is a moment equation solved in a complete sufficient statistic, and putting these together, its limits — non-existence, the absurd $(-1{)}^X$, and Stein inadmissibility for $p\ge 3$ — are not failures of the theorems but of the unbiasedness constraint itself, the bridge to admissible shrinkage estimation in 45.01.01, where dropping unbiasedness purchases uniformly smaller risk.

Full proof set Master

The Rao-Blackwell and Lehmann-Scheffé theorems are proved in the Key theorem. The remaining Master claims are recorded here.

Proposition 1 (convex-loss Rao-Blackwell). Let $L(\theta ,\cdot )$ be convex, $T$ sufficient, and $W$ an estimator of finite risk. Then $\eta (T)=\mathbb{E}[W\mid T]$ has $R(\theta ,\eta (T))\le R(\theta ,W)$ for all $\theta$.

Proof. Sufficiency makes $\eta (T)$ a $\theta$-free statistic. Conditional Jensen applied to the convex map $\delta \mapsto L(\theta ,\delta )$ gives, $P_{\theta }$-a.s.,

$$L(\theta ,\mathbb{E}[W\mid T])\le \mathbb{E}[L(\theta ,W)|T].$$

Take ${\mathbb{E}}_{\theta }$ of both sides; the tower property collapses the right-hand side:

$$R(\theta ,\eta (T))={\mathbb{E}}_{\theta }[L(\theta ,\eta (T))]\le {\mathbb{E}}_{\theta }\left[\mathbb{E}\right[L(\theta ,W)\mid T\left]\right]={\mathbb{E}}_{\theta }[L(\theta ,W)]=R(\theta ,W).$$

For $L(\theta ,\delta )=(\delta -\psi (\theta ){)}^2$ the inequality is the variance inequality, with the gap equal to ${\mathbb{E}}_{\theta }[\mathrm{Var}(W\mid T)]$. $\square$

Proposition 2 (orthogonality characterisation of UMVU). An unbiased estimator $W$ of $\psi$ with finite variance is UMVU if and only if ${\mathrm{Cov}}_{\theta }(W,U)=0$ for every $U\in {\mathcal{U}}_0$ and every $\theta$.

Proof. ($\Leftarrow$) Let $W^{\prime }$ be any other unbiased estimator of $\psi$. Then $U:=W^{\prime }-W\in {\mathcal{U}}_0$, and

$${\mathrm{Var}}_{\theta }(W^{\prime })={\mathrm{Var}}_{\theta }(W+U)={\mathrm{Var}}_{\theta }(W)+2{\mathrm{Cov}}_{\theta }(W,U)+{\mathrm{Var}}_{\theta }(U)={\mathrm{Var}}_{\theta }(W)+{\mathrm{Var}}_{\theta }(U)\ge {\mathrm{Var}}_{\theta }(W),$$

using ${\mathrm{Cov}}_{\theta }(W,U)=0$. So $W$ is UMVU. ($\Rightarrow$) Suppose ${\mathrm{Cov}}_{{\theta }_0}(W,U)\ne 0$ for some $U\in {\mathcal{U}}_0$ and some ${\theta }_0$. For real $c$, $W_c:=W+cU$ is unbiased for $\psi$ and

$${\mathrm{Var}}_{{\theta }_0}(W_c)={\mathrm{Var}}_{{\theta }_0}(W)+2c{\mathrm{Cov}}_{{\theta }_0}(W,U)+c^2{\mathrm{Var}}_{{\theta }_0}(U).$$

Choosing $c$ of small magnitude with sign opposite to ${\mathrm{Cov}}_{{\theta }_0}(W,U)$ makes the linear term dominate the quadratic term, giving ${\mathrm{Var}}_{{\theta }_0}(W_c)<{\mathrm{Var}}_{{\theta }_0}(W)$, contradicting UMVU. $\square$

Proposition 3 (Lehmann-Scheffé from completeness, via orthogonality). If $T$ is complete sufficient and $W=\eta (T)$ is an unbiased function of $T$, then $W$ is UMVU.

Proof. By Proposition 2 it suffices to show ${\mathrm{Cov}}_{\theta }(\eta (T),U)=0$ for every $U\in {\mathcal{U}}_0$. Fix $U\in {\mathcal{U}}_0$ and set $h(T):=\mathbb{E}[U\mid T]$, a function of $T$, $\theta$-free by sufficiency. Then ${\mathbb{E}}_{\theta }[h(T)]={\mathbb{E}}_{\theta }[U]=0$ for all $\theta$, so completeness forces $h(T)=0$ a.s. Hence, using ${\mathbb{E}}_{\theta }[\eta (T)]=\psi (\theta )$ and ${\mathbb{E}}_{\theta }[U]=0$,

$${\mathrm{Cov}}_{\theta }(\eta (T),U)={\mathbb{E}}_{\theta }[\eta (T)U]={\mathbb{E}}_{\theta }[\eta (T)\mathbb{E}[U\mid T]]={\mathbb{E}}_{\theta }[\eta (T)h(T)]=0,$$

the middle step by the tower property since $\eta (T)$ is $T$-measurable. So $\eta (T)$ is orthogonal to ${\mathcal{U}}_0$ and is UMVU. $\square$

Proposition 4 (non-existence of an unbiased estimator). For a single $X\sim \mathrm{Poisson}(\lambda )$, no estimator is unbiased for $\psi (\lambda )=1/\lambda$.

Proof. Suppose ${\mathbb{E}}_{\lambda }[W(X)]=1/\lambda$ for all $\lambda >0$. The left side is ${\sum }_{x\ge 0}W(x){\lambda }^x{e}^{-\lambda }/x!$, so ${\sum }_{x\ge 0}\frac{W(x)}{x!}{\lambda }^x=e^{\lambda }/\lambda$. The left side is a power series convergent for all $\lambda$, hence its sum is finite as $\lambda \to 0^{+}$; but $e^{\lambda }/\lambda \to +\infty$ as $\lambda \to 0^{+}$. The contradiction shows no such $W$ exists. $\square$

Proposition 5 (inadmissibility of the UMVUE — Stein). For $X\sim {\mathcal{N}}_p(\theta ,I_p)$ with $p\ge 3$, the UMVU estimator ${\delta }_0(X)=X$ of $\theta$ is inadmissible under squared-error loss $\Vert \delta -\theta {\Vert }^2$.

Proof. It suffices to exhibit a dominating estimator. Consider ${\delta }_c(X)=(1-c/\Vert X{\Vert }^2)X$. Stein's unbiased risk estimate (an integration-by-parts identity for the Gaussian) gives, for $\delta (X)=X+g(X)$ with $g$ weakly differentiable and suitably integrable,

$$R(\theta ,\delta )={\mathbb{E}}_{\theta }\Vert \delta -\theta {\Vert }^2=p+{\mathbb{E}}_{\theta }[\Vert g(X){\Vert }^2+2\nabla \cdot g(X)].$$

For $g(X)=-cX/\Vert X{\Vert }^2$ one computes $\Vert g{\Vert }^2=c^2/\Vert X{\Vert }^2$ and $\nabla \cdot g=-c(p-2)/\Vert X{\Vert }^2$, so

$$R(\theta ,{\delta }_c)=p+{\mathbb{E}}_{\theta }\left[\frac{c^2-2c(p-2)}{\Vert X{\Vert }^2}\right]=p-{\mathbb{E}}_{\theta }\left[\frac{2c(p-2)-c^2}{\Vert X{\Vert }^2}\right].$$

For $p\ge 3$ the factor $2c(p-2)-c^2$ is positive for $0<c<2(p-2)$ and ${\mathbb{E}}_{\theta }[\Vert X{\Vert }^{-2}]<\infty$, so $R(\theta ,{\delta }_c)<p=R(\theta ,{\delta }_0)$ for every $\theta$. The choice $c=p-2$ maximises the reduction, giving the James-Stein estimator. Thus ${\delta }_0=X$ is dominated and inadmissible. $\square$

Connections Master

Sufficiency and exponential families 45.01.02 are the direct upstream: Rao-Blackwell conditions on the sufficient statistic defined there, and the completeness of the natural statistic of a full-rank exponential family is what makes Lehmann-Scheffé applicable in every standard model, so the factorization theorem and the convexity-and-analyticity of the log-partition function are the same facts that supply the complete sufficient statistics on which UMVU estimation runs.

Ancillarity, completeness, and Basu's theorem 45.01.03 share the load-bearing hypothesis with Lehmann-Scheffé: completeness annihilates a difference of two functions of $T$ in both, killing ${\eta }_1(T)-{\eta }_2(T)$ here to force uniqueness of the UMVUE and killing $\psi (T)-p_C$ there to force independence of a complete sufficient statistic from any ancillary; Basu is moreover a computational engine for UMVU variances, as in the independence of $\bar{X}$ and $S^2$ that legitimises the normal-mean constructions.

Fisher information and the Cramér-Rao bound 45.01.05 are the comparison standard: a UMVUE is best among unbiased estimators but need not attain the information bound, since equality there requires affineness in the score, available only on the natural statistic of a one-parameter exponential family; where the two coincide the UMVUE is efficient, but UMVU is the sharper notion and the Cramér-Rao floor is frequently a strict under-bound for its variance.

Decision theory, risk, and admissibility 45.01.01 is where the limitations land: the convex-loss Rao-Blackwell theorem is the sufficiency-reduction principle for decision rules, and the inadmissibility of the UMVU multivariate normal mean under squared-error loss for dimension three and up — Stein's phenomenon — shows that the unbiasedness constraint producing the UMVUE is exactly what costs it admissibility, motivating the biased shrinkage estimators that dominate it in risk.

The Gauss-Markov theorem and the linear model 45.01.07 specialise the UMVU programme to linear estimands: under normal errors the best linear unbiased estimator coincides with the UMVUE, the least-squares estimator being an unbiased function of the complete sufficient statistic of the Gaussian linear model, so Lehmann-Scheffé upgrades the within-linear-class optimality of Gauss-Markov to optimality among all unbiased estimators.

Historical & philosophical context Master

The conditioning construction that reduces an estimator's variance is due to C. Radhakrishna Rao, in his 1945 paper in the Bulletin of the Calcutta Mathematical Society — the same paper that gave the information inequality — and independently to David Blackwell in a 1947 Annals of Mathematical Statistics note, whence the joint name ^{[Rao 1945; Blackwell 1947]}. Rao framed it through sufficiency and the recovery of information; Blackwell gave the convex-loss version with the conditional-Jensen argument that the modern statement uses.

Completeness and the uniqueness of the best unbiased estimator are due to Erich Lehmann and Henry Scheffé in their two-part 1950-1955 Sankhyā papers on completeness, similar regions, and unbiased estimation ^{[Lehmann Scheffé 1950]}, where completeness was introduced precisely to secure that a function of a complete sufficient statistic is the unique unbiased estimator. The synthesis of Rao-Blackwell improvement with Lehmann-Scheffé uniqueness into the standard UMVU recipe was consolidated by Lehmann in Theory of Point Estimation ^{[Lehmann Casella 1998]}.

The limits of the programme were exposed by Charles Stein, who proved in 1956 that the usual estimator of a multivariate normal mean is inadmissible under squared-error loss in dimension three and higher ^{[Stein 1956]}; the explicit dominating estimator was given by Willard James and Stein in 1961. Stein's result fixed the boundary of unbiased estimation: the constraint of unbiasedness, which Rao-Blackwell and Lehmann-Scheffé optimise within, is itself the source of the inadmissibility that shrinkage estimation repairs.

Bibliography Master

@article{rao1945information,
  author  = {Rao, C. Radhakrishna},
  title   = {Information and the accuracy attainable in the estimation of statistical parameters},
  journal = {Bulletin of the Calcutta Mathematical Society},
  volume  = {37},
  pages   = {81--91},
  year    = {1945}
}

@article{blackwell1947conditional,
  author  = {Blackwell, David},
  title   = {Conditional expectation and unbiased sequential estimation},
  journal = {Annals of Mathematical Statistics},
  volume  = {18},
  number  = {1},
  pages   = {105--110},
  year    = {1947}
}

@article{lehmannscheffe1950completeness,
  author  = {Lehmann, Erich L. and Scheff\'e, Henry},
  title   = {Completeness, similar regions, and unbiased estimation. Part I},
  journal = {Sankhy\={a}},
  volume  = {10},
  number  = {4},
  pages   = {305--340},
  year    = {1950}
}

@book{lehmanncasella1998theory,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2nd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}

@book{casellaberger2002statistical,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2nd},
  publisher = {Duxbury},
  year      = {2002}
}

@inproceedings{stein1956inadmissibility,
  author    = {Stein, Charles},
  title     = {Inadmissibility of the usual estimator for the mean of a multivariate normal distribution},
  booktitle = {Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability},
  volume    = {1},
  pages     = {197--206},
  publisher = {University of California Press},
  year      = {1956}
}

@article{jamesstein1961estimation,
  author  = {James, Willard and Stein, Charles},
  title   = {Estimation with quadratic loss},
  journal = {Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability},
  volume  = {1},
  pages   = {361--379},
  year    = {1961}
}