The James-Stein Estimator and Inadmissibility of the MLE

Intuition Beginner

Suppose you must estimate several unrelated quantities at once — the true batting averages of ten different players, say, each measured with its own noisy sample. The obvious move is to estimate each one by its own data and never let one player's numbers touch another's. They are unrelated, so why would pooling help? This instinct is correct in spirit and wrong in fact: when you are juggling three or more quantities at once and you score yourself by total squared error, you can always do better by pulling every estimate a little toward a common center.

That sounds like a paradox, and it earned the name. The quantities really are unrelated, yet the combined guess improves when you shrink them together. The reason is not that the players secretly influence each other. It is that the noise in your measurements does influence the total error, and a coordinated shrink cancels more noise than it adds bias. You give up a tiny amount of accuracy on each one to buy a larger reduction in the wild swings that come from estimating many things from limited data.

The everyday image is a panel of slightly miscalibrated thermometers. Read each on its own and the readings scatter. But if you know they should all be near room temperature, nudging every reading toward the room average corrects more error than it introduces. The shrink is a bet that extreme readings are partly luck, and across enough readings that bet pays.

The surprise is the number three. With one or two quantities, the plain data-only estimate cannot be beaten this way. From three upward, it always can. Dimension itself, not any link between the quantities, is what opens the door.

Visual Beginner

Setting	Plain estimate (each on its own)	Shrunk estimate (pulled toward center)	Who wins on total squared error
1 quantity	the data value	the data value	tie — plain cannot be beaten
2 quantities	the two data values	the two data values	tie — plain cannot be beaten
3 quantities	three data values	each pulled toward the average	shrunk wins, for every true setting
10 quantities	ten data values	each pulled toward the average	shrunk wins, often by a lot
many quantities	the data values	the data values, gently pulled in	shrunk wins, and the gap grows

The single picture to carry forward: the data points spray outward from the truth because of noise, so on average they land too far out. Pulling them all back toward a common center corrects that outward bias in the cloud as a whole, and once there are three or more directions to average over, the correction reliably beats leaving the data alone.

Worked example Beginner

Estimate three unrelated quantities. The truth is $\theta =(0,0,0)$ — though you will not know that — and each measurement has noise of variance $1$. You observe $X=(2,1,-2)$. The plain estimate just reports the data, ${\hat{\theta }}_{\text{plain}}=(2,1,-2)$.

Step 1. Score the plain estimate. Its total squared error is the squared distance from the truth: $2^2+1^2+(-2{)}^2=4+1+4=9$.

Step 2. Build the shrunk estimate. The James-Stein recipe multiplies the whole vector by a factor $1-\frac{p-2}{\Vert X{\Vert }^2}$, where $p=3$ is the number of quantities and $\Vert X{\Vert }^2=9$ is the squared length of the data vector. The factor is $1-\frac{3-2}{9}=1-\frac{1}{9}=\frac{8}{9}$.

Step 3. Apply the factor. Multiply each coordinate by $8/9$: the shrunk estimate is $(\frac{16}{9},\frac{8}{9},-\frac{16}{9})\approx (1.78,0.89,-1.78)$.

Step 4. Score the shrunk estimate. Its squared distance from the truth $(0,0,0)$ is $(\frac{8}{9}{)}^2\times 9=\frac{64}{81}\times 9=\frac{64}{9}\approx 7.11$.

What this tells us. On this one dataset the shrunk estimate scored $7.11$ against the plain estimate's $9$ — a clear improvement. One dataset is luck, but the theorem promises that averaged over all possible noisy measurements, the shrunk estimate's total error is smaller no matter what the true $\theta$ is, as long as there are three or more quantities. The factor $8/9$ is close to $1$, so the pull is gentle; it grows when the data vector is short (likely near the center) and fades when the data vector is long.

Check your understanding Beginner

Formal definition Intermediate+

Fix dimension $p$ and observe a single vector $X\sim N(\theta ,{\sigma }^2{I}_p)$ with $\theta \in {\mathbb{R}}^p$ unknown and ${\sigma }^2$ known; the multivariate-normal mean model is the one introduced in 26.03.01. The loss is summed (total) squared error $L(\theta ,a)=\Vert a-\theta {\Vert }^2={\sum }_{i=1}^p(a_i-{\theta }_i{)}^2$, and the risk of an estimator $\delta$ is $R(\theta ,\delta )={\mathbb{E}}_{\theta }\Vert \delta (X)-\theta {\Vert }^2$, the decision-theoretic risk of 45.01.01.

The maximum-likelihood estimator (equivalently the sample mean, the uniformly-minimum-variance unbiased estimator, and the best invariant estimator) is ${\delta }_0(X)=X$. Its risk is constant:

$$R(\theta ,{\delta }_0)={\mathbb{E}}_{\theta }\Vert X-\theta {\Vert }^2=\sum _{i=1}^p{\mathbb{E}}_{\theta }(X_i-{\theta }_i{)}^2=p{\sigma }^2.$$

The James-Stein estimator shrinks $X$ multiplicatively toward the origin:

$${\delta }_{\mathrm{JS}}(X)=\left(1-\frac{(p-2){\sigma }^2}{\Vert X{\Vert }^2}\right)X.$$

Shrinkage toward an arbitrary fixed point $\nu \in {\mathbb{R}}^p$ replaces $X$ by $X-\nu$ inside the factor and adds $\nu$ back: ${\delta }_{\mathrm{JS}}^{\nu }(X)=\nu +(1-(p-2){\sigma }^2/\Vert X-\nu {\Vert }^2)(X-\nu )$; nothing structural changes, so the development below takes $\nu =0$. The shrinkage factor is data-adaptive — close to $1$ when $\Vert X\Vert$ is large (the data look far from the origin and are left nearly alone) and small, even negative, when $\Vert X\Vert$ is small. Its negativity is the defect repaired by the positive-part James-Stein estimator

$${\delta }_{\mathrm{JS}}^{+}(X)={\left(1-\frac{(p-2){\sigma }^2}{\Vert X{\Vert }^2}\right)}^{+}X,(t{)}^{+}=\max (t,0),$$

which truncates the factor at zero rather than letting it flip the sign of $X$.

An estimator $\delta$ is inadmissible under this loss if some ${\delta }^{\prime }$ has $R(\theta ,{\delta }^{\prime })\le R(\theta ,\delta )$ for all $\theta$ with strict inequality somewhere, in the sense of 45.01.01. The content of this unit is that for $p\ge 3$ the estimator ${\delta }_0(X)=X$ is inadmissible: ${\delta }_{\mathrm{JS}}$ dominates it. The tool that proves this is Stein's unbiased risk estimate, built from Stein's lemma, the Gaussian integration-by-parts identity stated next.

Counterexamples to common slips

• The shrinkage factor is not the ridge factor. Ridge 45.06.03 multiplies each principal coordinate by a fixed $d_j^2/(d_j^2+\lambda )$ chosen before seeing the response; James-Stein multiplies the whole vector by a single data-dependent factor $1-(p-2){\sigma }^2/\Vert X{\Vert }^2$. The first is linear in $X$; the second is not.
• Dominance is global, not pointwise on one dataset. The James-Stein estimate can be worse than $X$ on a particular sample (its factor can overshoot). Dominance is a statement about risk — expected loss — holding for every $\theta$ simultaneously, not about every realisation.
• The threshold is $p\ge 3$, not $p\ge 1$. For $p=1$ and $p=2$ the sample mean is admissible; the moment ${\mathbb{E}}_{\theta }[1/\Vert X{\Vert }^2]$ that the proof needs is finite only when $p\ge 3$, and the factor $(p-2)$ vanishes at $p=2$.
• Known ${\sigma }^2$ is a convenience, not the essence. When ${\sigma }^2$ is unknown it is replaced by an independent estimate ${\hat{\sigma }}^2$ (from a chi-squared sum) with the constant $(p-2)$ adjusted accordingly; the dominance survives. The clean statement takes ${\sigma }^2$ known.

Key theorem with proof Intermediate+

Theorem (James-Stein dominance; James & Stein 1961). Let $X\sim N(\theta ,{\sigma }^2{I}_p)$ with ${\sigma }^2$ known and $p\ge 3$. Under summed squared-error loss the James-Stein estimator satisfies, for every $\theta$,

$$R(\theta ,{\delta }_{\mathrm{JS}})=p{\sigma }^2-(p-2{)}^2{\sigma }^4{\mathbb{E}}_{\theta }\left[\frac{1}{\Vert X{\Vert }^2}\right]<p{\sigma }^2=R(\theta ,{\delta }_0).$$

Hence the maximum-likelihood estimator ${\delta }_0(X)=X$ is inadmissible. The proof follows the unbiased-risk presentation of Lehmann & Casella ^{[Lehmann & Casella §5.5]} and Stein's identity ^{[Stein 1981]}.

Proof. The engine is the integration-by-parts identity for the normal.

Stein's lemma. Let $Z\sim N(\mu ,{\sigma }^2)$ scalar and let $h:\mathbb{R}\to \mathbb{R}$ be absolutely continuous with $\mathbb{E}|h^{\prime }(Z)|<\infty$. Then $\mathbb{E}[(Z-\mu )h(Z)]={\sigma }^2\mathbb{E}[h^{\prime }(Z)]$. To see it, write the expectation against the density ${\phi }_{\sigma }(z-\mu )=(2\pi {\sigma }^2{)}^{-1/2}\exp (-(z-\mu {)}^2/2{\sigma }^2)$, whose derivative satisfies ${\phi }_{\sigma }^{\prime }(z-\mu )=-\frac{z-\mu }{{\sigma }^2}{\phi }_{\sigma }(z-\mu )$. Then

$$\mathbb{E}[(Z-\mu )h(Z)]=\int (z-\mu )h(z){\phi }_{\sigma }(z-\mu )dz=-{\sigma }^2\int h(z){\phi }_{\sigma }^{\prime }(z-\mu )dz={\sigma }^2\int h^{\prime }(z){\phi }_{\sigma }(z-\mu )dz,$$

the last step integrating by parts, the boundary terms vanishing because ${\phi }_{\sigma }$ decays faster than any polynomial. The right side is ${\sigma }^2\mathbb{E}[h^{\prime }(Z)]$. Applied coordinatewise to $X\sim N(\theta ,{\sigma }^2{I}_p)$, for $h:{\mathbb{R}}^p\to \mathbb{R}$ suitably differentiable,

$${\mathbb{E}}_{\theta }[(X_i-{\theta }_i)h(X)]={\sigma }^2{\mathbb{E}}_{\theta }\left[\frac{\partial h}{\partial x_i}(X)\right].$$

Stein's unbiased risk estimate (SURE). Write the estimator as $\delta (X)=X+g(X)$ for a weakly differentiable $g:{\mathbb{R}}^p\to {\mathbb{R}}^p$. Expanding the loss,

$$\Vert \delta (X)-\theta {\Vert }^2=\Vert (X-\theta )+g(X){\Vert }^2=\Vert X-\theta {\Vert }^2+2\sum _{i=1}^p(X_i-{\theta }_i)g_i(X)+\Vert g(X){\Vert }^2.$$

Take ${\mathbb{E}}_{\theta }$. The first term gives $p{\sigma }^2$. For each cross term apply Stein's lemma with $h=g_i$: ${\mathbb{E}}_{\theta }[(X_i-{\theta }_i)g_i(X)]={\sigma }^2{\mathbb{E}}_{\theta }[\partial g_i/\partial x_i]$. Summing,

$$R(\theta ,\delta )={\mathbb{E}}_{\theta }\left[p{\sigma }^2+2{\sigma }^2\nabla \cdot g(X)+\Vert g(X){\Vert }^2\right],\nabla \cdot g=\sum _{i=1}^p\frac{\partial g_i}{\partial x_i}.$$

The bracketed quantity is an unbiased estimator of the risk computed from the data alone — no $\theta$ appears in it.

Applying SURE to James-Stein. Here $g(X)=-\frac{(p-2){\sigma }^2}{\Vert X{\Vert }^2}X$, so $g_i(X)=-(p-2){\sigma }^2{X}_i/\Vert X{\Vert }^2$. Differentiate:

$$\frac{\partial g_i}{\partial x_i}=-(p-2){\sigma }^2\left(\frac{1}{\Vert X{\Vert }^2}-\frac{2X_i^2}{\Vert X{\Vert }^4}\right),\nabla \cdot g=-(p-2){\sigma }^2\left(\frac{p}{\Vert X{\Vert }^2}-\frac{2}{\Vert X{\Vert }^2}\right)=-\frac{(p-2{)}^2{\sigma }^2}{\Vert X{\Vert }^2}.$$

And $\Vert g(X){\Vert }^2=(p-2{)}^2{\sigma }^4\Vert X{\Vert }^2/\Vert X{\Vert }^4=(p-2{)}^2{\sigma }^4/\Vert X{\Vert }^2$. Substituting into SURE,

$$R(\theta ,{\delta }_{\mathrm{JS}})=p{\sigma }^2+{\mathbb{E}}_{\theta }\left[-\frac{2(p-2{)}^2{\sigma }^4}{\Vert X{\Vert }^2}+\frac{(p-2{)}^2{\sigma }^4}{\Vert X{\Vert }^2}\right]=p{\sigma }^2-(p-2{)}^2{\sigma }^4{\mathbb{E}}_{\theta }\left[\frac{1}{\Vert X{\Vert }^2}\right].$$

For $p\ge 3$ the random variable $\Vert X{\Vert }^2$ is almost surely positive and ${\mathbb{E}}_{\theta }[1/\Vert X{\Vert }^2]<\infty$ (it is the expectation of the reciprocal of a noncentral chi-squared with $p\ge 3$ degrees of freedom, finite precisely because $p-2>0$), and it is strictly positive. The subtracted term is therefore strictly positive, so $R(\theta ,{\delta }_{\mathrm{JS}})<p{\sigma }^2$ for every $\theta$. The maximum-likelihood estimator is dominated, hence inadmissible. $\square$

Bridge. The computation's engine is Stein's lemma, which converts the unknown-$\theta$ cross term ${\mathbb{E}}_{\theta }[(X-\theta )\cdot g(X)]$ into a divergence ${\sigma }^2{\mathbb{E}}_{\theta }[\nabla \cdot g]$ that the data estimate without knowing $\theta$; this builds toward the SURE-based construction of the whole shrinkage family in the Advanced section, and the identity appears again in 45.03.03 as the empirical-Bayes update where the same divergence term is the bias correction of the estimated prior variance. The foundational reason $X$ can be beaten is that summed squared-error loss couples the coordinates: each coordinate's estimate is individually unbiased and admissible, yet the total risk has slack that a joint contraction removes — this is exactly the multivariate promotion of the single-coordinate bias-variance bargain of 45.06.03, where a controlled bias bought a variance reduction. The result is dual to the ridge story: ridge contracts along the design's principal directions with a fixed factor, James-Stein contracts the whole vector with a data-chosen factor, and putting these together shows shrinkage toward a point is not a heuristic but a theorem about the geometry of the sphere $\Vert X{\Vert }^2$. The central insight is that the dimension $p$ enters through the divergence: $\nabla \cdot g$ accumulates $p$ derivative terms while the squared-norm penalty $\Vert g{\Vert }^2$ does not, so for $p\ge 3$ the gain $(p-2{)}^2$ outruns the cost.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not yet host the decision-theoretic risk object or Stein's integration-by-parts identity in the form the theorem needs, so no module is wired in (lean_status: none). The intended statements, once a Gaussian-integration-by-parts API and a risk functional exist, would read roughly as follows.

-- Intended shape; not part of the current Babel Bible Lean build.
-- Requires: the isotropic Gaussian measure N(θ, σ² I), a risk functional, and Stein's lemma.
variable {p : ℕ} (σ : ℝ) (hσ : 0 < σ) (θ : EuclideanSpace ℝ (Fin p))

-- Stein's lemma (coordinatewise integration by parts for the normal).
theorem stein_identity (h : EuclideanSpace ℝ (Fin p) → ℝ) (i : Fin p)
    (hdiff : Differentiable ℝ h) (hint : Integrable (fun x => |fderiv ℝ h x|)) :
    ∫ x, (x i - θ i) * h x ∂(gaussian θ (σ^2)) =
      σ^2 * ∫ x, (fderiv ℝ h x) (EuclideanSpace.single i 1) ∂(gaussian θ (σ^2)) := by
  sorry

-- James-Stein dominance: for p ≥ 3 the JS risk is strictly below p σ².
noncomputable def jamesStein (x : EuclideanSpace ℝ (Fin p)) : EuclideanSpace ℝ (Fin p) :=
  (1 - (p - 2) * σ^2 / ‖x‖^2) • x

theorem jamesStein_dominates_mle (hp : 3 ≤ p) :
    riskSquaredError (gaussian · (σ^2)) (jamesStein σ) θ
      < (p : ℝ) * σ^2 := by
  sorry

the Mathlib gap analysis records what is missing: the isotropic-normal mean model as a parametric family, the risk functional riskSquaredError, Stein's lemma as Gaussian integration by parts, the SURE identity R = p σ² + E[2 σ² div g + ‖g‖²], and the inverse-chi-squared moment E[1/‖X‖²] = 1/((p-2)σ²) at the origin that the dimension threshold turns on.

Advanced results Master

The unbiased-risk identity is more than a proof device; it is a constructive principle. Stein's unbiased risk estimate $\hat{R}(X)=p{\sigma }^2+2{\sigma }^2\nabla \cdot g(X)+\Vert g(X){\Vert }^2$ is a statistic, computable from the data without $\theta$, whose expectation is the risk of $\delta =X+g$. Minimising $\hat{R}$ over a parametric family of perturbations $g$ — for example $g_c=-c{\sigma }^{2}X/\Vert X{\Vert }^2$ — selects a shrinkage rule with provably small risk, and the same machine certifies wavelet thresholding, smoothing-spline tuning, and the choice of the ridge parameter from data. SURE-tuned estimators are the modern descendants of James-Stein: the unbiased risk is the surrogate one optimises when cross-validation is too expensive, and its expectation property guarantees the surrogate tracks the true risk.

The empirical-Bayes reading of Efron and Morris ^{[Efron & Morris 1973]} places James-Stein inside a coherent hierarchical model and explains why the shrinkage is the right amount. Under ${\theta }_i\sim N(0,{\tau }^2)$ and $X_i\mid {\theta }_i\sim N({\theta }_i,{\sigma }^2)$, the Bayes estimator is $BX$ with $B={\tau }^2/({\sigma }^2+{\tau }^2)$; James-Stein replaces the unknown $B$ by the unbiased estimate $1-(p-2){\sigma }^2/\Vert X{\Vert }^2$ formed from the marginal $\Vert X{\Vert }^2\sim ({\sigma }^2+{\tau }^2){\chi }_p^2$. The estimator is thus a feasible approximation to the Bayes rule of 45.03.01 that learns the prior variance from the ensemble — the precise sense in which estimating many means jointly lets each one borrow strength from the rest. The hierarchical formulation, where ${\tau }^2$ itself carries a prior, is the subject of 45.03.03; James-Stein is its plug-in, point-estimated limit. This is the same shrinkage-as-prior-information principle that drives ridge regression 45.06.03, with the prior centred at the origin and its scale estimated rather than fixed.

The dimension threshold $p\ge 3$ has a geometric reading that survives generalisation. The improvement term is $(p-2{)}^2{\sigma }^4\mathbb{E}[1/\Vert X{\Vert }^2]$; the factor $(p-2)$ is the divergence of the radial vector field $X/\Vert X{\Vert }^2$, which is the Green's function of the Laplacian in ${\mathbb{R}}^p$, harmonic away from the origin exactly when $p\ge 3$ and the field is integrable against the Gaussian. In one or two dimensions the field is too singular and $\mathbb{E}[1/\Vert X{\Vert }^2]=\infty$, so the construction fails — and indeed Stein and Brown showed the sample mean is admissible there. Brown's 1971 characterisation ties admissibility of the mean to the recurrence of an associated diffusion: the mean is admissible if and only if Brownian motion in ${\mathbb{R}}^p$ is recurrent, which holds precisely for $p\le 2$. Admissibility of the maximum-likelihood estimator and recurrence of Brownian motion are the same dichotomy, with $p=3$ the transition.

Shrinkage estimators that dominate $X$ are never admissible if they retain the radial singularity or the truncation kink, but admissible dominators exist. Strawderman exhibited proper Bayes estimators — posterior means under suitable hierarchical priors — that dominate the maximum-likelihood estimator and are admissible by the unique-Bayes argument of 45.01.01. The positive-part estimator ${\delta }_{\mathrm{JS}}^{+}$ dominates ${\delta }_{\mathrm{JS}}$ but is itself inadmissible, its risk function non-smooth at $\Vert X{\Vert }^2=(p-2){\sigma }^2$; smoothing the truncation recovers admissibility. The lesson is that the admissible terminus of the shrinkage chain is a genuine Bayes rule, and James-Stein is the empirical-Bayes shadow of one.

Synthesis. The central insight is that Stein's lemma is the hinge on which the entire phenomenon turns: it converts the $\theta$-dependent cross term into a divergence the data can estimate, and through SURE this is exactly what makes risk computable without the truth and shrinkage provably beneficial. The foundational reason the maximum-likelihood estimator fails for $p\ge 3$ is that summed squared-error loss couples coordinates while the divergence $\nabla \cdot (X/\Vert X{\Vert }^2)=(p-2)/\Vert X{\Vert }^2$ accumulates a $p$-dependent gain that overtakes the squared-norm cost; this generalises the one-coordinate bias-variance bargain of 45.06.03 to all $p$ directions at once, and putting these together with the empirical-Bayes derivation shows the James-Stein factor is the data-estimated posterior contraction of 45.03.01, not a heuristic. The result is dual to ridge: both contract toward a point, ridge with a fixed design-dependent factor and James-Stein with a data-chosen one, and the bridge is the shared theorem that controlled bias buys total-risk reduction. The dimension threshold is exactly the Laplacian-Green's-function / Brownian-recurrence dichotomy at $p=3$, the same boundary that appears again in 45.03.03 when the hierarchical prior's scale is estimated, so admissibility, harmonic analysis on ${\mathbb{R}}^p$, and empirical Bayes are one theory read three ways.

Full proof set Master

The dominance theorem, Stein's lemma, the SURE identity, and the divergence computation are proved in full in the Key theorem and Exercises. The remaining Master claims are recorded here.

Proposition 1 (James-Stein risk at the origin and the unbounded gain factor). For $p\ge 3$, $R(0,{\delta }_{\mathrm{JS}})=2{\sigma }^2$, so the risk ratio $R(0,{\delta }_0)/R(0,{\delta }_{\mathrm{JS}})=p/2\to \infty$.

Proof. At $\theta =0$, $\Vert X{\Vert }^2/{\sigma }^2\sim {\chi }_p^2$, and for a chi-squared with $p\ge 3$ degrees of freedom $\mathbb{E}[1/{\chi }_p^2]=1/(p-2)$ (integrate the density $f(t)\propto t^{p/2-1}{e}^{-t/2}$ against $t^{-1}$, lowering the shape parameter by one). Hence ${\mathbb{E}}_0[1/\Vert X{\Vert }^2]={\sigma }^{-2}/(p-2)$, and substituting into the risk formula, $R(0,{\delta }_{\mathrm{JS}})=p{\sigma }^2-(p-2{)}^2{\sigma }^4\cdot {\sigma }^{-2}/(p-2)=p{\sigma }^2-(p-2){\sigma }^2=2{\sigma }^2$. The ratio to $R(0,{\delta }_0)=p{\sigma }^2$ is $p/2$, which grows without bound in $p$. $\square$

Proposition 2 (positive-part dominance). For $p\ge 3$, the positive-part estimator ${\delta }_{\mathrm{JS}}^{+}(X)=(1-(p-2){\sigma }^2/\Vert X{\Vert }^2{)}^{+}X$ satisfies $R(\theta ,{\delta }_{\mathrm{JS}}^{+})\le R(\theta ,{\delta }_{\mathrm{JS}})$ for all $\theta$, strictly for some $\theta$.

Proof. Write the shrinkage factor $\varphi (X)=1-(p-2){\sigma }^2/\Vert X{\Vert }^2$, so ${\delta }_{\mathrm{JS}}=\varphi (X)X$ and ${\delta }_{\mathrm{JS}}^{+}=\varphi (X{)}^{+}X$. On the event $A=\{\varphi (X)\ge 0\}$ the two estimators coincide. On $A^c=\{\varphi (X)<0\}$, ${\delta }_{\mathrm{JS}}=\varphi (X)X$ has the opposite sign to $X$ while ${\delta }_{\mathrm{JS}}^{+}=0$. For any fixed $\theta$, conditional on $X$ in $A^c$, compare squared losses: $\Vert \varphi (X)X-\theta {\Vert }^2-\Vert 0-\theta {\Vert }^2=\varphi (X{)}^2\Vert X{\Vert }^2-2\varphi (X)⟨X,\theta ⟩$. The pointwise inequality $\Vert \varphi (X{)}^{+}X-\theta {\Vert }^2\le \Vert \varphi (X)X-\theta {\Vert }^2$ holds on $A^c$ because replacing the reversed vector $\varphi (X)X$ (which points away from the half-space containing typical $\theta$) by the origin cannot increase the distance to $\theta$ in expectation over the sign symmetry of the conditional law of $X$ given $\Vert X\Vert$; integrating over $A^c$, which has positive probability for every $\theta$, gives $R(\theta ,{\delta }_{\mathrm{JS}}^{+})\le R(\theta ,{\delta }_{\mathrm{JS}})$, strict because $A^c$ carries positive mass and the conditional gain is strictly positive there. $\square$

Proposition 3 (admissibility of the sample mean for $p\le 2$). For $p\in \{1,2\}$ and $X\sim N(\theta ,{\sigma }^2{I}_p)$ under squared-error loss, ${\delta }_0(X)=X$ is admissible.

Proof. Use Blyth's limiting-Bayes method 45.01.01. Take the priors ${\pi }_n=N(0,n{\sigma }^2{I}_p)$. The Bayes estimator is ${\delta }_n(X)=\frac{n}{n+1}X$ with Bayes risk computable in closed form, and the difference $r({\pi }_n,{\delta }_0)-r({\pi }_n,{\delta }_n)$, after normalising by the prior mass on bounded sets, tends to $0$ as $n\to \infty$ precisely when ${\int }^{\infty }{r}^{1-p}dr$ diverges — that is, for $p\le 2$. By Blyth's lemma, if no estimator can beat ${\delta }_0$ by a margin that survives the limit of these increasingly diffuse priors, then ${\delta }_0$ is admissible. The integral ${\int }^{\infty }{r}^{1-p}dr$ diverges exactly for $p\le 2$ and converges for $p\ge 3$, so the method certifies admissibility in dimensions one and two and (correctly) fails to in dimension three and above, matching the James-Stein dominance. $\square$

Proposition 4 (multivariate Stein identity, vector form). For $X\sim N(\theta ,{\sigma }^2{I}_p)$ and weakly differentiable $g:{\mathbb{R}}^p\to {\mathbb{R}}^p$ with ${\mathbb{E}}_{\theta }\Vert g(X)\Vert <\infty$ and ${\mathbb{E}}_{\theta }|\nabla \cdot g(X)|<\infty$, ${\mathbb{E}}_{\theta }[⟨X-\theta ,g(X)⟩]={\sigma }^2{\mathbb{E}}_{\theta }[\nabla \cdot g(X)]$.

Proof. Apply the scalar identity (Key theorem) to each coordinate function $h=g_i$ in the variable $x_i$, holding the other coordinates fixed, using the product form of the isotropic Gaussian density ${\prod }_j{\phi }_{\sigma }(x_j-{\theta }_j)$ so that the one-dimensional integration by parts in $x_i$ goes through with the remaining coordinates as parameters: ${\mathbb{E}}_{\theta }[(X_i-{\theta }_i)g_i(X)]={\sigma }^2{\mathbb{E}}_{\theta }[\partial g_i/\partial x_i]$. Summing over $i$, the left side is ${\mathbb{E}}_{\theta }[⟨X-\theta ,g(X)⟩]$ and the right side is ${\sigma }^2{\mathbb{E}}_{\theta }[{\sum }_i\partial g_i/\partial x_i]={\sigma }^2{\mathbb{E}}_{\theta }[\nabla \cdot g(X)]$. The integrability hypotheses justify Fubini and the vanishing of the one-dimensional boundary terms. $\square$

Connections Master

Ridge regression and shrinkage estimation 45.06.03 is the linear-model sibling: both estimators contract toward a point and both trade unbiasedness for total-risk reduction, but ridge applies a fixed factor $d_j^2/(d_j^2+\lambda )$ in each principal direction while James-Stein applies one data-chosen factor $1-(p-2){\sigma }^2/\Vert X{\Vert }^2$ to the whole vector; the orthonormal-design ridge MSE computation there is the same bias-variance bargain that SURE makes for James-Stein, and the historical note in that unit already flags Stein's 1956 result as the decision-theoretic root of shrinkage.

Statistical decision theory 45.01.01 supplies the loss, risk, dominance, and admissibility vocabulary the whole unit speaks: James-Stein is the concrete witness that the sample mean — uniformly minimum-variance unbiased, best invariant, maximum-likelihood — is nonetheless inadmissible for $p\ge 3$, the Stein phenomenon listed there as the proof that admissibility is global and dimension-sensitive, and Blyth's limiting-Bayes method used in Proposition 3 is the admissibility tool from that unit.

Random variables and expected value 26.03.01 provides the multivariate-normal mean model $X\sim N(\theta ,{\sigma }^2{I}_p)$, the coordinatewise independence that makes $R(\theta ,X)=p{\sigma }^2$, and the chi-squared and noncentral-chi-squared laws of $\Vert X{\Vert }^2$ whose reciprocal moment $\mathbb{E}[1/\Vert X{\Vert }^2]$ controls the size of the improvement and its finiteness for $p\ge 3$.

Bayes estimation under loss 45.03.01 is the rule James-Stein approximates: the conjugate-normal posterior mean $BX$ with $B={\tau }^2/({\sigma }^2+{\tau }^2)$ is the genuine Bayes estimator, and James-Stein is the empirical-Bayes plug-in that replaces the unknown $B$ by an unbiased estimate formed from the marginal $\Vert X{\Vert }^2$, so the shrinkage is estimated prior information rather than assumed.

Hierarchical Bayes 45.03.03 is the forward target: placing a prior on the prior scale ${\tau }^2$ and integrating gives the full hierarchical estimator of which James-Stein is the point-estimated, empirical-Bayes limit; the Brown / Brownian-recurrence characterisation of the $p\ge 3$ threshold reappears there as the condition under which the hierarchical shrinkage strictly improves on no pooling.

Historical & philosophical context Master

Charles Stein announced the inadmissibility of the sample mean for the multivariate normal in three or more dimensions at the Third Berkeley Symposium in 1956 ^{[Stein 1981 — the 1981 Annals paper recapitulates the program]}, a result received as a paradox because the sample mean was the textbook-optimal estimator by every classical criterion. Stein's argument was an existence proof; the explicit dominating estimator ${\delta }_{\mathrm{JS}}=(1-(p-2){\sigma }^2/\Vert X{\Vert }^2)X$ was supplied by Willard James and Stein in 1961 ^{[James & Stein 1961]} at the Fourth Berkeley Symposium, together with the exact risk computation. The integration-by-parts identity that streamlines the proof — Stein's lemma — and the unbiased risk estimate (SURE) that turns it into a construction principle were given in Stein's 1981 Annals of Statistics paper ^{[Stein 1981]}.

Bradley Efron and Carl Morris in a sequence of papers from 1972 to 1975, of which the 1973 Journal of the American Statistical Association paper ^{[Efron & Morris 1973]} is central, reinterpreted James-Stein as an empirical-Bayes estimator, showing the shrinkage factor is the data-estimated posterior contraction under a normal prior with variance read off the ensemble; their 1977 Scientific American exposition popularised the result through the baseball batting-average example, where shrinking each player's early-season average toward the league mean predicts the rest of the season better than the raw averages. Lawrence Brown's 1971 Annals of Mathematical Statistics paper tied admissibility of the mean to the recurrence of Brownian motion, fixing $p=3$ as the transition, and William Strawderman exhibited proper-Bayes estimators dominating the mean and hence admissible, locating the admissible terminus of the shrinkage chain. The synthesis with ridge, subset selection, and the lasso into a single account of shrinkage in estimation and regression is given by Lehmann and Casella ^{[Lehmann & Casella §5.5]}.

Bibliography Master

@inproceedings{stein1956inadmissibility,
  author    = {Stein, Charles},
  title     = {Inadmissibility of the usual estimator for the mean of a multivariate normal distribution},
  booktitle = {Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability},
  volume    = {1},
  pages     = {197--206},
  publisher = {University of California Press},
  year      = {1956}
}

@inproceedings{james1961estimation,
  author    = {James, William and Stein, Charles},
  title     = {Estimation with Quadratic Loss},
  booktitle = {Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability},
  volume    = {1},
  pages     = {361--379},
  publisher = {University of California Press},
  year      = {1961}
}

@article{stein1981sure,
  author  = {Stein, Charles M.},
  title   = {Estimation of the Mean of a Multivariate Normal Distribution},
  journal = {The Annals of Statistics},
  volume  = {9},
  number  = {6},
  pages   = {1135--1151},
  year    = {1981}
}

@article{efronmorris1973stein,
  author  = {Efron, Bradley and Morris, Carl},
  title   = {Stein's Estimation Rule and Its Competitors --- An Empirical Bayes Approach},
  journal = {Journal of the American Statistical Association},
  volume  = {68},
  number  = {341},
  pages   = {117--130},
  year    = {1973}
}

@article{efronmorris1977paradox,
  author  = {Efron, Bradley and Morris, Carl},
  title   = {Stein's Paradox in Statistics},
  journal = {Scientific American},
  volume  = {236},
  number  = {5},
  pages   = {119--127},
  year    = {1977}
}

@article{brown1971admissible,
  author  = {Brown, Lawrence D.},
  title   = {Admissible Estimators, Recurrent Diffusions, and Insoluble Boundary Value Problems},
  journal = {The Annals of Mathematical Statistics},
  volume  = {42},
  number  = {3},
  pages   = {855--903},
  year    = {1971}
}

@book{lehmanncasella1998theory,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2nd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}