Fisher Information and the Cramér-Rao Lower Bound

Intuition Beginner

You measured a coin's bias, or a factory's average bolt length, and reported a single best guess. The next honest question is: how good can that guess possibly be? No matter how clever your recipe, there is a floor on how tightly any unbiased guess can cluster around the truth, set by how much the data itself reveals. Fisher information is the number that measures that revelation, and the Cramér-Rao bound turns it into the floor.

Think of the likelihood meter from the estimation story — the dial you turn to the setting that makes your data least surprising. If that meter has a sharp, narrow peak, the data points firmly at one value and small changes in the setting are loudly noticed; if the peak is broad and flat, many settings look almost equally good and the data barely cares. Fisher information measures the sharpness of that peak. Sharp peak, high information, tight possible estimate; flat peak, low information, loose estimate.

The Cramér-Rao bound is the precise trade: the variance of any unbiased estimate can be no smaller than one divided by the total information in the data. Collect more data and the information adds up, so the floor drops. An estimator that actually sits on the floor is called efficient — it wrings every drop of precision the data allows. The bound is the speed limit of estimation, and efficiency is driving exactly at it.

Visual Beginner

Picture two likelihood meters for two different experiments, each plotted as a hump over the candidate values. The first hump is tall and narrow, a steep spike around the true value; the second is low and wide, a gentle mound. Both peak at the same place, so both give the same best guess — but the narrow one pins the guess down far more confidently, because the data fights back hard against any wrong setting.

likelihood peak	Fisher information	floor on the variance of any unbiased estimate
tall and narrow (sharp)	high	low — a tight estimate is possible
low and wide (flat)	low	high — even the best estimate is loose

The takeaway: the sharpness of the likelihood peak is the information, and the information sets a floor under how good any unbiased estimate can be. The floor is one divided by the information. More data sharpens the peak, raises the information, and lowers the floor.

Worked example Beginner

You watch a process that produces rare events — calls to a help desk in an hour, say — that follow a pattern with a single unknown average rate. You collect $n=25$ hours of counts and want to know the best precision any unbiased estimate of the rate can reach.

Step 1. The information in one observation. For this counting pattern the Fisher information in a single hour turns out to be one divided by the rate: if the true rate is $4$ calls per hour, one observation carries information $1/4=0.25$.

Step 2. Information adds up. Twenty-five independent hours carry twenty-five times the information of one: total information is $25\times 0.25=6.25$.

Step 3. The floor on the variance. The Cramér-Rao bound says the variance of any unbiased estimate of the rate is at least one divided by the total information: $1/6.25=0.16$.

Step 4. Read it as a precision. A variance of $0.16$ means a typical error (the square root) of $\sqrt{0.16}=0.4$ calls per hour. No unbiased recipe, however clever, can do better than a typical error of $0.4$ from this much data.

Step 5. The natural estimate hits the floor. The plain average of the 25 counts is unbiased, and its variance is the rate divided by $n$, which is $4/25=0.16$ — exactly the floor. The average is efficient: it already extracts all the precision the data allows.

What this tells us: information from independent observations adds, the floor is one over the total, and the everyday sample average is, for this counting pattern, already as good as any unbiased estimate can be.

Check your understanding Beginner

Formal definition Intermediate+

Let $X$ be an observation with density (or mass function) $f(x\mid \theta )$ from a family indexed by a scalar $\theta$ in an open set $\Theta \subseteq \mathbb{R}$, with the regularity conditions: the support $\{x:f(x\mid \theta )>0\}$ does not depend on $\theta$; $f$ is differentiable in $\theta$; and the operations of differentiation in $\theta$ and integration in $x$ may be interchanged. The likelihood, log-likelihood, and score for a sample are as in 45.01.04.

Definition (score). The score is the $\theta$-derivative of the log-density,

$$s(\theta ;x)=\frac{\partial }{\partial \theta }\log f(x\mid \theta )=\frac{{\partial }_{\theta }f(x\mid \theta )}{f(x\mid \theta )}.$$

For an i.i.d. sample $X_1,\dots ,X_n$ the sample score is $s_n(\theta )={\sum }_{i=1}^{n}s(\theta ;X_i)$, the gradient of the log-likelihood $\ell (\theta )$.

Definition (Fisher information). The Fisher information in one observation is the variance of the score,

$$I_1(\theta )={\mathrm{Var}}_{\theta }(s(\theta ;X))={\mathbb{E}}_{\theta }\left[(\frac{\partial }{\partial \theta }\log f(X\mid \theta ){)}^2\right].$$

The second equality uses the score identity ${\mathbb{E}}_{\theta }[s(\theta ;X)]=0$, derived below. Under twice-differentiability the same regularity gives the equivalent curvature form

$$I_1(\theta )=-{\mathbb{E}}_{\theta }\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(X\mid \theta )\right],$$

minus the expected second derivative of the log-density. The information in the whole sample is $I_n(\theta )={\mathrm{Var}}_{\theta }(s_n(\theta ))$; by independence and the score identity, $I_n(\theta )=n{I}_1(\theta )$.

Definition (efficiency). Let $W=W(X_1,\dots ,X_n)$ estimate a scalar quantity $\tau (\theta )$ with ${\mathbb{E}}_{\theta }[W]=\tau (\theta )$ (unbiased for $\tau$). The Cramér-Rao lower bound asserts ${\mathrm{Var}}_{\theta }(W)\ge [{\tau }^{\prime }(\theta ){]}^2/I_n(\theta )$; the special case $\tau (\theta )=\theta$ reads ${\mathrm{Var}}_{\theta }(W)\ge 1/(n{I}_1(\theta ))$. The efficiency of an unbiased $W$ for $\theta$ is the ratio $e(W)=[n{I}_1(\theta ){]}^{-1}/{\mathrm{Var}}_{\theta }(W)\in (0,1]$, and $W$ is efficient when $e(W)=1$, i.e. it attains the bound.

Definition (reparametrisation of information). If $\theta =\theta (\eta )$ is a smooth invertible reparametrisation, the information transforms as $I^{(\eta )}(\eta )=I^{(\theta )}(\theta (\eta ))({\theta }^{\prime }(\eta ){)}^2$. Information is not invariant under reparametrisation; it is a quadratic-form (metric-like) object, which is the geometric content sharpened at Master tier.

Definition (multiparameter information matrix). For $\theta \in \Theta \subseteq {\mathbb{R}}^k$ the score is the gradient vector $s(\theta ;x)={\nabla }_{\theta }\log f(x\mid \theta )$, and the Fisher information matrix is

$$I(\theta )={\mathrm{Cov}}_{\theta }(s(\theta ;X))={\mathbb{E}}_{\theta }\left[s{s}^{\top }\right]=-{\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2\log f(X\mid \theta )],$$

a symmetric positive-semidefinite $k\times k$ matrix, equal to minus the expected Hessian of the log-density under regularity.

The symbols introduced here are: $s(\theta ;x)$ (score), $I_1(\theta )$ and $I_n(\theta )$ (scalar Fisher information, one observation and $n$), $I(\theta )$ (information matrix), $\tau (\theta )$ (an estimand), $e(W)$ (efficiency). The estimator notation $W$, the bias and variance, and the log-likelihood $\ell$ are carried from 45.01.04.

Counterexamples to common slips Intermediate+

• "The two formulas for $I_1(\theta )$ always agree." They agree only under the regularity that licenses differentiating under the integral. When the support of $f$ moves with $\theta$ — uniform on $(0,\theta )$ — the score identity ${\mathbb{E}}_{\theta }[s]=0$ fails, both formulas lose their meaning, and the curvature form $-\mathbb{E}[{\partial }^2\log f]$ does not equal $\mathrm{Var}(s)$.

• "The Cramér-Rao bound holds for every estimator." It bounds the variance of estimators of a fixed differentiable estimand $\tau (\theta )$ — most often unbiased ones, where $\tau (\theta )=\theta$. For a biased estimator the bound becomes $[{\tau }^{\prime }(\theta ){]}^2/I_n(\theta )$ with $\tau (\theta )={\mathbb{E}}_{\theta }[W]$, and a biased estimator can have variance below $1/(n{I}_1)$ without contradiction.

• "An estimator with variance below $1/(n{I}_1)$ is impossible." Only among regular families and unbiased estimators. The uniform-$(0,\theta )$ MLE $X_{(n)}$ has variance of order $n^{-2}$, far below any $1/(n{I}_1)$ rate, because the family is non-regular and the bound does not apply.

• "Fisher information is a property of the data alone." It is a property of the model and the parametrisation: $I^{(\eta )}=I^{(\theta )}({\theta }^{\prime }{)}^2$ changes under reparametrisation, so the numerical information depends on the coordinate chosen, even though the estimation problem is the same.

Key theorem with proof Intermediate+

The signature result is the Cramér-Rao inequality: the variance of any unbiased estimator is bounded below by the reciprocal of the Fisher information, scaled by the squared sensitivity of the estimand. The proof is a single application of the Cauchy-Schwarz inequality to the covariance of the estimator with the score, and the equality case identifies exactly which estimators attain the floor.

Theorem (Cramér-Rao information inequality). Let $X_1,\dots ,X_n$ be i.i.d. from a regular family $f(x\mid \theta )$, $\theta \in \Theta \subseteq \mathbb{R}$ open, and let $W=W(X_1,\dots ,X_n)$ satisfy ${\mathbb{E}}_{\theta }[W]=\tau (\theta )$ with $\tau$ differentiable, where differentiation under the integral defining ${\mathbb{E}}_{\theta }[W]$ is permitted. Then for every $\theta$ with $I_n(\theta )>0$,

$${\mathrm{Var}}_{\theta }(W)\ge \frac{[{\tau }^{\prime }(\theta ){]}^2}{I_n(\theta )}=\frac{[{\tau }^{\prime }(\theta ){]}^2}{n{I}_1(\theta )}.$$

Equality holds for all such $\theta$ if and only if $W-\tau (\theta )=a(\theta )s_n(\theta )$ almost surely for some scalar $a(\theta )$, i.e. $W$ is an affine function of the sample score.

Proof. First, the score has mean zero. Differentiating the normalisation $\int f(x\mid \theta )dx=1$ under the integral sign,

$$0=\frac{\partial }{\partial \theta }\int f(x\mid \theta )dx=\int {\partial }_{\theta }f(x\mid \theta )dx=\int \frac{{\partial }_{\theta }f}{f}fdx={\mathbb{E}}_{\theta }[s(\theta ;X)].$$

Hence ${\mathbb{E}}_{\theta }[s_n(\theta )]=0$ and ${\mathrm{Var}}_{\theta }(s_n)={\mathbb{E}}_{\theta }[s_n^2]=I_n(\theta )$. Next, differentiate the unbiasedness relation ${\mathbb{E}}_{\theta }[W]=\int W(x){\prod }_{i}f(x_i\mid \theta )dx=\tau (\theta )$ under the integral:

$${\tau }^{\prime }(\theta )=\int W(x)\frac{\partial }{\partial \theta }\prod _{i}f(x_i\mid \theta )dx=\int W(x)s_n(\theta )\prod _{i}f(x_i\mid \theta )dx={\mathbb{E}}_{\theta }[W{s}_n],$$

using ${\partial }_{\theta }{\prod }_{i}f=({\sum }_i{\partial }_{\theta }\log f(x_i)){\prod }_{i}f=s_n{\prod }_{i}f$. Because ${\mathbb{E}}_{\theta }[s_n]=0$,

$${\mathrm{Cov}}_{\theta }(W,s_n)={\mathbb{E}}_{\theta }[W{s}_n]-{\mathbb{E}}_{\theta }[W]{\mathbb{E}}_{\theta }[s_n]={\tau }^{\prime }(\theta ).$$

The Cauchy-Schwarz inequality for the covariance, ${\mathrm{Cov}}_{\theta }(W,s_n{)}^2\le {\mathrm{Var}}_{\theta }(W){\mathrm{Var}}_{\theta }(s_n)$, then gives

$$[{\tau }^{\prime }(\theta ){]}^2\le {\mathrm{Var}}_{\theta }(W)I_n(\theta ),\text{i.e.}{\mathrm{Var}}_{\theta }(W)\ge \frac{[{\tau }^{\prime }(\theta ){]}^2}{I_n(\theta )}.$$

Cauchy-Schwarz holds with equality if and only if $W-{\mathbb{E}}_{\theta }[W]$ and $s_n$ are almost surely proportional, $W-\tau (\theta )=a(\theta )s_n(\theta )$, which is the stated attainment condition. Taking the constant $a(\theta )=\mathrm{Cov}(W,s_n)/\mathrm{Var}(s_n)={\tau }^{\prime }(\theta )/I_n(\theta )$ recovers the floor exactly. $\square$

Bridge. The bound builds toward the asymptotic theory of estimation, because the reciprocal information $1/(n{I}_1(\theta ))$ is exactly the variance that the maximum likelihood estimator achieves in the large-sample limit, and it appears again in the standard error one attaches to any likelihood-based estimate. The foundational reason the proof is one line of Cauchy-Schwarz is that the score is the natural direction of steepest likelihood ascent, so any unbiased estimator must correlate with it at the fixed rate ${\tau }^{\prime }(\theta )$, and Cauchy-Schwarz converts a fixed correlation into a variance floor; this is exactly the same covariance-with-the-score computation that the affine equality case turns into a recipe for efficient estimators. The equality condition $W-\tau =a(\theta )s_n$ is dual to the exponential-family likelihood equation of 45.01.04: there the MLE equates the sufficient statistic to its mean, and here the efficient estimator is an affine function of that same sufficient statistic, so the central insight is that exponential families are precisely where the floor is reachable. Putting these together, the bound sets the bar that asymptotic efficiency 45.04.03 attains, and the multiparameter and nuisance-parameter refinements that generalise the scalar inequality are taken up next.

Exercises Intermediate+

Advanced results Master

The scalar inequality of the Key theorem extends to vector parameters as a matrix inequality in the Loewner order, with the nuisance-parameter case read off from a block inversion; the equality case ties efficiency to exponential families; and the regularity hypotheses that make the bound a theorem are exactly the ones that fail on support-dependent families. The asymptotic counterpart — that the maximum likelihood estimator attains $I(\theta {)}^{-1}$ in the limit — is developed in 45.04.03.

Theorem 1 (information equality). Under the regularity conditions, the variance form and the curvature form of the information matrix coincide: $I(\theta )={\mathbb{E}}_{\theta }[s{s}^{\top }]=-{\mathbb{E}}_{\theta }[{\nabla }^2\log f]$. The identity follows by differentiating the score identity $\int ({\nabla }_{\theta }\log f)fdx=0$ once more: $\int ({\nabla }^2\log f)fdx+\int (\nabla \log f)(\nabla \log f{)}^{\top }fdx=0$, so $-\mathbb{E}[{\nabla }^2\log f]=\mathbb{E}[s{s}^{\top }]$. The curvature form makes the information the expected sharpness of the log-likelihood peak, and the deterministic Hessian of an exponential family (where $-{\nabla }^2\ell =n{\mathrm{Cov}}_{\eta }(T)$, 45.01.04) is why observed and expected information coincide there ^{[Casella, G. & Berger, R. L. — Statistical Inference (2nd ed.)]}.

Theorem 2 (multiparameter Cramér-Rao matrix bound). Let $W$ be an estimator of $\tau (\theta )\in {\mathbb{R}}^p$ with ${\mathbb{E}}_{\theta }[W]=\tau (\theta )$ and Jacobian ${\tau }^{\prime }(\theta )=\partial \tau /\partial \theta \in {\mathbb{R}}^{p\times k}$. Under regularity with $I(\theta )\succ 0$,

$${\mathrm{Cov}}_{\theta }(W)⪰{\tau }^{\prime }(\theta )I(\theta {)}^{-1}{\tau }^{\prime }(\theta {)}^{\top }$$

in the Loewner (positive-semidefinite) order. For $\tau (\theta )=\theta$ this is ${\mathrm{Cov}}_{\theta }(W)⪰I(\theta {)}^{-1}$; in particular each diagonal entry obeys ${\mathrm{Var}}_{\theta }(W_j)\ge [I(\theta {)}^{-1}{]}_{jj}$. The bound on a single coordinate is the inverse of the full matrix, not the reciprocal of the diagonal entry $I_{jj}$, and these differ whenever the parameters are informationally coupled ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Theorem 3 (nuisance-parameter information loss). Partition $\theta =(\psi ,\lambda )$ into a parameter of interest $\psi$ and a nuisance $\lambda$, with information block matrix I = \begin{psmallmatrix} I_{\psi\psi} & I_{\psi\lambda} \\ I_{\lambda\psi} & I_{\lambda\lambda} \end{psmallmatrix}. The variance bound for an unbiased estimator of $\psi$ is the inverse of the effective information $I_{\psi \psi \cdot \lambda }=I_{\psi \psi }-I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }$, the Schur complement of the nuisance block. Since $I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }⪰0$, the effective information never exceeds $I_{\psi \psi }$, so estimating an unknown nuisance can only inflate the floor on $\psi$ relative to the known-nuisance case; the inflation vanishes precisely when $I_{\psi \lambda }=0$, i.e. when the parameters are orthogonal ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Theorem 4 (attainment is exponential). The Cramér-Rao bound for a scalar estimand is attained by an unbiased estimator $W$ for all $\theta$ in an open set if and only if the model is a one-parameter exponential family on the natural sufficient statistic $T$ and $W$ is an affine function of $T$. The forward direction is the equality case of the Key theorem: $W-\tau (\theta )=a(\theta )s(\theta )$ for all $\theta$ forces ${\partial }_{\theta }\log f(x\mid \theta )=(W(x)-\tau (\theta ))/a(\theta )$, whose $x$-dependence enters only through $W$; integrating in $\theta$ exhibits the exponential form $\log f=\eta (\theta )W(x)-B(\theta )+c(x)$. The reverse direction is the computation of Exercise 8. Exact finite-sample efficiency is therefore a rigid property, available only in the exponential class; outside it, efficiency is at best asymptotic ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Theorem 5 (additivity and reparametrisation). For independent (not necessarily identical) observations the information matrices add: $I_{1:n}(\theta )={\sum }_{i=1}^n{I}^{(i)}(\theta )$, specialising to $n{I}_1(\theta )$ in the i.i.d. case. Under a smooth invertible reparametrisation $\theta =g(\eta )$ with Jacobian $J=\partial \theta /\partial \eta$, the information transforms as $I^{(\eta )}(\eta )=J^{\top }{I}^{(\theta )}(g(\eta ))J$. The quadratic transformation law identifies $I(\theta )$ as a Riemannian metric on the parameter manifold — the Fisher-Rao metric — under which the bound $\mathrm{Cov}⪰I^{-1}$ is coordinate-free even though the matrix entries are not ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Synthesis. The whole subject is one inequality and the structure of its equality case. The foundational reason the Cramér-Rao bound holds is that any unbiased estimator must covary with the score at the fixed rate ${\tau }^{\prime }(\theta )$, and Cauchy-Schwarz turns that fixed covariance into the variance floor $[{\tau }^{\prime }{]}^2/I_n$; this is exactly the calculation that, read as an equality, says efficiency means being affine in the score. The central insight is that the score is the canonical direction in the model, so its variance — the Fisher information — is simultaneously the curvature of the log-likelihood, the metric on parameter space, and the inverse of the best achievable variance, three faces of one quadratic form. The multiparameter bound $\mathrm{Cov}⪰I^{-1}$ generalises the scalar reciprocal, and the nuisance-parameter Schur complement is dual to it: marginalising over what you do not care about replaces the information by its effective part, never increasing it. Putting these together, attainment is rigid — exact efficiency is the exponential family of 45.01.02 and 45.01.04 and nothing else — which is the bridge to the asymptotic theory, where the rigidity relaxes: the maximum likelihood estimator is efficient only in the limit, attaining $I(\theta {)}^{-1}$ as $n\to \infty$ in 45.04.03, and the same information that sets the finite-sample floor here sets the asymptotic variance there.

Full proof set Master

The Cramér-Rao inequality (scalar) and the attainment characterisation are proved in the Key theorem; the exponential-family attainment is Exercise 8; the uniform breakdown is Exercise 7. The remaining Master claims are recorded here.

Proposition 1 (information equality, multiparameter). Under the regularity conditions, $I(\theta )={\mathbb{E}}_{\theta }[s{s}^{\top }]=-{\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2\log f]$.

Proof. The score identity ${\mathbb{E}}_{\theta }[{\nabla }_{\theta }\log f]=\int ({\nabla }_{\theta }\log f)fdx=0$ holds coordinatewise by differentiating $\int fdx=1$ under the integral. Differentiate this vector identity in $\theta$ once more, interchanging derivative and integral:

$$0={\nabla }_{\theta }\int ({\nabla }_{\theta }\log f)fdx=\int ({\nabla }_{\theta }^2\log f)fdx+\int ({\nabla }_{\theta }\log f)({\nabla }_{\theta }f{)}^{\top }dx.$$

In the second integral write ${\nabla }_{\theta }f=({\nabla }_{\theta }\log f)f$, so it equals $\int ({\nabla }_{\theta }\log f)({\nabla }_{\theta }\log f{)}^{\top }fdx={\mathbb{E}}_{\theta }[s{s}^{\top }]$. Hence ${\mathbb{E}}_{\theta }[{\nabla }_{\theta }^2\log f]+{\mathbb{E}}_{\theta }[s{s}^{\top }]=0$, i.e. $-{\mathbb{E}}_{\theta }[{\nabla }^2\log f]={\mathbb{E}}_{\theta }[s{s}^{\top }]={\mathrm{Cov}}_{\theta }(s)=I(\theta )$, the last equality using ${\mathbb{E}}_{\theta }[s]=0$. $\square$

Proposition 2 (multiparameter matrix bound). Under the hypotheses of Theorem 2, ${\mathrm{Cov}}_{\theta }(W)⪰{\tau }^{\prime }(\theta )I(\theta {)}^{-1}{\tau }^{\prime }(\theta {)}^{\top }$.

Proof. As in the scalar proof, differentiating ${\mathbb{E}}_{\theta }[W]=\tau (\theta )$ under the integral gives the cross-covariance ${\mathrm{Cov}}_{\theta }(W,s)={\mathbb{E}}_{\theta }[W{s}^{\top }]={\tau }^{\prime }(\theta )$, a $p\times k$ matrix, using ${\mathbb{E}}_{\theta }[s]=0$. Consider the stacked vector $(W-\tau (\theta ),s)$; its covariance matrix is positive semidefinite:

$$\Sigma =\left(\begin{array}{cc}\mathrm{Cov}(W)& {\tau }^{\prime }(\theta )\\ {\tau }^{\prime }(\theta {)}^{\top }& I(\theta )\end{array}\right)⪰0.$$

Since $I(\theta )\succ 0$, the Schur-complement criterion for $\Sigma ⪰0$ is exactly $\mathrm{Cov}(W)-{\tau }^{\prime }(\theta )I(\theta {)}^{-1}{\tau }^{\prime }(\theta {)}^{\top }⪰0$, the asserted bound. Equality in the Loewner order holds iff $W-\tau (\theta )={\tau }^{\prime }(\theta )I(\theta {)}^{-1}s$ almost surely, the matrix analogue of affine-in-the-score. $\square$

Proposition 3 (nuisance-parameter Schur complement). With $\theta =(\psi ,\lambda )$ and $I$ blocked accordingly, the bound on an unbiased estimator of $\psi$ is $[I_{\psi \psi \cdot \lambda }{]}^{-1}=[I_{\psi \psi }-I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }{]}^{-1}$, and $I_{\psi \psi \cdot \lambda }⪯I_{\psi \psi }$.

Proof. By Theorem 2 with $\tau (\theta )=\psi$ (so ${\tau }^{\prime }=[\mathrm{Id}0]$), the bound is the top-left block of $I(\theta {)}^{-1}$. Block inversion of a symmetric positive-definite I = \begin{psmallmatrix} I_{\psi\psi} & I_{\psi\lambda} \\ I_{\lambda\psi} & I_{\lambda\lambda} \end{psmallmatrix} gives the top-left block of the inverse as the inverse of the Schur complement $I_{\psi \psi \cdot \lambda }=I_{\psi \psi }-I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }$. Because $I_{\lambda \lambda }\succ 0$, the correction $I_{\psi \lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda \psi }$ is positive semidefinite, so $I_{\psi \psi \cdot \lambda }⪯I_{\psi \psi }$, whence $[I_{\psi \psi \cdot \lambda }{]}^{-1}⪰I_{\psi \psi }^{-1}$: the floor with an unknown nuisance dominates the floor with a known one. The two agree iff $I_{\psi \lambda }=0$. $\square$

Proposition 4 (attainment forces the exponential form). If an unbiased estimator $W$ of $\tau (\theta )$ attains the scalar bound for all $\theta$ in an open interval, the model is a one-parameter exponential family and $W$ is affine in its natural statistic.

Proof. Equality in Cauchy-Schwarz for all $\theta$ means $W(x)-\tau (\theta )=a(\theta )s(\theta ;x)$ almost surely, with $a(\theta )={\tau }^{\prime }(\theta )/I_1(\theta )\ne 0$. Solving for the score, ${\partial }_{\theta }\log f(x\mid \theta )=a(\theta {)}^{-1}(W(x)-\tau (\theta ))$. Integrate in $\theta$ from a fixed ${\theta }_0$:

$$\log f(x\mid \theta )=\log f(x\mid {\theta }_0)+W(x){\int }_{{\theta }_0}^{\theta }a(u{)}^{-1}du-{\int }_{{\theta }_0}^{\theta }a(u{)}^{-1}\tau (u)du.$$

Set $\eta (\theta )={\int }_{{\theta }_0}^{\theta }a(u{)}^{-1}du$, $T(x)=W(x)$, $-B(\theta )=-{\int }_{{\theta }_0}^{\theta }{a}^{-1}\tau du$, and $c(x)=\log f(x\mid {\theta }_0)$. Then $f(x\mid \theta )=\exp (\eta (\theta )T(x)-B(\theta )+c(x))$, a one-parameter exponential family with natural statistic $T=W$. Hence $W$ is the natural sufficient statistic (in particular affine in it). The converse — that such a family attains the bound for $\tau (\eta )=A^{\prime }(\eta )$ — is Exercise 8. $\square$

Proposition 5 (additivity over independent observations). If $X_1,\dots ,X_n$ are independent with information matrices $I^{(i)}(\theta )$, the sample information is ${\sum }_i{I}^{(i)}(\theta )$; in the i.i.d. case it is $n{I}_1(\theta )$.

Proof. The sample score is $s_n(\theta )={\sum }_i{s}^{(i)}(\theta ;X_i)$, each summand with mean zero by the score identity. Independence makes the summands uncorrelated, so ${\mathrm{Cov}}_{\theta }(s_n)={\sum }_i{\mathrm{Cov}}_{\theta }(s^{(i)})={\sum }_i{I}^{(i)}(\theta )$. The identical-distribution case collapses the sum to $n{I}_1(\theta )$. The same additivity is visible in the curvature form, since the log-likelihood is the sum of the per-observation log-densities and the Hessian is additive. $\square$

Connections Master

• The maximum likelihood machinery of 45.01.04 supplies every input to the bound: the score is the gradient of the log-likelihood whose root is the MLE, the Fisher information is minus the expected Hessian whose data version is the curvature a Newton or Fisher-scoring step uses, and the exponential-family moment-matching identity proved there is exactly the equality case here — the efficient estimator is the natural sufficient statistic, affine in the score. This unit reads the same score and information through the lens of a variance floor rather than an optimisation.

• Sufficiency and exponential families 45.01.02 are where the bound is attainable: the attainment theorem says exact finite-sample efficiency happens only for one-parameter exponential families on their natural statistic, so the factorisation and completeness theory of that unit identifies precisely the estimators that sit on the Cramér-Rao floor. The information matrix $I(\eta )={\mathrm{Cov}}_{\eta }(T)={\nabla }^{2}A(\eta )$ is the same Hessian of the log-partition function studied there, now interpreted as a metric and a variance bound.

• The asymptotic theory of estimation 45.04.03 is where the bound becomes generically reachable: the maximum likelihood estimator is asymptotically normal with variance $I(\theta {)}^{-1}/n$, attaining the Cramér-Rao floor in the large-sample limit even when no finite-sample efficient estimator exists, and the regularity conditions and the information $I_1(\theta )$ assembled here are the exact hypotheses and inputs of that limit theorem. The bound set here is the bar that asymptotic efficiency attains there.

• The estimation-and-inference vocabulary of 26.05.01 is the downstream consumer: the inverse Fisher information is the asymptotic variance from which standard errors and Wald confidence intervals are built, the score is the basis of the score (Rao) test, and the curvature of the log-likelihood at the MLE is the observed information that estimates $I(\theta )$ in practice. The bound is what makes "this is the most precise unbiased estimate possible" a statement with a number attached.

Historical & philosophical context Master

The information inequality is named for Harald Cramér and C. Radhakrishna Rao, who stated and proved it independently in the mid-1940s — Rao in a 1945 paper in the Bulletin of the Calcutta Mathematical Society and Cramér in his 1946 treatise Mathematical Methods of Statistics ^{[Rao 1945; Cramér 1946]}. Closely related forms were given a little earlier by Maurice Fréchet and by Georges Darmois, so the result is sometimes called the Fréchet-Darmois-Cramér-Rao inequality; the multiparameter matrix version and the systematic regularity theory are due to the later text-treatments, notably Lehmann's.

The Fisher information itself is older, introduced by Ronald A. Fisher in his 1922 foundations paper and developed through the 1920s as the measure of the "intrinsic accuracy" a sample carries about a parameter ^{[Fisher 1922]}. Fisher's view that the curvature of the log-likelihood quantifies information, and that the maximum likelihood estimator extracts asymptotically all of it, supplied the conceptual frame the Cramér-Rao bound later made into a finite-sample theorem. Rao's 1945 paper also introduced what is now the Fisher-Rao Riemannian metric on the parameter manifold, recognising the information matrix as a metric tensor and thereby founding information geometry, a programme carried forward by Nikolai Chentsov and Shun'ichi Amari. The non-regular families on which the bound fails — the uniform endpoint chief among them — were the standard demarcation, marking where the differentiate-under-the-integral calculus that the whole theory rests on ceases to apply.

Bibliography Master

@book{casellaberger2002,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2},
  publisher = {Duxbury / Thomson Learning},
  year      = {2002}
}

@book{lehmanncasella1998,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}

@book{wasserman2004,
  author    = {Wasserman, Larry},
  title     = {All of Statistics: A Concise Course in Statistical Inference},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {2004}
}

@article{rao1945,
  author  = {Rao, C. Radhakrishna},
  title   = {Information and the accuracy attainable in the estimation of statistical parameters},
  journal = {Bulletin of the Calcutta Mathematical Society},
  volume  = {37},
  year    = {1945},
  pages   = {81--91}
}

@book{cramer1946,
  author    = {Cram\'er, Harald},
  title     = {Mathematical Methods of Statistics},
  publisher = {Princeton University Press},
  year      = {1946}
}

@article{fisher1922,
  author  = {Fisher, Ronald A.},
  title   = {On the mathematical foundations of theoretical statistics},
  journal = {Philosophical Transactions of the Royal Society of London, Series A},
  volume  = {222},
  year    = {1922},
  pages   = {309--368}
}

@book{amari2016,
  author    = {Amari, Shun'ichi},
  title     = {Information Geometry and Its Applications},
  series    = {Applied Mathematical Sciences},
  volume    = {194},
  publisher = {Springer},
  year      = {2016}
}