Sufficiency, the Factorization Theorem, and Exponential Families

Intuition Beginner

Suppose you flip a bent coin one hundred times to learn how biased it is. You could write down the whole sequence — heads, heads, tails, heads, and so on — but that is far more than you need. To estimate the bias, the only thing that matters is the count of heads. Once you know that 63 of the 100 flips came up heads, the exact order in which they arrived tells you nothing more about the bias. The count is a faithful summary: it has squeezed out everything relevant and thrown away only noise.

A summary that loses nothing relevant about the unknown quantity is called a sufficient statistic. The word means what it says: the summary is sufficient on its own. Anyone holding only the count of heads is in exactly the same position, for the purpose of learning the bias, as someone holding the full sequence.

How can you tell whether a summary is this faithful? The honest test is to ask: if I tell you the summary, but hide the raw data, can you regenerate fake raw data that looks statistically identical to the real thing — without knowing the unknown quantity? For the coin, yes: knowing 63 heads in 100 flips, you can shuffle 63 heads and 37 tails into a random order, and the bias never enters the shuffle. Because you can replay the data from the summary alone, the summary held everything the data knew.

The takeaway: data reduction is not about saving space. It is about identifying the genuinely informative core of an experiment, so that estimators, tests, and decisions can be built on a small faithful summary instead of the raw flood. The same idea, sharpened, organises a huge class of standard models — the so-called exponential families — where the faithful summary always has a fixed small size no matter how much data you collect.

Visual Beginner

Picture the raw data on the left as a long ticket tape of individual outcomes, and the sufficient statistic on the right as a single dial showing one number. An arrow runs from the tape to the dial: many outcomes collapse into one reading. A second, dashed arrow runs backwards from the dial to a fresh blank tape — this is the regeneration step, and the key point is that the unknown quantity (the bias) is nowhere on that backward arrow.

Stage	What you hold	Depends on the unknown bias?
Raw data	the full sequence of 100 flips	yes — through every flip
Summary (count of heads)	a single number, here 63	yes — this is the informative part
Replay the raw data from the summary	a reshuffle of 63 heads, 37 tails	no — pure shuffling, bias absent

The picture captures the whole idea: a faithful summary is one you can run backwards to fake the data, with the unknown quantity playing no part in the faking.

Worked example Beginner

We test the count-of-heads summary on a small experiment. Flip a coin with unknown chance $p$ of heads four times, independently. Write a head as $1$ and a tail as $0$. Suppose the outcome is the sequence $(1,0,1,1)$, so the total count of heads is $T=3$.

Step 1. Chance of this exact sequence. Each head contributes a factor $p$, each tail a factor $1-p$. With three heads and one tail, the chance of the sequence $(1,0,1,1)$ is $p\times (1-p)\times p\times p=p^3(1-p)$.

Step 2. Chance of any sequence with the same count. Any sequence with exactly three heads and one tail has the same chance $p^3(1-p)$, because it has the same number of $p$ factors and $(1-p)$ factors. There are four such sequences (the single tail can sit in any of four positions): $(0,1,1,1)$, $(1,0,1,1)$, $(1,1,0,1)$, $(1,1,1,0)$.

Step 3. Chance the count equals three. Add up the four equal chances: the chance that $T=3$ is $4\times p^3(1-p)$.

Step 4. Replay: chance of our sequence given the count is three. Divide the chance of the one sequence by the chance of the count:

$$\frac{p^3(1-p)}{4p^3(1-p)}=\frac{1}{4}.$$

The $p$ cancels completely. Given that three of four flips were heads, each of the four equally-counted sequences is equally likely, with chance $1/4$, and the value of $p$ never appears.

What this tells us: once the count is fixed, the conditional chance of any particular raw sequence is just plain combinatorics — a uniform shuffle — with no trace of $p$. That cancellation is precisely what makes the count a sufficient statistic. You can regenerate the data from the count alone, and the unknown $p$ sits entirely inside the count.

Check your understanding Beginner

Formal definition Intermediate+

Fix a measurable space $(\mathcal{X},\mathcal{A})$ and a statistical model: a family $\mathcal{P}=\{P_{\theta }:\theta \in \Theta \}$ of probability measures on $(\mathcal{X},\mathcal{A})$ indexed by a parameter $\theta \in \Theta$. A point $x\in \mathcal{X}$ is the data (typically $x=(x_1,\dots ,x_n)$ a sample). A statistic is a measurable map $T:(\mathcal{X},\mathcal{A})\to (\mathcal{T},\mathcal{B})$. Probability spaces and measurability are taken from 37.01.01; conditional expectation $\mathbb{E}[\cdot \mid T]$ and the regular conditional distribution are the Radon-Nikodym constructions used in 37.04.01; expectation of random variables is 26.03.01.

Definition (sufficient statistic). A statistic $T$ is sufficient for $\mathcal{P}$ if for every $A\in \mathcal{A}$ there is a version of the conditional probability $P_{\theta }(A\mid T)$ that does not depend on $\theta$. Equivalently, there is a Markov kernel $K(\cdot \mid t)$ on $\mathcal{A}$, indexed by $t\in \mathcal{T}$ and free of $\theta$, such that for all $\theta$ and all $A$,

$$P_{\theta }(A)={\int }_{\mathcal{T}}K(A\mid t)P_{\theta }^T(dt),$$

where $P_{\theta }^T=T_{\ast }{P}_{\theta }$ is the law of $T$ under $P_{\theta }$. The kernel $K$ is the replay rule: it regenerates the data from $T$ with no reference to $\theta$.

Definition (dominated family and densities). The family $\mathcal{P}$ is dominated by a $\sigma$-finite measure $\mu$ if $P_{\theta }\ll \mu$ for every $\theta$; write $p_{\theta }=d{P}_{\theta }/d\mu$ for the density. By the Halmos-Savage theorem a dominated family always admits a privileged dominating measure of the form ${\mu }_0={\sum }_i{c}_i{P}_{{\theta }_i}$ (a countable convex combination) with $\mathcal{P}\equiv {\mu }_0$ (mutual absolute continuity of the null sets); densities $p_{\theta }=d{P}_{\theta }/d{\mu }_0$ may be taken with respect to it.

Definition (minimal sufficiency). A sufficient statistic $T$ is minimal sufficient if for every other sufficient statistic $S$ there is a measurable map $f$ with $T=f(S)$ almost surely under every $P_{\theta }$. A minimal sufficient statistic realises the coarsest faithful summary; it is unique up to bijective measurable relabelling.

Definition (exponential family, natural form). A dominated family $\mathcal{P}$ is a $k$-parameter exponential family in canonical (natural) form if there are a measurable natural statistic $T=(T_1,\dots ,T_k):\mathcal{X}\to {\mathbb{R}}^k$, a measurable carrier $h\ge 0$, and a natural parameter $\eta \in {\mathbb{R}}^k$ such that the densities, with respect to a $\sigma$-finite $\mu$, are

$$p_{\eta }(x)=h(x)\exp (⟨\eta ,T(x)⟩-A(\eta )),A(\eta )=\log {\int }_{\mathcal{X}}h(x)e^{⟨\eta ,T(x)⟩}\mu (dx).$$

The log-partition function (or cumulant function) $A$ normalises each density. The natural parameter space is

$$\mathcal{N}=\{\eta \in {\mathbb{R}}^k:\int h{e}^{⟨\eta ,T⟩}d\mu <\infty \},$$

the set on which $A(\eta )<\infty$; $\mathcal{N}$ is convex. A general parametrisation $\theta \mapsto \eta (\theta )$ presents the same densities $p_{\eta (\theta )}$; the family is full (or regular) if $\eta$ ranges over all of $\mathcal{N}$ (which is then assumed to have non-empty interior), and curved if $\theta \mapsto \eta (\theta )$ is a smooth map of a lower-dimensional $\Theta$ into $\mathcal{N}$ whose image is a proper curved submanifold, so the natural parameters are constrained.

Counterexamples to common slips Intermediate+

• Sufficiency is not "captures the mean". The sample mean is sufficient for a normal family with known variance, but for a Cauchy location family no single coordinate is sufficient: the minimal sufficient statistic is the whole order statistic. Reducing to a one-number summary is a property of the model, not a universal right.

• The support must not move with the parameter for the exponential form. The uniform family $\mathrm{Unif}(0,\theta )$ has density ${\theta }^{-1}1\{0\le x\le \theta \}$, which is not of exponential form because the carrier's support depends on $\theta$. Its sufficient statistic ${\max }_i{x}_i$ is genuine, but the exponential-family moment machinery below does not apply.

• Minimal sufficient is coarser than sufficient, not finer. The full data is always sufficient and the full order statistic is sufficient for an i.i.d. model, but neither is minimal in general. Minimality is the coarsest faithful summary, obtained by merging sample points whose likelihood ratios are parameter-free.

• The natural parameter space can be a proper subset, or open, or all of ${\mathbb{R}}^k$. For the Poisson family $\mathcal{N}=\mathbb{R}$; for the Gamma shape-rate family $\mathcal{N}$ is an open half-plane. Differentiating $A$ to get moments is licensed only on $\mathrm{int}\mathcal{N}$, where $A$ is finite and smooth.

Key theorem with proof Intermediate+

The organising result is the factorization criterion: sufficiency is equivalent to a multiplicative split of the likelihood. We state and prove it in the discrete case, where the conditional-distribution definition is concrete, and indicate the dominated extension.

Theorem (Fisher-Neyman factorization). Let $\mathcal{P}=\{P_{\theta }\}$ be dominated by a $\sigma$-finite $\mu$ with densities $p_{\theta }=d{P}_{\theta }/d\mu$, and let $T$ be a statistic. Then $T$ is sufficient for $\mathcal{P}$ if and only if there exist a measurable function $g_{\theta }\ge 0$ depending on $x$ only through $T(x)$, and a measurable $h\ge 0$ free of $\theta$, such that

$$p_{\theta }(x)=g_{\theta }(T(x))h(x)\text{for }\mu \text{-almost every }x\text{ and every }\theta .$$

Proof (discrete case). Take $\mathcal{X}$ countable and $\mu$ counting measure, so $p_{\theta }(x)=P_{\theta }(\{x\})$ and $P_{\theta }(T=t)={\sum }_{x:T(x)=t}{p}_{\theta }(x)$.

($\Rightarrow$) Sufficiency implies factorization. Assume $T$ is sufficient, so $P_{\theta }(X=x\mid T=t)$ has a version $K(x\mid t)$ free of $\theta$. For $x$ with $T(x)=t$,

$$p_{\theta }(x)=P_{\theta }(X=x)=P_{\theta }(X=x\mid T=t)P_{\theta }(T=t)=K(x\mid t)P_{\theta }(T=t).$$

Set $h(x)=K(x\mid T(x))$, which is free of $\theta$, and $g_{\theta }(t)=P_{\theta }(T=t)$, which depends on $x$ only through $t=T(x)$. Then $p_{\theta }(x)=g_{\theta }(T(x))h(x)$, the asserted factorization.

($\Leftarrow$) Factorization implies sufficiency. Assume $p_{\theta }(x)=g_{\theta }(T(x))h(x)$. Compute the law of $T$:

$$P_{\theta }(T=t)=\sum _{x:T(x)=t}{g}_{\theta }(t)h(x)=g_{\theta }(t)\sum _{x:T(x)=t}h(x)=g_{\theta }(t)H(t),H(t):=\sum _{x:T(x)=t}h(x).$$

For $x$ with $T(x)=t$ and $P_{\theta }(T=t)>0$,

$$P_{\theta }(X=x\mid T=t)=\frac{p_{\theta }(x)}{P_{\theta }(T=t)}=\frac{g_{\theta }(t)h(x)}{g_{\theta }(t)H(t)}=\frac{h(x)}{H(t)},$$

because the $g_{\theta }(t)$ factors cancel. The right side is free of $\theta$, so $K(x\mid t)=h(x)/H(t)$ is a parameter-free version of the conditional distribution. Hence $T$ is sufficient. $\square$

For a general dominated family the same equivalence holds with densities against the privileged measure ${\mu }_0$ of Halmos-Savage ^{[Halmos Savage 1949]}; the conditional-distribution definition is realised by the regular conditional distribution constructed from conditional expectation 37.04.01, and the cancellation step becomes the statement that $d{P}_{\theta }/d{\mu }_0$ is $\sigma (T)$-measurable up to the $\theta$-free factor $h$. The exponential-family density $p_{\eta }(x)=h(x)\exp (⟨\eta ,T(x)⟩-A(\eta ))$ is factorized on sight: $g_{\eta }(T(x))=\exp (⟨\eta ,T(x)⟩-A(\eta ))$ depends on $x$ only through $T(x)$, so $T$ is sufficient for every exponential family.

Bridge. The factorization theorem builds toward every later use of data reduction — the Rao-Blackwell improvement of an estimator by conditioning on a sufficient statistic 45.01.06, Basu's theorem on the independence of a complete sufficient statistic from an ancillary 45.01.03, and maximum likelihood in exponential families — and it appears again wherever a likelihood is written down, because the split $p_{\theta }=g_{\theta }(T)h$ is exactly how one reads a sufficient statistic off a model by inspection. The foundational reason the criterion works is that conditioning on $T$ cancels the $\theta$-dependent factor $g_{\theta }(T)$, leaving the parameter-free replay kernel $h/H$; this is exactly the cancellation seen numerically in the coin example, now proved in general. The factorization is dual to the conditional-distribution definition: one is the likelihood-side statement, the other the data-side statement, and putting these together, the central insight is that an exponential family is the canonical case in which the factor $g_{\eta }(T)=\exp (⟨\eta ,T⟩-A(\eta ))$ carries the entire $\eta$-dependence through the inner product $⟨\eta ,T⟩$, so the natural statistic $T$ is automatically sufficient and its dimension stays fixed as the sample grows.

Exercises Intermediate+

Advanced results Master

The factorization theorem in full generality is the Halmos-Savage criterion ^{[Halmos Savage 1949]}. For a family dominated by a $\sigma$-finite $\mu$, one passes to the privileged measure ${\mu }_0={\sum }_i{c}_i{P}_{{\theta }_i}$ with $c_i>0$, $\sum c_i=1$, chosen so that ${\mu }_0$-null sets are exactly the simultaneously-$P_{\theta }$-null sets; such a ${\mu }_0$ exists by a maximal-element argument on the lattice of dominated sub-families. A statistic $T$ is sufficient if and only if the densities $d{P}_{\theta }/d{\mu }_0$ admit $\sigma (T)$-measurable versions for every $\theta$, equivalently if and only if $p_{\theta }=g_{\theta }(T)h$ (${\mu }_0$-a.e.). The measure-theoretic subtlety the discrete proof hides is that one must work with the common dominating ${\mu }_0$ rather than a $\theta$-dependent reference; the regular conditional distribution given $T$ supplied by conditional expectation 37.04.01 is what realises the replay kernel $K(\cdot \mid t)$.

Minimal sufficiency has a clean likelihood-ratio description, the Lehmann-Scheffé criterion ^{[Lehmann Scheffé 1950]}. Define an equivalence on the sample space by $x\sim y$ iff $p_{\theta }(x)/p_{\theta }(y)$ is constant in $\theta$ (interpreting $0/0$ on the common null set as $\sim$). The map sending $x$ to its equivalence class is minimal sufficient: it is sufficient by the factorization built along a reference parameter, and it is coarser than every sufficient statistic because any sufficient $S$ has parameter-free ratios on its level sets. For an i.i.d. sample from a full-rank $k$-parameter exponential family the equivalence classes are precisely the level sets of the natural statistic ${\sum }_{i=1}^{n}T(x_i)$, so this $k$-dimensional statistic is minimal sufficient for every sample size $n\ge k$ — the dimension of the minimal sufficient statistic stays bounded as the data grow. This bounded-dimension phenomenon, due to Pitman, Koopman, and Darmois, characterises exponential families among regular families with fixed support: under smoothness and common support, a family admits a fixed-dimension sufficient statistic for all $n$ if and only if it is exponential.

The analytic core of the exponential-family theory is the regularity of the log-partition function on the interior of the natural parameter space. On $\mathrm{int}\mathcal{N}$ the integral $Z(\eta )=\int h{e}^{⟨\eta ,T⟩}d\mu$ is finite in a complex neighbourhood, hence $A=\log Z$ is real-analytic and even extends holomorphically ^{[Brown 1986]}; differentiation under the integral sign is justified to all orders. The first two derivatives give the moment and cumulant identities

$$\nabla A(\eta )={\mathbb{E}}_{\eta }[T],{\nabla }^{2}A(\eta )={\mathrm{Cov}}_{\eta }(T),$$

so $A$ is convex, and strictly convex on $\mathrm{int}\mathcal{N}$ exactly when $T$ is not concentrated on a proper affine subspace (the family is then minimal / identifiable). The map $\eta \mapsto \mu (\eta ):=\nabla A(\eta )={\mathbb{E}}_{\eta }[T]$ is the mean parametrisation; strict convexity of $A$ makes it a diffeomorphism of $\mathrm{int}\mathcal{N}$ onto the interior of the convex hull of the support of $T$, and its inverse is the gradient of the Legendre-Fenchel conjugate $A^{\ast }(\mu )={\sup }_{\eta }(⟨\eta ,\mu ⟩-A(\eta ))$ — the dual coordinate that information geometry calls the expectation parameter, and which in large deviations is the rate function for the empirical mean of $T$ (cross-reference the convex duality $A\leftrightarrow A^{\ast }$). Higher derivatives of $A$ are the higher cumulants of $T$, which is why $A$ is the cumulant generating function of $T$ under the carrier law.

Curved exponential families arise when a smooth lower-dimensional parameter $\theta \in \Theta \subseteq {\mathbb{R}}^d$ enters through $\eta =\eta (\theta )$ with image a $d$-dimensional curved submanifold of the $k$-dimensional $\mathcal{N}$, $d<k$ ^{[Brown 1986]}. The natural statistic ${\sum }_{i}T(x_i)$ remains sufficient and minimal for the enveloping full family, but it is no longer a complete sufficient statistic for the curved subfamily, and the maximum likelihood estimator solves the constrained likelihood equation $\nabla \eta (\theta {)}^{\top }({\sum }_{i}T(x_i)-n\nabla A(\eta (\theta )))=0$ — the projection of the sample mean of $T$ onto the tangent space of the curve in the mean parametrisation. The curvature of this submanifold (Efron's statistical curvature) controls the second-order efficiency loss of estimation, making curved families the natural setting for higher-order asymptotic theory.

Synthesis. The foundational reason sufficiency, factorization, and exponential families form one circle is that the conditional-distribution definition, the likelihood split $p_{\theta }=g_{\theta }(T)h$, and the canonical density $h{e}^{⟨\eta ,T⟩-A(\eta )}$ are three readings of the single fact that the $\theta$-dependence of a model can be quarantined inside a statistic. The factorization is dual to the conditional-distribution definition exactly as the likelihood side is dual to the data side; the exponential family is the canonical case, and this is exactly where the quarantine is linear — the parameter enters only through the inner product $⟨\eta ,T⟩$, so the log-partition function $A$ becomes a cumulant generating function whose convexity, established by differentiating under the integral, generalises the elementary mean-and-variance computations to all cumulants at once. Putting these together, the central insight is that the convex-conjugate pair $(A,A^{\ast })$ supplies the dual flat coordinates — natural parameter $\eta$ and mean parameter $\mu =\nabla A(\eta )$ — on which Rao-Blackwell improvement, complete-sufficient estimation, and maximum likelihood all become projections, and the bridge is the bounded-dimension theorem of Pitman-Koopman-Darmois: a fixed-size faithful summary exists for all sample sizes if and only if the model is exponential, which is the structural fact that makes exponential families the load-bearing examples of the entire estimation theory that builds toward Rao-Blackwell 45.01.06, Basu 45.01.03, and the large-deviation rate function $A^{\ast }$.

Full proof set Master

The factorization theorem (discrete) and the moment identities (Exercises 4-5) are proved above. The remaining Master claims are recorded here.

Proposition 1 (existence of the privileged dominating measure). If $\mathcal{P}=\{P_{\theta }\}$ is dominated by a $\sigma$-finite $\mu$, there exist ${\theta }_1,{\theta }_2,\dots$ and weights $c_i>0$ with ${\sum }_i{c}_i=1$ such that ${\mu }_0={\sum }_i{c}_i{P}_{{\theta }_i}$ satisfies: a set is ${\mu }_0$-null iff it is $P_{\theta }$-null for every $\theta$.

Proof. Call a probability measure of the form $Q={\sum }_i{c}_i{P}_{{\theta }_i}$ (countable convex combination) a test measure. Among test measures consider the collection of their null-set classes; a set $N$ is $Q$-null iff each $P_{{\theta }_i}$ in the support gives it measure zero. Since $\mu$ is $\sigma$-finite we may assume $\mu$ a probability measure (re-weight on a countable partition into finite-measure pieces). Maximise the $\mu$-measure $\mu (\{p_{\theta }>0\text{ for the chosen combination}\})$: choose ${\theta }_i$ greedily so that the support $S_0=\{d{\mu }_0/d\mu >0\}$ has $\mu (S_0)$ within $2^{-i}$ of the supremum of $\mu (\bigcup \{p_{\theta }>0\})$ over countable selections. The supremum is attained by a countable union because $\mu$ is finite, so ${\mu }_0={\sum }_i{c}_i{P}_{{\theta }_i}$ with the selected ${\theta }_i$ has support $S_0$ equal (mod $\mu$-null) to ${\bigcup }_{\theta }\{p_{\theta }>0\}$. For any $\theta$, $P_{\theta }(\{d{\mu }_0/d\mu =0\})={\int }_{S_0^c}{p}_{\theta }d\mu =0$ by maximality (else adding $\theta$ would enlarge $S_0$). Hence $P_{\theta }\ll {\mu }_0$ for all $\theta$, and a ${\mu }_0$-null set $N$ has $P_{{\theta }_i}(N)=0$ for the chosen ${\theta }_i$; the maximality forces $P_{\theta }(N)=0$ for all $\theta$. The converse is immediate. $\square$

Proposition 2 (convexity of the natural parameter space and of $A$). The set $\mathcal{N}=\{\eta :Z(\eta )<\infty \}$ is convex, and $A=\log Z$ is convex on $\mathcal{N}$.

Proof. For ${\eta }_1,{\eta }_2\in \mathcal{N}$ and $\alpha \in [0,1]$, Hölder's inequality with conjugate exponents $1/\alpha$ and $1/(1-\alpha )$ applied to the carrier measure $hd\mu$ gives

$$Z(\alpha {\eta }_1+(1-\alpha ){\eta }_2)=\int h{e}^{⟨\alpha {\eta }_1+(1-\alpha ){\eta }_2,T⟩}d\mu \le (\int h{e}^{⟨{\eta }_1,T⟩}d\mu {)}^{\alpha }(\int h{e}^{⟨{\eta }_2,T⟩}d\mu {)}^{1-\alpha }=Z({\eta }_1{)}^{\alpha }Z({\eta }_2{)}^{1-\alpha },$$

finite, so $\alpha {\eta }_1+(1-\alpha ){\eta }_2\in \mathcal{N}$: the set is convex. Taking logarithms, $A(\alpha {\eta }_1+(1-\alpha ){\eta }_2)\le \alpha A({\eta }_1)+(1-\alpha )A({\eta }_2)$, the convexity of $A$. $\square$

Proposition 3 (smoothness and the cumulant identities). On $\mathrm{int}\mathcal{N}$, $A$ is $C^{\infty }$ (indeed real-analytic), with $\nabla A(\eta )={\mathbb{E}}_{\eta }[T]$ and ${\nabla }^{2}A(\eta )={\mathrm{Cov}}_{\eta }(T)$; $A$ is strictly convex iff $T$ is not supported in a proper affine subspace.

Proof. Fix ${\eta }_0\in \mathrm{int}\mathcal{N}$ and a closed ball $\bar{B}({\eta }_0,r)\subset \mathcal{N}$. For $\eta \in B({\eta }_0,r/2)$ and any multi-index, the integrand $T^{\alpha }h{e}^{⟨\eta ,T⟩}$ is dominated by $ch{e}^{⟨{\eta }_0,T⟩}(e^{⟨{\eta }^{\prime },T⟩}+e^{⟨{\eta }^{\prime \prime },T⟩})$ for suitable ${\eta }^{\prime },{\eta }^{\prime \prime }\in \mathcal{N}$, using that polynomials are absorbed by the exponential gap $r/2$ and that $e^{⟨\eta ,T⟩}\le e^{⟨{\eta }^{\prime },T⟩}+e^{⟨{\eta }^{\prime \prime },T⟩}$ on the ball; this dominating function is $\mu$-integrable because ${\eta }^{\prime },{\eta }^{\prime \prime }\in \mathcal{N}$. Dominated convergence then licenses differentiation under the integral to all orders, giving ${\partial }_{{\eta }_j}Z=\int T_{j}h{e}^{⟨\eta ,T⟩}d\mu$ and inductively all moments of $T$. Hence $\nabla A=\nabla Z/Z={\mathbb{E}}_{\eta }[T]$ (Exercise 4) and ${\nabla }^{2}A={\mathrm{Cov}}_{\eta }(T)$ (Exercise 5). Real-analyticity follows because the same domination holds for complex $\eta$ in a tube $\{\mathrm{Re}\eta \in \mathrm{int}\mathcal{N}\}$, so $Z$ is holomorphic there and $A=\log Z$ is analytic where $Z\ne 0$ (which holds since $Z>0$). The Hessian ${\mathrm{Cov}}_{\eta }(T)$ is positive definite unless $⟨v,T⟩$ is $P_{\eta }$-a.s. constant for some $v\ne 0$, i.e. unless $T$ lies in a proper affine subspace; in the contrary case $A$ is strictly convex. $\square$

Proposition 4 (sufficiency of the natural statistic; minimality in the full-rank case). For an exponential family $p_{\eta }=h{e}^{⟨\eta ,T⟩-A(\eta )}$, $T$ is sufficient; for an i.i.d. sample of size $n$ from a full-rank family ($\mathcal{N}$ has non-empty interior and ${\mathrm{Cov}}_{\eta }(T)\succ 0$), $T_n:={\sum }_{i=1}^{n}T(x_i)$ is minimal sufficient.

Proof. The joint density of $x_1,\dots ,x_n$ is ${\prod }_{i}h(x_i)\exp (⟨\eta ,{\sum }_{i}T(x_i)⟩-nA(\eta ))=({\prod }_{i}h(x_i))g_{\eta }(T_n)$ with $g_{\eta }(t)=e^{⟨\eta ,t⟩-nA(\eta )}$ depending on the data only through $T_n$. By Fisher-Neyman, $T_n$ is sufficient. For minimality, apply the Lehmann-Scheffé criterion: the likelihood ratio at two samples $x,y$ is

$$\frac{p_{\eta }(x)}{p_{\eta }(y)}=\frac{{\prod }_{i}h(x_i)}{{\prod }_{i}h(y_i)}\exp (⟨\eta ,T_n(x)-T_n(y)⟩),$$

which is free of $\eta$ iff $⟨\eta ,T_n(x)-T_n(y)⟩$ is constant in $\eta$ over the open set $\mathrm{int}\mathcal{N}$, i.e. iff $T_n(x)=T_n(y)$ (an open set of $\eta$ separates distinct vectors). Thus the equivalence classes of the Lehmann-Scheffé relation are exactly the level sets of $T_n$, so $T_n$ is minimal sufficient. $\square$

Connections Master

The probability-space and measurability foundations of 37.01.01 are what make a statistical model $\{P_{\theta }\}$ and a statistic $T$ well-defined objects: sufficiency is a statement about the laws $P_{\theta }^T=T_{\ast }{P}_{\theta }$ and the regular conditional distribution given $\sigma (T)$, both constructed in the Kolmogorov framework of that unit, and the dominated-family hypothesis is absolute continuity of measures in the same sense.

The conditional-expectation apparatus of 37.04.01 is load-bearing twice over: the very definition of sufficiency is the $\theta$-freeness of a version of $P_{\theta }(A\mid T)={\mathbb{E}}_{\theta }[1_A\mid T]$, and the replay kernel $K(\cdot \mid t)$ is the regular conditional distribution built from that conditional expectation; the Radon-Nikodym construction there also supplies the densities $d{P}_{\theta }/d{\mu }_0$ on which the Halmos-Savage factorization is stated.

The random-variable and expectation theory of 26.03.01 is where the moment identities live: $\nabla A(\eta )={\mathbb{E}}_{\eta }[T]$ and ${\nabla }^{2}A(\eta )={\mathrm{Cov}}_{\eta }(T)$ are statements about expectations and covariances of the natural statistic under the tilted laws $p_{\eta }$, and the convexity of $A$ is the Jensen/Hölder behaviour of those expectations.

The Rao-Blackwell theorem 45.01.06 consumes this unit directly: it improves an unbiased estimator by conditioning on a sufficient statistic, and the variance never increases precisely because conditioning on a sufficient $T$ removes only $\theta$-free noise — the factorization $p_{\theta }=g_{\theta }(T)h$ proved here is what guarantees the conditioned estimator is still a statistic (free of $\theta$) and hence usable.

Basu's theorem 45.01.03 pairs completeness of a sufficient statistic with ancillarity, and exponential families are its canonical home: a full-rank exponential family's natural statistic is complete and sufficient, so Basu's theorem makes it independent of every ancillary statistic, a fact used throughout distributional calculations built on the convexity and analyticity of $A$ established here.

The convex-duality machinery of the Legendre-Fenchel transform appears as the conjugate $A^{\ast }(\mu )={\sup }_{\eta }(⟨\eta ,\mu ⟩-A(\eta ))$ of the log-partition function: $A^{\ast }$ is the rate function for the empirical mean of the natural statistic in Cramér-type large deviations 37.07.03, and the mean parametrisation $\mu =\nabla A(\eta )$ with inverse $\eta =\nabla A^{\ast }(\mu )$ is the dual-flat coordinate system of information geometry.

Historical & philosophical context Master

The concept of sufficiency is due to Ronald A. Fisher, who introduced it in his 1922 memoir On the mathematical foundations of theoretical statistics (Phil. Trans. R. Soc. A 222) and stated the factorization idea informally; Jerzy Neyman gave the factorization criterion explicit form in the 1930s, whence the name Fisher-Neyman. The rigorous measure-theoretic treatment — replacing the discrete and density arguments by a Radon-Nikodym factorization valid for general dominated families — was supplied by Paul Halmos and Leonard Savage in Application of the Radon-Nikodym theorem to the theory of sufficient statistics (Ann. Math. Statist. 20, 1949, 225) ^{[Halmos Savage 1949]}, which introduced the privileged dominating measure used in Proposition 1.

Minimal sufficiency and its likelihood-ratio criterion are due to Erich Lehmann and Henry Scheffé in Completeness, similar regions, and unbiased estimation (Sankhyā 10, 1950, 305) ^{[Lehmann Scheffé 1950]}, where completeness was also introduced as the property securing uniqueness of unbiased estimators. The characterisation of exponential families as the regular families admitting a fixed-dimension sufficient statistic for all sample sizes was found independently by E. J. G. Pitman, B. O. Koopman, and Georges Darmois around 1935-1936. The systematic analytic theory of the natural parameter space, the analyticity of the log-partition function, and the moment identities by differentiation was given its definitive modern form by Lawrence Brown in Fundamentals of Statistical Exponential Families (1986) ^{[Brown 1986]}, building on the convex-analytic viewpoint that ties $A$ and its conjugate $A^{\ast }$ to the dual coordinates later central in Amari's information geometry. Exponential families occupy this central position because the linearity of the parameter in the exponent makes the entire inferential calculus — sufficiency, completeness, conjugate priors, maximum likelihood, and large-deviation asymptotics — computable from a single convex function.

Bibliography Master

@book{casellaberger2002,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2nd},
  publisher = {Duxbury},
  year      = {2002}
}

@book{lehmanncasella1998,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2nd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}

@book{brown1986,
  author    = {Brown, Lawrence D.},
  title     = {Fundamentals of Statistical Exponential Families, with Applications in Statistical Decision Theory},
  series    = {IMS Lecture Notes-Monograph Series},
  volume    = {9},
  publisher = {Institute of Mathematical Statistics},
  year      = {1986}
}

@article{halmossavage1949,
  author  = {Halmos, Paul R. and Savage, Leonard J.},
  title   = {Application of the Radon-Nikodym theorem to the theory of sufficient statistics},
  journal = {Annals of Mathematical Statistics},
  volume  = {20},
  number  = {2},
  pages   = {225--241},
  year    = {1949}
}

@article{lehmannscheffe1950,
  author  = {Lehmann, Erich L. and Scheff\'e, Henry},
  title   = {Completeness, similar regions, and unbiased estimation. Part I},
  journal = {Sankhy\={a}},
  volume  = {10},
  number  = {4},
  pages   = {305--340},
  year    = {1950}
}

@article{fisher1922,
  author  = {Fisher, Ronald A.},
  title   = {On the mathematical foundations of theoretical statistics},
  journal = {Philosophical Transactions of the Royal Society A},
  volume  = {222},
  pages   = {309--368},
  year    = {1922}
}

@book{barndorffnielsen1978,
  author    = {Barndorff-Nielsen, Ole},
  title     = {Information and Exponential Families in Statistical Theory},
  publisher = {Wiley},
  year      = {1978}
}