Conjugate Priors and Exponential-Family Bayesian Inference

Intuition Beginner

A Bayesian analysis starts with a prior — your belief about an unknown quantity before seeing data — and updates it with the data to produce a posterior. The update is honest but messy: usually you must compute an awkward integral just to normalise the posterior, and the result may be a distribution with no name. For a few special pairings of prior and data, something nicer happens. The posterior comes out in the same family as the prior, only with its dials turned. A beta prior meets coin-flip data and gives a beta posterior; a gamma prior meets count data and gives a gamma posterior. A prior family with this self-reproducing property is called conjugate.

Conjugacy is convenient, but the deeper reason it matters is the picture it gives you. The prior behaves like a batch of imaginary observations you bring to the table before the real data arrive. Updating just means adding the real observations to the imaginary ones and re-reading the same dials. If your prior is worth five pretend coin flips and the data bring fifty real ones, the data dominate. If your prior is worth five hundred pretend flips, your prior wins until a great deal of real data arrives.

This is why the conjugate posterior estimate always lands between your prior guess and what the data alone would say. It is a blend, weighted by how much each side knows. The blend shifts toward the data as data accumulate, which is the same shrinkage idea behind ridge regression and many modern methods.

Visual Beginner

Picture the conjugate update as a single bookkeeping table. The prior carries a small set of pseudo-counts; the data carry their own counts; the posterior is the row-by-row sum. The shape of the distribution never changes — only the numbers inside it move.

Quantity	Prior (pseudo-data)	Data (real)	Posterior (sum)
Coin flips: successes	$\alpha =2$	$x=8$	$\alpha +x=10$
Coin flips: failures	$\beta =2$	$n-x=2$	$\beta +n-x=4$
Best guess of rate	$2/4=0.50$	$8/10=0.80$	$10/14\approx 0.71$

The posterior rate $0.71$ sits between the prior guess $0.50$ and the data's $0.80$, pulled toward the data because ten real flips outweigh four pseudo-flips. Add more real data and the posterior slides further toward $0.80$.

Worked example Beginner

A factory line produces occasional defects. You model the number of defects per shift as Poisson with unknown rate $\lambda$. From past lines you believe $\lambda$ is around $3$, and you encode this as a Gamma prior with shape $\alpha =6$ and rate $\beta =2$, whose mean is $\alpha /\beta =3$. The gamma is conjugate to the Poisson, so the posterior will again be a gamma.

You now observe four shifts with $2,4,3,5$ defects, a total of $S=14$ defects over $n=4$ shifts.

Step 1. Update the shape. The conjugate rule adds the total count to the prior shape: posterior shape $=\alpha +S=6+14=20$.

Step 2. Update the rate. The rule adds the number of shifts to the prior rate: posterior rate $=\beta +n=2+4=6$.

Step 3. Read the posterior estimate. The posterior is $\text{Gamma}(20,6)$, with mean $20/6\approx 3.33$.

Step 4. Compare the three numbers. The prior mean was $3.00$. The data alone (the average count) say $14/4=3.50$. The posterior mean $3.33$ sits between them.

What this tells us. The prior acted like $\beta =2$ shifts' worth of pseudo-data carrying $\alpha =6$ pseudo-defects. The real data added $4$ shifts and $14$ defects. The posterior simply pooled the two, and the estimate landed between belief and data — closer to the data because four real shifts slightly outweigh two pseudo-shifts. With forty real shifts instead of four, the posterior mean would sit almost on top of the data average.

Check your understanding Beginner

Formal definition Intermediate+

Fix a statistical model $\{P_{\theta }:\theta \in \Theta \}$ dominated by a $\sigma$-finite measure, with likelihood $f(x\mid \theta )$, and a prior probability measure $\pi$ on $\Theta$. The posterior is $\pi (\theta \mid x)\propto f(x\mid \theta )\pi (\theta )$, normalised by the marginal $m(x)={\int }_{\Theta }f(x\mid \theta )\pi (\theta )d\theta$ (the disintegration recalled in 45.03.01).

Definition (conjugate family). Let $\mathcal{F}=\{{\pi }_{\xi }:\xi \in \Xi \}$ be a family of priors on $\Theta$ indexed by a hyperparameter $\xi$. The family $\mathcal{F}$ is conjugate to the likelihood $f(\cdot \mid \theta )$ if for every $\xi \in \Xi$ and every $x$ with $m(x)>0$ there is a hyperparameter ${\xi }^{\prime }(x,\xi )\in \Xi$ such that

$${\pi }_{\xi }(\theta \mid x)={\pi }_{{\xi }^{\prime }(x,\xi )}(\theta ).$$

The map $\xi \mapsto {\xi }^{\prime }(x,\xi )$ is the hyperparameter update. Conjugacy is a property of the pair (prior family, likelihood); a family conjugate to one likelihood need not be conjugate to another.

The exponential-family construction. Let the likelihood be a $k$-parameter exponential family in natural form (from 45.01.02),

$$f(x\mid \eta )=h(x)\exp (⟨\eta ,T(x)⟩-A(\eta )),\eta \in \mathcal{N}=\{\eta :A(\eta )<\infty \},$$

with natural statistic $T$, log-partition function $A$, and convex natural parameter space $\mathcal{N}$. The natural conjugate prior on the natural parameter $\eta$ is

$${\pi }_{\nu ,\kappa }(\eta )=C(\nu ,\kappa )\exp (⟨\nu ,\eta ⟩-\kappa A(\eta )),(\nu ,\kappa )\in \Xi ,$$

where $\Xi =\{(\nu ,\kappa ):{\int }_{\mathcal{N}}\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))d\eta <\infty \}$ is the set of hyperparameters for which the prior is a proper probability density and $C(\nu ,\kappa )$ is the normalising constant. The hyperparameter $\kappa$ plays the role of a prior sample size and $\nu$ of a prior sufficient-statistic total: $\nu /\kappa$ is the prior guess for the mean of $T$.

The posterior update. With a single observation $x$, the posterior is again of conjugate form with

$$\nu \mapsto \nu +T(x),\kappa \mapsto \kappa +1;$$

for an i.i.d. sample $x_1,\dots ,x_n$ the update is $\nu \mapsto \nu +{\sum }_{i=1}^{n}T(x_i)$ and $\kappa \mapsto \kappa +n$. The prior contributes $\kappa$ pseudo-observations carrying a pooled natural statistic $\nu$; the data contribute $n$ observations carrying ${\sum }_{i}T(x_i)$, and the posterior pools the two.

Definition (posterior predictive distribution). The posterior predictive density of a future observation $\tilde{x}$ given data $x$ is

$$p(\tilde{x}\mid x)={\int }_{\mathcal{N}}f(\tilde{x}\mid \eta )\pi (\eta \mid x)d\eta ,$$

the data-generating density averaged over the posterior; it is the marginal of $\tilde{x}$ under the updated model and propagates parameter uncertainty into predictions.

Counterexamples to common slips Intermediate+

• A conjugate family is not unique, and mixtures stay conjugate. Any finite mixture of natural conjugate priors ${\sum }_j{w}_j{\pi }_{{\nu }_j,{\kappa }_j}$ has a posterior that is again a mixture of conjugate priors (with reweighted mixing weights), so "conjugate" never means a single canonical prior. Mixtures of conjugates can approximate any prior while preserving closed-form updates.

• The natural conjugate prior lives on $\eta$, not necessarily on the original $\theta$. Conjugacy is a statement in the natural parametrisation; reparametrising to a mean or scale parameter transforms the prior by a Jacobian, so the same prior need not look like a member of a "named" family in the original coordinates. The beta, gamma, and Dirichlet are the natural conjugates re-expressed in convenient coordinates.

• Conjugacy does not require a flat or noninformative prior. Setting $\kappa \to 0$ to seek "no information" can push $(\nu ,\kappa )$ outside $\Xi$ and yield an improper prior whose posterior may also be improper. Propriety of the conjugate prior is a condition on $(\nu ,\kappa )$, not an automatic feature of the form.

Key theorem with proof Intermediate+

The organising result is that the exponential-family conjugate prior closes the Bayesian update into hyperparameter arithmetic, and that the conjugacy is reflected in a linear posterior expectation of the mean parameter.

Theorem (exponential-family conjugacy and the Diaconis-Ylvisaker linear posterior expectation). Let $f(x\mid \eta )=h(x)\exp (⟨\eta ,T(x)⟩-A(\eta ))$ be a full exponential family and let ${\pi }_{\nu ,\kappa }(\eta )\propto \exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$ be the natural conjugate prior with $(\nu ,\kappa )\in \Xi$.

(i) (Closure.) For data $x_1,\dots ,x_n$ the posterior is ${\pi }_{\nu +S,\kappa +n}$ with $S={\sum }_{i=1}^{n}T(x_i)$, provided $(\nu +S,\kappa +n)\in \Xi$.

(ii) (Linear posterior expectation.) For $(\nu ,\kappa )\in \mathrm{int}\Xi$ with $\kappa >0$, the prior mean of the mean parameter $\mu (\eta ):=\nabla A(\eta )={\mathbb{E}}_{\eta }[T]$ is ${\mathbb{E}}_{\pi }[\mu (\eta )]=\nu /\kappa$, and after one observation the posterior mean is

$${\mathbb{E}}_{\pi }[\mu (\eta )\mid x]=\frac{\nu +T(x)}{\kappa +1}.$$

This posterior expectation is affine in $T(x)$; conversely, for a full family with $T$ not affinely degenerate, linearity of ${\mathbb{E}}_{\pi }[\mu (\eta )\mid x]$ in $T(x)$ forces $\pi$ to be of the natural conjugate form.

Proof of (i). The likelihood of the sample is ${\prod }_{i}h(x_i)\exp (⟨\eta ,S⟩-nA(\eta ))$ with $S={\sum }_{i}T(x_i)$. Multiplying by the prior,

$$\pi (\eta \mid x)\propto \exp (⟨\eta ,S⟩-nA(\eta ))\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))=\exp (⟨\nu +S,\eta ⟩-(\kappa +n)A(\eta )),$$

the carrier ${\prod }_{i}h(x_i)$ being a constant in $\eta$ absorbed into normalisation. The right side is, up to its normalising constant, exactly ${\pi }_{\nu +S,\kappa +n}(\eta )$, which is a proper density precisely when $(\nu +S,\kappa +n)\in \Xi$. $\square$

Proof of (ii). Write the conjugate density as ${\pi }_{\nu ,\kappa }(\eta )=C(\nu ,\kappa )\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$. The normaliser satisfies $C(\nu ,\kappa {)}^{-1}={\int }_{\mathcal{N}}\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))d\eta$. The key identity comes from integrating the divergence of the density. Consider the vector field $\eta \mapsto {\pi }_{\nu ,\kappa }(\eta )$; on the interior of $\mathcal{N}$ the density decays at the boundary for $(\nu ,\kappa )\in \mathrm{int}\Xi$, so the integral of its gradient vanishes:

$$0={\int }_{\mathcal{N}}{\nabla }_{\eta }{\pi }_{\nu ,\kappa }(\eta )d\eta ={\int }_{\mathcal{N}}(\nu -\kappa \nabla A(\eta )){\pi }_{\nu ,\kappa }(\eta )d\eta =\nu -\kappa {\mathbb{E}}_{\pi }[\nabla A(\eta )].$$

Rearranging, ${\mathbb{E}}_{\pi }[\mu (\eta )]={\mathbb{E}}_{\pi }[\nabla A(\eta )]=\nu /\kappa$. Applying the same identity to the posterior ${\pi }_{\nu +T(x),\kappa +1}$ from part (i) gives ${\mathbb{E}}_{\pi }[\mu (\eta )\mid x]=(\nu +T(x))/(\kappa +1)$, affine in $T(x)$ with slope $1/(\kappa +1)$.

For the converse, suppose ${\mathbb{E}}_{\pi }[\mu (\eta )\mid x]$ is affine in $T(x)$ for $m$-almost every $x$. The boundary identity above shows that for any prior $\pi$ on $\eta$ with vanishing boundary flux, ${\mathbb{E}}_{\pi }[\nabla A(\eta )\mid x]={\nabla }_{\nu }\log \left(\text{posterior normaliser}\right)$, and Diaconis and Ylvisaker prove that requiring this gradient to be affine in $T(x)$ across the data range pins the posterior normaliser's $\nu$-dependence to the form $⟨\nu ,\eta ⟩-\kappa A(\eta )$, hence $\pi$ to the natural conjugate density (up to the constraint $T$ is not concentrated on a proper affine subspace, the non-degeneracy hypothesis) ^{[Diaconis Ylvisaker 1979]}. $\square$

Bridge. This theorem builds toward the standard conjugate pairs and the posterior-predictive computations of the Advanced section, and it appears again in ridge regression as a conjugate normal-normal posterior mean 45.06.03 and in empirical Bayes where the marginal of the conjugate model is the objective 45.04.05. This is exactly the device that turns the Bayesian integral into hyperparameter addition: the linearity of the exponent in $\eta$ inherited from the exponential family of 45.01.02 is what lets the prior's $(\nu ,\kappa )$ and the data's $(S,n)$ pool by simple summation. The central insight is that conjugacy is sufficiency read on the prior side — the natural statistic $T$ that quarantines the parameter in the likelihood is the same $T$ that the prior absorbs, so a fixed-dimension prior stays fixed-dimension under unlimited data. Putting these together, the affine posterior expectation ${\mathbb{E}}_{\pi }[\mu (\eta )\mid x]=(\nu +T(x))/(\kappa +1)$ exhibits the shrinkage of 45.03.01 as a convex combination of the prior mean $\nu /\kappa$ and the datum $T(x)$, and the slope $1/(\kappa +1)$ is the prior-sample-size dial that generalises the beta-binomial, gamma-Poisson, and normal-normal weights into one formula, the prior precision $\kappa$ playing in the next chapter the role that is dual to the ridge regularisation strength.

Exercises Intermediate+

Advanced results Master

The natural conjugate prior and the parametric structure of $\Xi$

The natural conjugate prior ${\pi }_{\nu ,\kappa }(\eta )\propto \exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$ inherits its analytic life from the log-partition function $A$ of 45.01.02. Its hyperparameter domain $\Xi =\{(\nu ,\kappa ):{\int }_{\mathcal{N}}{e}^{⟨\nu ,\eta ⟩-\kappa A(\eta )}d\eta <\infty \}$ is convex, by the same Hölder argument that makes $\mathcal{N}$ convex, and the propriety of the prior is exactly membership in $\Xi$ ^{[Diaconis Ylvisaker 1979]}. The Diaconis-Ylvisaker theorem characterises the natural conjugate among all priors by the affineness of the posterior expectation ${\mathbb{E}}_{\pi }[\nabla A(\eta )\mid x]=(\nu +T(x))/(\kappa +1)$ in the natural statistic: a full exponential family with $T$ not affinely concentrated admits a prior with posterior mean of the mean parameter affine in $T(x)$ if and only if that prior is the natural conjugate, and then the slope is the universal $1/(\kappa +1)$. The mean parameter $\mu =\nabla A(\eta )$ is the bridge to interpretation: $\nu /\kappa$ is the prior guess for $\mu$, and the posterior pulls it toward the data's $T(x)$ with weight fixed by $\kappa$.

The standard pairs are the catalogue of this one construction. The beta is the natural conjugate of the Bernoulli/binomial, the gamma of the Poisson and of the exponential rate, the Dirichlet of the multinomial; for the normal with both parameters unknown the conjugate is the normal-inverse-gamma in one dimension and the normal-inverse-Wishart in $d$ dimensions, factoring as a prior on the variance (or covariance) times a conditionally normal prior on the mean given the variance, which is the only conjugate structure compatible with the joint exponential form $T=(\sum x_i,\sum x_i{x}_i^{\top })$.

Posterior predictive distributions and the marginal likelihood

The posterior predictive $p(\tilde{x}\mid x)=\int f(\tilde{x}\mid \eta )\pi (\eta \mid x)d\eta$ is computable in closed form for conjugate models because the integral is a ratio of conjugate normalisers. With $\pi (\eta \mid x)={\pi }_{{\nu }^{\prime },{\kappa }^{\prime }}$ the predictive of a future natural-statistic increment is $C({\nu }^{\prime },{\kappa }^{\prime })/C({\nu }^{\prime }+T(\tilde{x}),{\kappa }^{\prime }+1)$ times the carrier, so the beta-binomial, the gamma-Poisson predictive (a negative binomial), and the normal predictive (a Student-$t$, with the inflated variance recording parameter uncertainty) all drop out of the same $C(\cdot )$ algebra. The same ratio gives the marginal likelihood $m(x)=\int {\prod }_{i}f(x_i\mid \eta )\pi (\eta )d\eta =({\prod }_{i}h(x_i))C(\nu ,\kappa )/C(\nu +S,\kappa +n)$, the evidence used for model comparison and the objective that empirical Bayes maximises over $(\nu ,\kappa )$ ^{[Bernardo Smith 1994]}. The predictive inflates the plug-in variance precisely by the posterior variance of the mean parameter — predicting from the posterior is strictly more honest than predicting from a point estimate, and the gap closes at the parametric rate as $n$ grows.

Noninformative and Jeffreys priors, with propriety

When prior information is weak one seeks a default prior. The flat prior $\pi (\theta )\propto 1$ is not invariant under reparametrisation and can fail propriety; Jeffreys' rule $\pi (\theta )\propto \sqrt{\det I(\theta )}$, with $I$ the Fisher information, is reparametrisation-invariant by construction (the information transforms by the squared Jacobian, whose square root cancels the change-of-variables Jacobian), which is its principal virtue ^{[Robert 2007]}. For the binomial Jeffreys is the proper $\mathrm{Beta}(\frac{1}{2},\frac{1}{2})$; for a location parameter it is the improper flat prior; for a scale it is the improper $1/\sigma$. Impropriety is tolerable when the formal posterior is proper, but it must be checked: an improper prior can yield an improper posterior, in which case no inference exists, and in hierarchical models improper hyperpriors are a frequent silent cause of improper joint posteriors. Jeffreys priors also lose their appeal in multiparameter settings, where the joint Jeffreys rule can give poor marginal behaviour and is superseded by reference priors (Bernardo) that maximise the expected Kullback-Leibler divergence from prior to posterior. The honest summary is that there is no unique noninformative prior; the choice is a modelling decision constrained by invariance and propriety.

Limitations of conjugacy

Conjugacy is a computational luxury, not a modelling principle, and its reach is narrow. It is essentially confined to exponential-family likelihoods with their natural conjugates; the moment a model leaves that class — a logistic regression with a normal prior, a hierarchical model with non-conjugate hyperpriors, a mixture model, a likelihood with a parameter-dependent support — the posterior loses closed form and the normalising integral becomes intractable. Even within exponential families the conjugate prior may be too rigid to encode genuine prior beliefs, since its shape is dictated by $A$ rather than by the analyst; mixtures of conjugates buy flexibility at the cost of growing mixture components under repeated updating. The general resolution is to abandon closed forms and sample from the posterior, which is the motivation for Markov chain Monte Carlo 45.03.05: Gibbs sampling exploits conditional conjugacy (each full conditional is conjugate even when the joint is not), and Metropolis-Hastings handles the fully non-conjugate case. Conjugacy thus survives in modern practice less as a way to avoid computation than as the building block of the conditional updates inside a sampler.

Synthesis. The central insight is that conjugacy is sufficiency viewed from the prior: the natural statistic $T$ that quarantines the parameter in the exponential-family likelihood of 45.01.02 is the very object the natural conjugate prior absorbs, so this is exactly why the Bayesian update collapses to hyperparameter addition $(\nu ,\kappa )\mapsto (\nu +S,\kappa +n)$. The foundational reason the standard pairs — beta-binomial, gamma-Poisson, normal-normal, normal-inverse-Wishart, Dirichlet-multinomial — are one phenomenon is that each is the natural conjugate $\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$ for its family's $A$, and the Diaconis-Ylvisaker linear posterior expectation $(\nu +T(x))/(\kappa +1)$ generalises their separate shrinkage formulas into one affine rule. Putting these together, the conjugate posterior mean is the convex combination realising the shrinkage of 45.03.01, and it is dual to the ridge penalty of 45.06.03 under prior-precision equals regularisation-strength, while the closed-form marginal $m(x)=({\prod }_{i}h)C(\nu ,\kappa )/C(\nu +S,\kappa +n)$ appears again in empirical Bayes 45.04.05. The bridge is the log-partition function $A$ and its conjugate-normaliser $C$: conjugacy, the posterior predictive, the marginal likelihood, and the Jeffreys choice are one theory about the analytic structure of $A$, and the boundary where it ends — non-exponential or hierarchical models with no tractable $C$ — is where Markov chain Monte Carlo 45.03.05 takes over.

Full proof set Master

Proposition 1 (closure of the natural conjugate prior under updating). For a full exponential family $f(x\mid \eta )=h(x)\exp (⟨\eta ,T(x)⟩-A(\eta ))$ and prior ${\pi }_{\nu ,\kappa }\propto \exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$ with $(\nu ,\kappa )\in \Xi$, the posterior given i.i.d. $x_1,\dots ,x_n$ is ${\pi }_{\nu +S,\kappa +n}$ with $S={\sum }_{i}T(x_i)$, proper iff $(\nu +S,\kappa +n)\in \Xi$.

Proof. The sample likelihood is $({\prod }_{i}h(x_i))\exp (⟨\eta ,S⟩-nA(\eta ))$. Bayes' rule gives $\pi (\eta \mid x)\propto \exp (⟨\eta ,S⟩-nA(\eta ))\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))=\exp (⟨\nu +S,\eta ⟩-(\kappa +n)A(\eta ))$, the $\eta$-free carrier absorbed into the normaliser. This is ${\pi }_{\nu +S,\kappa +n}$ up to normalisation, which exists as a probability density iff its kernel is integrable over $\mathcal{N}$, i.e. iff $(\nu +S,\kappa +n)\in \Xi$. $\square$

Proposition 2 (convexity of $\Xi$). The hyperparameter domain $\Xi =\{(\nu ,\kappa ):{\int }_{\mathcal{N}}{e}^{⟨\nu ,\eta ⟩-\kappa A(\eta )}d\eta <\infty \}$ is convex.

Proof. The map $(\nu ,\kappa ,\eta )\mapsto ⟨\nu ,\eta ⟩-\kappa A(\eta )$ is affine in $(\nu ,\kappa )$ for each fixed $\eta$, hence the integrand $e^{⟨\nu ,\eta ⟩-\kappa A(\eta )}$ is a log-convex function of $(\nu ,\kappa )$. For $({\nu }_1,{\kappa }_1),({\nu }_2,{\kappa }_2)\in \Xi$ and $t\in [0,1]$, Hölder's inequality with exponents $1/t$, $1/(1-t)$ gives

$${\int }_{\mathcal{N}}{e}^{⟨t{\nu }_1+(1-t){\nu }_2,\eta ⟩-(t{\kappa }_1+(1-t){\kappa }_2)A(\eta )}d\eta \le (\int e^{⟨{\nu }_1,\eta ⟩-{\kappa }_{1}A}d\eta {)}^t(\int e^{⟨{\nu }_2,\eta ⟩-{\kappa }_{2}A}d\eta {)}^{1-t}<\infty ,$$

so the convex combination lies in $\Xi$. $\square$

Proposition 3 (linear posterior expectation; Diaconis-Ylvisaker). For $(\nu ,\kappa )\in \mathrm{int}\Xi$, ${\mathbb{E}}_{{\pi }_{\nu ,\kappa }}[\nabla A(\eta )]=\nu /\kappa$ and ${\mathbb{E}}_{{\pi }_{\nu ,\kappa }}[\nabla A(\eta )\mid x]=(\nu +T(x))/(\kappa +1)$.

Proof. On $\mathrm{int}\Xi$ the density ${\pi }_{\nu ,\kappa }$ is continuously differentiable on $\mathrm{int}\mathcal{N}$ and its boundary flux through $\partial \mathcal{N}$ vanishes (the kernel is integrable with integrable gradient, so the surface term in the divergence theorem is zero). Therefore

$$0={\int }_{\mathcal{N}}{\nabla }_{\eta }{\pi }_{\nu ,\kappa }(\eta )d\eta ={\int }_{\mathcal{N}}(\nu -\kappa \nabla A(\eta )){\pi }_{\nu ,\kappa }(\eta )d\eta =\nu -\kappa \mathbb{E}[\nabla A(\eta )],$$

giving $\mathbb{E}[\nabla A(\eta )]=\nu /\kappa$. By Proposition 1 the posterior is ${\pi }_{\nu +T(x),\kappa +1}$; applying the same identity to it yields $\mathbb{E}[\nabla A(\eta )\mid x]=(\nu +T(x))/(\kappa +1)$. $\square$

Proposition 4 (closed-form marginal likelihood and posterior predictive). With $C(\nu ,\kappa )$ the conjugate normaliser, the marginal likelihood is $m(x)=({\prod }_{i}h(x_i))C(\nu ,\kappa )/C(\nu +S,\kappa +n)$, and the one-step posterior predictive density is $p(\tilde{x}\mid x)=h(\tilde{x})C(\nu +S,\kappa +n)/C(\nu +S+T(\tilde{x}),\kappa +n+1)$.

Proof. By definition $C(\nu ,\kappa {)}^{-1}={\int }_{\mathcal{N}}{e}^{⟨\nu ,\eta ⟩-\kappa A(\eta )}d\eta$, so the conjugate density integrates to one. The marginal is

$$m(x)={\int }_{\mathcal{N}}(\prod _{i}h(x_i))e^{⟨\eta ,S⟩-nA(\eta )}C(\nu ,\kappa )e^{⟨\nu ,\eta ⟩-\kappa A(\eta )}d\eta =(\prod _{i}h(x_i))C(\nu ,\kappa ){\int }_{\mathcal{N}}{e}^{⟨\nu +S,\eta ⟩-(\kappa +n)A(\eta )}d\eta ,$$

and the remaining integral is $C(\nu +S,\kappa +n{)}^{-1}$, proving the marginal formula. The posterior predictive is $p(\tilde{x}\mid x)=\int f(\tilde{x}\mid \eta ){\pi }_{\nu +S,\kappa +n}(\eta )d\eta =h(\tilde{x})C(\nu +S,\kappa +n)\int e^{⟨\nu +S+T(\tilde{x}),\eta ⟩-(\kappa +n+1)A(\eta )}d\eta$, and the integral is $C(\nu +S+T(\tilde{x}),\kappa +n+1{)}^{-1}$. $\square$

Proposition 5 (reparametrisation invariance of the Jeffreys prior). Under a smooth reparametrisation $\theta =g(\varphi )$ the Jeffreys prior ${\pi }_J(\theta )\propto \sqrt{\det I(\theta )}$ transforms as a density: ${\pi }_J(\varphi )={\pi }_J(g(\varphi ))|\det Dg(\varphi )|$, so Jeffreys' rule gives the same prior measure in every parametrisation.

Proof. The Fisher information transforms by $I(\varphi )=Dg(\varphi {)}^{\top }I(g(\varphi ))Dg(\varphi )$, where $Dg$ is the Jacobian, since the score transforms by $D{g}^{\top }$ under the chain rule and information is its covariance. Taking determinants, $\det I(\varphi )=(\det Dg(\varphi ){)}^2\det I(g(\varphi ))$, so $\sqrt{\det I(\varphi )}=|\det Dg(\varphi )|\sqrt{\det I(g(\varphi ))}$. Hence the Jeffreys density in $\varphi$ equals the Jeffreys density in $\theta$ pulled back with the Jacobian factor $|\det Dg(\varphi )|$, which is exactly the change-of-variables law for a measure. The induced prior measure is therefore invariant: computing Jeffreys in any coordinate system and transforming gives the same probability assignment. $\square$

Connections Master

• Bayes estimation under loss 45.03.01 is the immediate consumer: the conjugate posterior mean $(\nu +T(x))/(\kappa +1)$ is the squared-error Bayes estimator of the mean parameter, and the three closed-form examples computed there as estimators (normal-normal, beta-binomial, gamma-Poisson) are the special cases of the single hyperparameter-update rule proved here, with the prior-sample-size $\kappa$ controlling the shrinkage that unit identifies.

• Sufficiency, factorization, and exponential families 45.01.02 supply the likelihood side: the natural statistic $T$ and log-partition function $A$ that this unit's conjugate prior is built against are exactly the objects defined there, and conjugacy is sufficiency read on the prior — the prior absorbs the same $T$ that quarantines the parameter, which is why a fixed-dimension conjugate prior survives unlimited data.

• Markov chain Monte Carlo 45.03.05 is the resolution where conjugacy ends: Gibbs sampling builds on conditional conjugacy (each full conditional of a hierarchical model is a conjugate update of the form derived here) and Metropolis-Hastings handles the non-conjugate posteriors whose intractable normaliser $C$ this unit identifies as the limitation of closed-form Bayes.

• Ridge regression and regularised estimation 45.06.03 is the conjugate normal-normal model in disguise: the posterior mean under a Gaussian prior on regression coefficients is the ridge estimator, and the prior precision $\kappa$ of this unit is the ridge penalty $\lambda ={\sigma }^2/{\tau }^2$, so the precision-pooling update derived here is the ridge normal equation read as a prior.

• Empirical Bayes and shrinkage estimation 45.04.05 consumes the marginal likelihood $m(x)=({\prod }_{i}h)C(\nu ,\kappa )/C(\nu +S,\kappa +n)$ proved here: empirical Bayes estimates the conjugate hyperparameters $(\nu ,\kappa )$ by maximising this closed-form marginal over the data, turning the prior itself into a data-driven object while retaining the conjugate update for the parameter.

Historical & philosophical context Master

The conjugate update for the binomial is implicit in Thomas Bayes' posthumous 1763 essay and in Pierre-Simon Laplace's development of inverse probability, where the uniform prior on a success rate yields the beta posterior and Laplace's rule of succession is its predictive. The general programme of choosing a prior so that the posterior stays analytically tractable was made systematic by Howard Raiffa and Robert Schlaifer in Applied Statistical Decision Theory ^{[Raiffa Schlaifer 1961]} (1961), who coined the use of "conjugate" priors and tabulated the standard families for decision-theoretic use. The exponential-family unification — that every standard conjugate pair is an instance of the prior $\exp (⟨\nu ,\eta ⟩-\kappa A(\eta ))$ and that conjugacy is characterised by a linear posterior expectation of the mean parameter — is due to Persi Diaconis and Donald Ylvisaker in Conjugate priors for exponential families (Ann. Statist. 7, 1979) ^{[Diaconis Ylvisaker 1979]}, who also proved that mixtures of conjugate priors are dense in the space of priors and remain closed under updating.

The noninformative-prior tradition runs from Bayes' and Laplace's flat priors through Harold Jeffreys' invariance rule $\pi \propto \sqrt{\det I}$ in Theory of Probability ^{[Jeffreys 1961]} (1939; 3rd ed. 1961), and on to José Bernardo's reference priors, which maximise the expected information gain from prior to posterior and repair the multiparameter deficiencies of Jeffreys' rule. The modern computational treatment, in which conjugacy is the analytic core that survives as the conditional update inside Gibbs samplers and the building block of hierarchical models, is laid out in Andrew Gelman and colleagues' Bayesian Data Analysis ^{[Gelman 2013]} (1995; 3rd ed. 2013) and in Christian Robert's The Bayesian Choice ^{[Robert 2007]} (1994; 2nd ed. 2007), the latter organising the exponential-family conjugate theory around the same natural-parameter structure used here.

Bibliography Master

@book{raiffaschlaifer1961applied,
  author    = {Raiffa, Howard and Schlaifer, Robert},
  title     = {Applied Statistical Decision Theory},
  publisher = {Harvard Business School},
  year      = {1961}
}

@article{diaconisylvisaker1979conjugate,
  author  = {Diaconis, Persi and Ylvisaker, Donald},
  title   = {Conjugate priors for exponential families},
  journal = {Annals of Statistics},
  volume  = {7},
  number  = {2},
  pages   = {269--281},
  year    = {1979}
}

@book{jeffreys1961theory,
  author    = {Jeffreys, Harold},
  title     = {Theory of Probability},
  edition   = {3rd},
  publisher = {Oxford University Press},
  year      = {1961}
}

@book{bernardosmith1994bayesian,
  author    = {Bernardo, Jos\'e M. and Smith, Adrian F. M.},
  title     = {Bayesian Theory},
  publisher = {Wiley},
  year      = {1994}
}

@book{gelman2013bayesian,
  author    = {Gelman, Andrew and Carlin, John B. and Stern, Hal S. and Dunson, David B. and Vehtari, Aki and Rubin, Donald B.},
  title     = {Bayesian Data Analysis},
  edition   = {3rd},
  publisher = {CRC Press},
  year      = {2013}
}

@book{robert2007bayesian,
  author    = {Robert, Christian P.},
  title     = {The Bayesian Choice},
  edition   = {2nd},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {2007}
}

@book{casellaberger2002statistical,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2nd},
  publisher = {Duxbury},
  year      = {2002}
}