Mutual Information, KL Divergence, and Variational Characterizations (Donsker-Varadhan)

Intuition Beginner

You have two random variables, say $X$ and $Y$. Entropy tells you how uncertain each one is on its own. But what if you want to know how much learning one tells you about the other? That is mutual information.

Think of a noisy communication channel. I send you a bit $X$ (0 or 1), and noise corrupts it. You receive $Y$. If the channel is perfect, then $Y$ tells you $X$ exactly, and the mutual information equals the entropy of $X$ — all the uncertainty in $X$ is resolved by seeing $Y$. If the channel is useless (random noise unrelated to $X$), then $Y$ tells you nothing, and the mutual information is zero.

Mutual information sits between zero and the entropy of each variable. It is zero when the variables are independent and equals the entropy when one completely determines the other.

Now flip the perspective. Suppose nature draws from a true distribution $p$, but you model the world with a different distribution $q$. How bad is your model? The KL divergence $D(p\Vert q)$ measures the penalty. It is always non-negative and equals zero only when $p$ and $q$ are the same distribution. It is not a distance (it is asymmetric: $D(p\Vert q)$ and $D(q\Vert p)$ differ), but it behaves like one in many proofs.

The deep result is that KL divergence has a variational characterisation. Instead of computing it from the log-ratio formula, you can express it as the largest gap between what you expect under $p$ and what you would predict under $q$, across all bounded test functions. This is the Donsker-Varadhan formula, and it connects information theory to convex duality and large deviations.

Visual Beginner

Figure: a Venn diagram with two overlapping circles. The left circle represents $H(X)$, the right circle represents $H(Y)$, and the overlap is the mutual information $I(X;Y)$. The non-overlapping part of the left circle is $H(X|Y)$, and the non-overlapping part of the right circle is $H(Y|X)$. The union is $H(X,Y)$. Below the diagram, a table of concrete values.

Scenario	H(X)	H(Y)	I(X;Y)	H(X|Y)	H(Y|X)
X and Y independent fair coins	1.00	1.00	0.00	1.00	1.00
X = Y (same fair coin)	1.00	1.00	1.00	0.00	0.00
X fair coin, Y = X with 10% flip prob	1.00	1.00	0.53	0.47	0.47
X uniform die, Y = X mod 2	2.58	1.00	1.00	1.58	0.00

The pattern: mutual information is the overlap in the Venn diagram. When there is no overlap (independence), $I=0$. When one circle sits inside the other (one variable determines the other), $I$ equals the smaller entropy.

Worked example Beginner

A binary symmetric channel flips each input bit with probability 0.1. The input $X$ is a fair coin (0 or 1 with equal probability). The output is $Y$.

Step 1. Entropies. $H(X)=1$ bit. By symmetry, $Y$ is also uniform, so $H(Y)=1$ bit.

Step 2. Joint distribution. The channel gives $p(Y=X)=0.9$ and $p(Y\ne X)=0.1$. So each of the four pairs $(0,0),(0,1),(1,0),(1,1)$ has probability: $p(0,0)=0.45$, $p(0,1)=0.05$, $p(1,0)=0.05$, $p(1,1)=0.45$.

Step 3. Conditional entropy. Given $Y$, the conditional distribution of $X$ is: $p(X=Y|Y)=0.9$ and $p(X\ne Y|Y)=0.1$. So $H(X|Y)=H(0.9)=-0.9{\log }_{2}0.9-0.1{\log }_{2}0.1\approx 0.469$ bits.

Step 4. Mutual information. $I(X;Y)=H(X)-H(X|Y)=1-0.469=0.531$ bits. About 53% of the information in $X$ survives the noise.

Step 5. KL divergence interpretation. The joint distribution $p_{XY}$ has probabilities $(0.45,0.05,0.05,0.45)$. If $X$ and $Y$ were independent, the product $p_X\cdot p_Y$ would be $(0.25,0.25,0.25,0.25)$. The KL divergence is:

$D(p_{XY}\Vert p_X{p}_Y)=0.45{\log }_2(0.45/0.25)+0.05{\log }_2(0.05/0.25)+0.05{\log }_2(0.05/0.25)+0.45{\log }_2(0.45/0.25)\approx 0.531$ bits.

This equals the mutual information, which is always the case: $I(X;Y)=D(p_{XY}\Vert p_X{p}_Y)$.

Check your understanding Beginner

Formal definition Intermediate+

Let $X$ and $Y$ be discrete random variables with joint pmf $p(x,y)$ and marginals $p(x)={\sum }_{y}p(x,y)$, $p(y)={\sum }_{x}p(x,y)$.

Definition (Mutual information). The mutual information between $X$ and $Y$ is

$$I(X;Y)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y){\log }_2\frac{p(x,y)}{p(x)p(y)}.$$

Equivalently, $I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y)$.

Definition (KL divergence, relative entropy). For two probability mass functions $p$ and $q$ on the same alphabet $\mathcal{X}$ (with the convention that $0\log (0/0)=0$ and $p\log (p/0)=+\infty$ when $p>0$), the Kullback-Leibler divergence is

$$D(p\Vert q)=\sum _{x\in \mathcal{X}}p(x){\log }_2\frac{p(x)}{q(x)}.$$

The mutual information is a special case: $I(X;Y)=D(p_{XY}\Vert p_X\otimes p_Y)$.

Definition (Conditional mutual information). For random variables $X$, $Y$, and $Z$:

$$I(X;Y|Z)=H(X|Z)-H(X|Y,Z)=\sum _{z}p(z)\sum _{x,y}p(x,y|z){\log }_2\frac{p(x,y|z)}{p(x|z)p(y|z)}.$$

Counterexamples to common slips

• KL divergence is not symmetric. In general $D(p\Vert q)\ne D(q\Vert p)$. Computing both for the distributions $p=(0.5,0.5)$ and $q=(0.9,0.1)$ gives $D(p\Vert q)\approx 0.74$ bits but $D(q\Vert p)\approx 0.48$ bits.

• KL divergence is not a metric. It satisfies non-negativity and $D(p\Vert q)=0⟺p=q$, but not symmetry or the triangle inequality. The symmetrised version $(D(p\Vert q)+D(q\Vert p))/2$ is called the Jeffreys divergence.

• Mutual information can exceed neither $H(X)$ nor $H(Y)$. Since $I(X;Y)=H(X)-H(X|Y)$ and $H(X|Y)\ge 0$, we have $I(X;Y)\le H(X)$. Similarly $I(X;Y)\le H(Y)$.

• $D(p\Vert q)=\infty$ is possible. If $p$ puts positive probability on an event that $q$ assigns zero probability, then $D(p\Vert q)=+\infty$. This means $D(p\Vert q)$ is finite only when $p$ is absolutely continuous with respect to $q$ (the support of $p$ is contained in the support of $q$).

Key theorem with proof Intermediate+

Theorem (Non-negativity of KL divergence — Gibbs' inequality). For any two probability distributions $p$ and $q$ on the same alphabet:

$$D(p\Vert q)\ge 0,$$

with equality if and only if $p(x)=q(x)$ for all $x$.

Proof. Apply the inequality $\ln t\le t-1$ (with equality iff $t=1$) to $t=q(x)/p(x)$:

$$\begin{array}{rl}-D(p\Vert q)& =-\sum _{x}p(x)\ln \frac{p(x)}{q(x)}=\sum _{x}p(x)\ln \frac{q(x)}{p(x)}\\ & \le \sum _{x}p(x)\left(\frac{q(x)}{p(x)}-1\right)=\sum _{x}q(x)-\sum _{x}p(x)=1-1=0.\end{array}$$

Therefore $D(p\Vert q)\ge 0$. Equality requires $\ln (q(x)/p(x))=q(x)/p(x)-1$ for all $x$ with $p(x)>0$, which holds iff $q(x)/p(x)=1$, i.e., $p(x)=q(x)$. $\square$

Corollary (Non-negativity of mutual information). $I(X;Y)\ge 0$, with equality iff $X$ and $Y$ are independent.

Proof. $I(X;Y)=D(p_{XY}\Vert p_X{p}_Y)\ge 0$ by Gibbs' inequality. Equality holds iff $p_{XY}=p_X{p}_Y$, i.e., independence. $\square$

Theorem (Log-sum inequality). For non-negative numbers $a_1,\dots ,a_n$ and $b_1,\dots ,b_n$:

$$\sum _{i=1}^n{a}_i\log \frac{a_i}{b_i}\ge \left(\sum _{i=1}^n{a}_i\right)\log \frac{{\sum }_{i=1}^n{a}_i}{{\sum }_{i=1}^n{b}_i},$$

with the convention $0\log 0=0$, $a\log (a/0)=\infty$ for $a>0$, and equality iff $a_i/b_i$ is constant for all $i$ with $a_i+b_i>0$.

Proof. Let $A=\sum a_i$ and $B=\sum b_i$. Define probability distributions $p_i=a_i/A$ and $q_i=b_i/B$. Then Gibbs' inequality gives $D(p\Vert q)\ge 0$, which rearranges to the log-sum inequality. $\square$

Bridge. The non-negativity of KL divergence via Gibbs' inequality builds toward the asymptotic equipartition property in 46.02.01 by supplying the key inequality $D(p\Vert q)\ge 0$ that underpins the typical-set bound and the source coding theorem. It appears again in the channel coding theorem 46.02.03 as the foundation for Fano's inequality and the converse direction, and in Sanov's theorem 37.07.05 where the KL divergence $D(q\Vert p)$ becomes the large-deviation rate function. The foundational reason mutual information reduces to $D(p_{XY}\Vert p_X{p}_Y)$ is that independence is the null hypothesis of no relationship; the log-sum inequality is the computational engine behind every information-theoretic inequality; and the central insight connecting these is that all information measures are special cases of a single asymmetric divergence functional, which generalises to the Donsker-Varadhan variational formula below. The bridge is that convexity and variational duality unify these quantities under one roof.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not define mutual information or KL divergence on discrete distributions. The PMF type exists, but there is no functional $I(X;Y)$ on joint PMFs, no $D(p\Vert q)$ on pairs of distributions, and no proof of non-negativity via the log-sum inequality. The chain rule for mutual information, the data processing inequality, and the convexity of KL divergence are all absent. The Donsker-Varadhan variational formula would require connecting convex duality (Fenchel-Legendre transforms, which exist in Mathlib for convex functions on normed spaces) to information-theoretic quantities, a connection that has not been made. A Codex.InformationTheory.MutualInfo module defining $I(X;Y)$ and $D(p\Vert q)$ on PMF with Gibbs' inequality and the basic calculus would be the load-bearing first step; this unit ships without it.

Advanced results Master

The Donsker-Varadhan variational formula

The KL divergence has a dual representation as the solution to a convex optimisation problem. This is the Donsker-Varadhan formula.

Theorem (Donsker-Varadhan, 1975-1983). Let $P$ and $Q$ be probability measures on a measurable space $(\Omega ,\mathcal{F})$ with $P\ll Q$ ($P$ is absolutely continuous with respect to $Q$). Then

$$D(P\Vert Q)=\underset{f\in {\mathcal{B}}_b(\Omega )}{\sup }\left\{{\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]\right\},$$

where the supremum is taken over all bounded measurable functions $f:\Omega \to \mathbb{R}$.

Proof sketch. The direction $D(P\Vert Q)\ge {\sup }_f\{{\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]\}$ follows from Jensen's inequality: ${\mathbb{E}}_P[f]={\mathbb{E}}_Q[f\cdot (dP/dQ)]$, and $\log {\mathbb{E}}_Q[e^f]\ge {\mathbb{E}}_Q[f]$. More precisely, for any bounded measurable $f$:

$${\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]={\mathbb{E}}_Q\left[f\cdot \frac{dP}{dQ}\right]-\log {\mathbb{E}}_Q[e^f].$$

By the variational characterisation of the cumulant generating function, the right side is bounded by $D(P\Vert Q)$ via the Gibbs variational principle. For the reverse inequality, choose $f=\log (dP/dQ)-c$ for a constant $c$ that makes $f$ bounded (via truncation and taking limits). Then ${\mathbb{E}}_P[f]=D(P\Vert Q)-c$ and ${\mathbb{E}}_Q[e^f]=e^{-c}$, so ${\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]=D(P\Vert Q)$. Approximation by bounded functions gives the general case. $\square$

The key insight: the Donsker-Varadhan formula identifies KL divergence as the Fenchel-Legendre conjugate of the cumulant generating function ${\Lambda }_Q(f)=\log {\mathbb{E}}_Q[e^f]$. Specifically, $D(P\Vert Q)={\Lambda }_Q^{\ast }(dP/dQ)$ where ${\Lambda }_Q^{\ast }$ is the convex conjugate. This places KL divergence squarely in the framework of convex duality.

The data processing inequality and sufficient statistics

The data processing inequality is the fundamental no-free-lunch theorem of information theory.

Theorem (Data processing inequality). If $X\to Y\to Z$ is a Markov chain (meaning the conditional distribution of $Z$ given $(X,Y)$ depends only on $Y$), then

$$I(X;Z)\le I(X;Y).$$

Proof. By the chain rule for mutual information:

$$I(X;Y,Z)=I(X;Y)+I(X;Z|Y).$$

Since $X\to Y\to Z$ is Markov, $X$ and $Z$ are conditionally independent given $Y$, so $I(X;Z|Y)=0$. Hence $I(X;Y,Z)=I(X;Y)$.

Also by the chain rule: $I(X;Y,Z)=I(X;Z)+I(X;Y|Z)$. Since $I(X;Y|Z)\ge 0$, we get $I(X;Z)\le I(X;Y,Z)=I(X;Y)$. $\square$

Corollary (Processing cannot increase information). If $Z=f(Y)$ for some function $f$, then $X\to Y\to Z$ is Markov (since $Z$ depends only on $Y$), so $I(X;f(Y))\le I(X;Y)$.

Definition (Sufficient statistic). A function $T:\mathcal{X}\to \mathcal{T}$ is a sufficient statistic for the parameter $\theta$ in a family of distributions $\{p_{\theta }\}$ if

$$I(\theta ;X)=I(\theta ;T(X)).$$

Equivalently, $T$ is sufficient iff $p_{\theta }(x|T(x))=p(x|T(x))$ does not depend on $\theta$. The data processing inequality always gives $I(\theta ;T(X))\le I(\theta ;X)$; sufficiency is the equality case where no information about $\theta$ is lost by compressing $X$ through $T$.

Pinsker's inequality

KL divergence bounds total variation distance.

Theorem (Pinsker's inequality). For any two distributions $P$ and $Q$ on the same measurable space:

$$\delta (P,Q)=\underset{A}{\sup }|P(A)-Q(A)|\le \sqrt{\frac{1}{2}D(P\Vert Q)}.$$

The proof uses the inequality $2(p-q{)}^2\le p\log (p/q)$ for $p,q\in (0,1)$, extended to distributions by a coupling argument. Pinsker's inequality is the bridge between information-theoretic divergences and the classical $L^1$-type distances used in statistics and probability.

Worked example. Let $p=(0.5,0.5)$ and $q=(0.6,0.4)$. Total variation distance $\delta (p,q)=0.1$. KL divergence $D(p\Vert q)=0.5\log (0.5/0.6)+0.5\log (0.5/0.4)\approx 0.5(-0.182)+0.5(0.322)=0.070$ nats. Pinsker gives $\delta \le \sqrt{0.035}\approx 0.187$, which is satisfied since $0.1\le 0.187$.

Convexity properties

KL divergence has powerful convexity properties that underpin most of its applications.

Theorem (Joint convexity of KL divergence). $D(p\Vert q)$ is jointly convex in $(p,q)$: for $\lambda \in [0,1]$ and distributions $p_1,p_2,q_1,q_2$:

$$D(\lambda p_1+(1-\lambda )p_2\Vert \lambda q_1+(1-\lambda )q_2)\le \lambda D(p_1\Vert q_1)+(1-\lambda )D(p_2\Vert q_2).$$

This follows from the log-sum inequality and is the algebraic engine behind the convexity of mutual information.

Theorem (Convexity of mutual information in $p(x)$). For a fixed channel $p(y|x)$, the mutual information $I(X;Y)={\sum }_{x,y}p(x)p(y|x)\log (p(y|x)/p(y))$ is a concave function of the input distribution $p(x)$.

This concavity is the basis for the Arimoto-Blahut algorithm for computing channel capacity, and it guarantees that the maximum of $I(X;Y)$ over input distributions exists and is achieved at an extreme point of the simplex.

Theorem (Convexity of mutual information in $p(y|x)$). For a fixed input distribution $p(x)$, the mutual information $I(X;Y)$ is a convex function of the channel $p(y|x)$.

This convexity is used in rate-distortion theory, where the distortion-rate function is obtained by minimising mutual information over channels subject to a distortion constraint.

Fano's inequality and the converse direction

Mutual information controls the probability of error in estimation.

Theorem (Fano's inequality). Let $X$ be a random variable and $\hat{X}$ an estimator based on observing $Y$. If $P_e=\mathrm{Pr}[\hat{X}\ne X]$, then

$$H(X|Y)\le H_b(P_e)+P_e{\log }_2(|\mathcal{X}|-1),$$

where $H_b$ is the binary entropy. Equivalently, since $I(X;Y)=H(X)-H(X|Y)$:

$$I(X;Y)\ge H(X)-H_b(P_e)-P_e{\log }_2(|\mathcal{X}|-1).$$

This provides a lower bound on mutual information in terms of the error probability. If the mutual information is small, then the error probability must be large — you cannot reliably decode $X$ from $Y$ if there is not enough mutual information between them.

Synthesis. Mutual information $I(X;Y)=D(p_{XY}\Vert p_X{p}_Y)$ is the unique measure of statistical dependence that respects the chain rule $I(X_1,X_2;Y)=I(X_1;Y)+I(X_2;Y|X_1)$ and collapses to zero exactly at independence. The data processing inequality builds toward the channel coding theorem by guaranteeing that no post-processing of $Y$ can extract more information about $X$ than $Y$ already contains, and it appears again in the characterisation of sufficient statistics where equality $I(\theta ;X)=I(\theta ;T(X))$ is equivalent to conditional independence. The Donsker-Varadhan formula generalises the KL divergence from a sum of log-ratios to a variational supremum over all bounded test functions, the central insight that connects information theory to convex duality and large deviations. Putting these together: entropy measures uncertainty, mutual information measures shared uncertainty, KL divergence measures mismatch, and the variational characterisation reveals that all three are facets of a single convex-analytic object.

Full proof set Master

Proposition (Mutual information chain rule). For random variables $X_1,\dots ,X_n$ and $Y$:

$$I(X_1,\dots ,X_n;Y)=\sum _{i=1}^{n}I(X_i;Y|X_1,\dots ,X_{i-1}).$$

Proof. By induction on $n$. The base case $n=1$ is $I(X_1;Y)=I(X_1;Y)$. For the inductive step, apply the two-variable chain rule $I(A,B;Y)=I(A;Y)+I(B;Y|A)$ with $A=X_1,\dots ,X_{n-1}$ and $B=X_n$:

$$I(X_1,\dots ,X_n;Y)=I(X_1,\dots ,X_{n-1};Y)+I(X_n;Y|X_1,\dots ,X_{n-1}).$$

The first term decomposes by the induction hypothesis. $\square$

Proposition (Donsker-Varadhan lower bound). For any bounded measurable function $f$ and distributions $P,Q$ with $P\ll Q$:

$${\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]\le D(P\Vert Q).$$

Proof. Let $g=dP/dQ$ be the Radon-Nikodym derivative. By Jensen's inequality applied to the convex function $\varphi (t)=t\log t$:

$${\mathbb{E}}_P[f]={\mathbb{E}}_Q[fg]={\mathbb{E}}_Q[fg].$$

Consider the variational problem. For any bounded $f$:

$$\begin{array}{rl}{\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]& ={\mathbb{E}}_Q[fg]-\log {\mathbb{E}}_Q[e^f]\\ & ={\mathbb{E}}_Q[fg]-\log {\mathbb{E}}_Q[e^f].\end{array}$$

Let $h=e^f/{\mathbb{E}}_Q[e^f]$, so ${\mathbb{E}}_Q[h]=1$ and $f=\log h+\log {\mathbb{E}}_Q[e^f]$. Then:

$${\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]={\mathbb{E}}_Q[g\log h]\le {\mathbb{E}}_Q[g\log g]=D(P\Vert Q),$$

where the inequality uses ${\mathbb{E}}_Q[g\log h]\le {\mathbb{E}}_Q[g\log g]$ (equivalently, $D(P\Vert Q)-{\mathbb{E}}_P[f]+\log {\mathbb{E}}_Q[e^f]={\mathbb{E}}_Q[g(\log g-\log h)]\ge 0$ by the non-negativity of KL divergence applied to the distributions with densities $g$ and $h$ with respect to $Q$). $\square$

Proposition (KL divergence is convex in the second argument). For fixed $p$ and distributions $q_1,q_2$ with $\lambda \in [0,1]$:

$$D(p\Vert \lambda q_1+(1-\lambda )q_2)\le \lambda D(p\Vert q_1)+(1-\lambda )D(p\Vert q_2).$$

Proof. This is a special case of joint convexity. Alternatively, the function $q\mapsto D(p\Vert q)$ is convex because $\log$ is concave: $-\log (\lambda q_1(x)+(1-\lambda )q_2(x))\le -\lambda \log q_1(x)-(1-\lambda )\log q_2(x)$ by concavity, so $p(x)\log (p(x)/(\lambda q_1+(1-\lambda )q_2)(x))\le \lambda p(x)\log (p(x)/q_1(x))+(1-\lambda )p(x)\log (p(x)/q_2(x))$. Summing over $x$ gives the result. $\square$

Connections Master

• 46.01.01 — Entropy, conditional entropy, and the chain rule are the direct prerequisites; mutual information is defined as $I(X;Y)=H(X)+H(Y)-H(X,Y)$ using the entropy function from that unit.
• 37.01.01 — Probability spaces and Kolmogorov's extension theorem provide the measure-theoretic foundation on which joint distributions and conditional distributions are defined.
• 46.02.01 — The AEP uses the chain rule for mutual information to decompose $I(X^n;Y^n)$ and identify the mutual information rate for stationary sources.
• 37.07.05 — Sanov's theorem in large deviations identifies the KL divergence $D(q\Vert p)$ as the rate function for the probability of seeing an empirical distribution $q$ when the true distribution is $p$; the Donsker-Varadhan formula is the dual representation of this rate.
• 46.02.03 — The channel coding theorem identifies channel capacity as the maximum mutual information ${\max }_{p(x)}I(X;Y)$ over input distributions; the concavity of mutual information in $p(x)$ guarantees the maximum is attained.
• 45.07.01 — PAC learning bounds use the KL divergence and Pinsker's inequality to control the gap between empirical and population risk; mutual information bounds appear in information-theoretic generalisation bounds.

Historical & philosophical context Master

The Kullback-Leibler divergence was introduced by Solomon Kullback and Richard Leibler in 1951 (On information and sufficiency, Annals of Mathematical Statistics 22, 79-86). They extended Shannon's 1948 information-theoretic framework to a general measure of discrepancy between two statistical distributions, motivated by the Neyman-Pearson theory of hypothesis testing. The Neyman-Pearson lemma (1933) had already established that the likelihood ratio is the optimal test statistic; Kullback and Leibler showed that the expected log-likelihood ratio — which is exactly $D(p\Vert q)$ — measures the information for discriminating between hypotheses.

The variational characterisation of KL divergence was developed by Monroe Donsker and Srinivasa Varadhan in a landmark four-part series published between 1933 and 1983 in Communications on Pure and Applied Mathematics (parts I-IV: volumes 28, 29, 36). Their original motivation was the large-deviation theory of Markov processes. The formula $D(P\Vert Q)={\sup }_f\{{\mathbb{E}}_P[f]-\log {\mathbb{E}}_Q[e^f]\}$ appeared as a tool for computing the Legendre transform of the principal eigenvalue of a Markov generator, which governs the large-deviation rate function for occupation measures. The connection to convex duality (specifically, the fact that KL divergence is the Fenchel conjugate of the cumulant generating function) was made explicit by Dupuis and Ellis in their 1997 monograph A Weak Convergence Approach to the Theory of Large Deviations.

Imre Csiszár introduced the $f$-divergence framework in 1963 (Informationstheoretische Ungleichungen, in German) and independently by Ali and Silvey in 1966, showing that KL divergence is a special case of a broader family of divergences $D_f(P\Vert Q)={\mathbb{E}}_Q[f(dP/dQ)]$ for convex $f$. The $f$-divergence framework unified KL divergence, total variation, and the chi-squared divergence under one roof, and it has become central to modern statistics and machine learning.

The data processing inequality was implicit in Shannon's 1948 paper (the degraded channel argument) and was formalised by Kullback and Leibler. The sufficient-statistics characterisation via mutual information was made precise by Kullback in his 1959 textbook Information Theory and Statistics (Wiley), connecting the information-theoretic and the classical Fisher-Neyman factorisation criteria for sufficiency.

Bibliography Master

@article{kullback-leibler1951,
  author  = {Kullback, S. and Leibler, R. A.},
  title   = {On Information and Sufficiency},
  journal = {Annals of Mathematical Statistics},
  volume  = {22},
  pages   = {79--86},
  year    = {1951},
}
@book{cover-thomas2006,
  author    = {Cover, T. M. and Thomas, J. A.},
  title     = {Elements of Information Theory},
  edition   = {2nd},
  publisher = {Wiley},
  year      = {2006},
}
@book{csiszar-korner2011,
  author    = {Csisz{\'a}r, I. and K{\"o}rner, J.},
  title     = {Information Theory: Coding Theorems for Discrete Memoryless Systems},
  edition   = {2nd},
  publisher = {Cambridge University Press},
  year      = {2011},
}
@article{donsker-varadhan1975,
  author  = {Donsker, M. D. and Varadhan, S. R. S.},
  title   = {Asymptotic Evaluation of Certain Markov Process Expectations for Large Time, {I}},
  journal = {Communications on Pure and Applied Mathematics},
  volume  = {28},
  pages   = {1--47},
  year    = {1975},
}
@article{donsker-varadhan1976,
  author  = {Donsker, M. D. and Varadhan, S. R. S.},
  title   = {Asymptotic Evaluation of Certain Markov Process Expectations for Large Time, {II}},
  journal = {Communications on Pure and Applied Mathematics},
  volume  = {29},
  pages   = {389--461},
  year    = {1976},
}
@article{donsker-varadhan1983,
  author  = {Donsker, M. D. and Varadhan, S. R. S.},
  title   = {Asymptotic Evaluation of Certain Markov Process Expectations for Large Time, {III}},
  journal = {Communications on Pure and Applied Mathematics},
  volume  = {36},
  pages   = {183--212},
  year    = {1983},
}
@book{dupuis-ellis1997,
  author    = {Dupuis, P. and Ellis, R. S.},
  title     = {A Weak Convergence Approach to the Theory of Large Deviations},
  publisher = {Wiley},
  year      = {1997},
}
@book{kullback1959,
  author    = {Kullback, S.},
  title     = {Information Theory and Statistics},
  publisher = {Wiley},
  year      = {1959},
}
@article{pinsker1964,
  author  = {Pinsker, M. S.},
  title   = {Information and Information Stability of Random Variables and Processes},
  journal = {Problems of Information Transmission},
  year    = {1964},
  note    = {Originally published in Russian by Nauka, Moscow},
}
@article{csiszar1967,
  author  = {Csisz{\'a}r, I.},
  title   = {Information-Type Measures of Difference of Probability Distributions and Indirect Observations},
  journal = {Studia Scientiarum Mathematicarum Hungarica},
  volume  = {2},
  pages   = {299--318},
  year    = {1967},
}