Information Inequalities: Log-Sum Inequality, Subadditivity, and Pinsker's Inequality

Intuition Beginner

You have two probability distributions, $p$ and $q$, and you want to know how different they are. KL divergence $D(p\Vert q)$ measures the mismatch, but it is asymmetric and hard to interpret directly. Total variation distance is simpler: it asks "what is the biggest difference in probability that $p$ and $q$ assign to any event?" If $p$ and $q$ agree on every event, the distance is zero. If they disagree a lot, it can reach 1.

Pinsker's inequality connects these two notions. It says: if the KL divergence is small, then the distributions must be close in total variation. Specifically, the total variation distance is at most the square root of half the KL divergence. This is useful because KL divergence is often easier to compute, while total variation is what you need for probability bounds.

Subadditivity is a simpler idea. If you have several random variables, the total uncertainty in all of them together cannot exceed the sum of their individual uncertainties. Knowing some variables might reduce the surprise of others (because they carry shared information), but it can never increase the total. Equality holds exactly when the variables are independent.

Fano's inequality answers a practical question: if you try to guess a random variable $X$ based on partial information $Y$, how likely are you to be wrong? It says that if the conditional uncertainty $H(X|Y)$ is large, then the error probability must be large. You cannot reliably guess something that remains very uncertain even after seeing the data.

Visual Beginner

Figure: two bar charts side by side. The left chart shows distributions $p=(0.5,0.5)$ and $q=(0.7,0.3)$ as overlapping bars. The total variation distance $\delta (p,q)=0.2$ is shaded. The right chart shows $p=(0.5,0.5)$ and $r=(0.95,0.05)$ with $\delta (p,r)=0.45$ shaded. Below each pair: the KL divergence value and the Pinsker bound.

Distributions	TV distance	KL divergence (nats)	Pinsker bound	Bound tight?
$p=(0.5,0.5)$, $q=(0.6,0.4)$	0.10	0.020	0.10	Close
$p=(0.5,0.5)$, $q=(0.7,0.3)$	0.20	0.082	0.20	Close
$p=(0.5,0.5)$, $q=(0.9,0.1)$	0.40	0.511	0.51	Loose
$p=(0.5,0.5)$, $q=(1.0,0.0)$	0.50	infinity	infinity	Exact

The pattern: Pinsker's bound is tight for small divergences but loosens as the distributions move further apart. When the KL divergence is infinite (one distribution has zero probability where the other does not), the bound is vacuous.

Worked example Beginner

A factory produces widgets in three grades: A, B, and C. The true production distribution is $p=(0.60,0.30,0.10)$. A quality inspector uses a simplified model $q=(0.50,0.35,0.15)$. How different are these distributions?

Step 1. Compute total variation distance: $\delta (p,q)=\frac{1}{2}(|0.60-0.50|+|0.30-0.35|+|0.10-0.15|)=\frac{1}{2}(0.10+0.05+0.05)=0.10$.

Step 2. Compute KL divergence in nats: $D(p\Vert q)=0.60\ln (0.60/0.50)+0.30\ln (0.30/0.35)+0.10\ln (0.10/0.15)=0.60\times 0.182+0.30\times (-0.155)+0.10\times (-0.405)=0.109-0.047-0.041=0.022$ nats.

Step 3. Apply Pinsker's inequality: $\delta (p,q)\le \sqrt{D(p\Vert q)/2}=\sqrt{0.022/2}=\sqrt{0.011}\approx 0.105$.

The actual total variation (0.10) is close to the bound (0.105), confirming that these distributions are genuinely close and Pinsker's inequality is nearly tight here.

Step 4. What does this mean? The inspector's model differs from reality by at most 10 percentage points on any event. For practical decisions like "is the fraction of grade-A widgets above 55%?", the model and reality give answers within 10% of each other. The KL divergence of 0.022 nats quantifies the information loss from using the wrong model, and Pinsker translates that into a concrete probability guarantee.

Check your understanding Beginner

Formal definition Intermediate+

Let $p$ and $q$ be probability distributions on the same finite alphabet $\mathcal{X}$.

Definition (Total variation distance). The total variation distance between $p$ and $q$ is

$$\delta (p,q)=\frac{1}{2}\sum _{x\in \mathcal{X}}|p(x)-q(x)|=\underset{A\subseteq \mathcal{X}}{\sup }|p(A)-q(A)|.$$

Definition (Log-sum inequality). For non-negative reals $a_1,\dots ,a_n$ and $b_1,\dots ,b_n$ (not all zero):

$$\sum _{i=1}^n{a}_i\log \frac{a_i}{b_i}\ge \left(\sum _{i=1}^n{a}_i\right)\log \frac{{\sum }_{i=1}^n{a}_i}{{\sum }_{i=1}^n{b}_i},$$

with the conventions $0\log 0=0$, $0\log (0/0)=0$, and $a\log (a/0)=+\infty$ for $a>0$. Equality holds if and only if $a_i/b_i$ is constant for all $i$ with $a_i+b_i>0$.

Definition (Binary KL divergence). For $p,q\in [0,1]$, the binary KL divergence is

$$d(p\Vert q)=p\log \frac{p}{q}+(1-p)\log \frac{1-p}{1-q}.$$

This is the KL divergence between the Bernoulli distributions $\text{Ber}(p)$ and $\text{Ber}(q)$.

Counterexamples to common slips

• Pinsker's inequality is one-directional. It gives $\delta \le \sqrt{D/2}$, not equality. The reverse inequality $D\le c\cdot \delta$ holds for some constant $c$ depending on the support, but there is no universal reverse Pinsker without additional assumptions on the distributions.

• Subadditivity is not superadditivity. While $H(X,Y)\le H(X)+H(Y)$ always holds, the reverse $H(X,Y)\ge H(X)$ also holds but for a different reason (conditioning reduces entropy). The two inequalities combine to give $H(X)\le H(X,Y)\le H(X)+H(Y)$.

• Fano's inequality does not give an upper bound on error. It gives a lower bound on error probability in terms of conditional entropy. High conditional entropy forces high error, but low conditional entropy does not guarantee low error.

Key theorem with proof Intermediate+

Theorem (Pinsker's inequality). For any two probability distributions $P$ and $Q$ on the same measurable space:

$$\delta (P,Q)\le \sqrt{\frac{1}{2}D(P\Vert Q)}.$$

Equivalently, $2\delta (P,Q{)}^2\le D(P\Vert Q)$.

Proof. We prove the binary case first, then extend to general distributions by a coupling argument.

Step 1 (Binary case). Let $p,q\in [0,1]$ and set $\Delta =|p-q|$. We show $d(p\Vert q)\ge 2{\Delta }^2$. Define $g(\Delta )=d(p\Vert q)$ where $q=p+\Delta$. Then $g(0)=0$, $g^{\prime }(0)=0$, and $g^{\prime \prime }(0)=2$ (direct computation of the second derivative of the binary KL at $q=p$). By the Taylor expansion and the convexity of $q\mapsto d(p\Vert q)$:

$$d(p\Vert q)=\frac{1}{2}{g}^{\prime \prime }(\xi ){\Delta }^2\ge 2{\Delta }^2,$$

where $\xi \in (0,\Delta )$ and the inequality uses $g^{\prime \prime }(\xi )\ge 4$ for the binary KL divergence (which can be verified by direct calculation of $g^{\prime \prime }(\Delta )$ and showing it is bounded below by 4 for $\Delta \in [0,\min (p,1-p)]$).

Step 2 (General finite alphabet). For distributions on a finite alphabet, partition the outcomes into sets $A=\{x:p(x)\ge q(x)\}$ and $A^c=\{x:p(x)<q(x)\}$. Define binary distributions $\tilde{p}=(p(A),p(A^c))$ and $\tilde{q}=(q(A),q(A^c))$. By the data processing inequality for KL divergence (applying the deterministic partition function), $D(\tilde{p}\Vert \tilde{q})\le D(p\Vert q)$. Also $\delta (\tilde{p},\tilde{q})=\delta (p,q)$. Applying the binary case to $\tilde{p}$ and $\tilde{q}$:

$$D(p\Vert q)\ge D(\tilde{p}\Vert \tilde{q})\ge 2\delta (\tilde{p},\tilde{q}{)}^2=2\delta (p,q{)}^2.\square$$

Theorem (Subadditivity of entropy). For any collection of discrete random variables $X_1,\dots ,X_n$:

$$H(X_1,\dots ,X_n)\le \sum _{i=1}^{n}H(X_i).$$

Equality holds if and only if $X_1,\dots ,X_n$ are mutually independent.

Proof. By the chain rule, $H(X_1,\dots ,X_n)={\sum }_{i=1}^{n}H(X_i|X_1,\dots ,X_{i-1})$. Since conditioning reduces entropy, $H(X_i|X_1,\dots ,X_{i-1})\le H(X_i)$ for each $i$. Summing gives subadditivity. Equality requires $H(X_i|X_{<i})=H(X_i)$ for all $i$, which by the equality condition for conditioning-reduces-entropy holds iff each $X_i$ is independent of $(X_1,\dots ,X_{i-1})$, i.e., mutual independence. $\square$

Theorem (Data processing inequality for KL divergence). Let $P$ and $Q$ be distributions on $\mathcal{X}$ and let $f:\mathcal{X}\to \mathcal{Y}$ be a measurable function. If $P_f=P\circ f^{-1}$ and $Q_f=Q\circ f^{-1}$ are the pushforward distributions on $\mathcal{Y}$, then

$$D(P_f\Vert Q_f)\le D(P\Vert Q).$$

Proof. Let $p_y={\sum }_{x:f(x)=y}p(x)$ and $q_y={\sum }_{x:f(x)=y}q(x)$. Then:

$$D(P\Vert Q)=\sum _{x}p(x)\log \frac{p(x)}{q(x)}=\sum _y\sum _{x:f(x)=y}p(x)\log \frac{p(x)}{q(x)}.$$

By the log-sum inequality applied to each $y$ with $a_x=p(x)$ and $b_x=q(x)$ for $x\in f^{-1}(y)$:

$$\sum _{x:f(x)=y}p(x)\log \frac{p(x)}{q(x)}\ge p_y\log \frac{p_y}{q_y}.$$

Summing over $y$ gives $D(P\Vert Q)\ge D(P_f\Vert Q_f)$. $\square$

Bridge. The log-sum inequality builds toward the asymptotic equipartition property in 46.02.01 by serving as the algebraic engine behind the data processing inequality, which controls how quickly information degrades under noisy processing. It appears again in the channel coding converse 46.03.01 where Fano's inequality translates conditional entropy into a lower bound on error probability, and in hypothesis testing 46.04.01 where the data processing inequality for KL divergence limits the discriminability of distributions after processing. The foundational reason these inequalities chain together is that the log-sum inequality is the single algebraic fact from which all basic information inequalities follow; the central insight is that convexity of the log-ratio functional generates every comparison theorem in the subject; and putting these together gives a unified inequality framework. The bridge is that Pinsker's inequality connects the information-theoretic world (KL divergence) to the statistical world (total variation), making it possible to translate abstract divergence bounds into concrete probability guarantees.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not define total variation distance on probability measures connected to KL divergence, and the Pinsker inequality is absent. The log-sum inequality and Gibbs' inequality are unformalised. Fano's inequality requires a definition of estimation error probability tied to conditional entropy, which does not exist in Mathlib's current information-theoretic layer. The data processing inequality for KL divergence requires a pushforward construction on PMFs coupled with the log-sum inequality; neither the pushforward-inequality connection nor the log-sum proof exists. The binary Pinsker inequality (the key technical step) requires bounding the binary KL divergence below by a quadratic in the parameter difference, which involves calculus on the Bernoulli family that is straightforward but unformalised. A Codex.InformationTheory.Pinsker module proving the Pinsker inequality for PMF pairs via the binary reduction would be the load-bearing first step; this unit ships without it.

Advanced results Master

Tightness of Pinsker's inequality and refinements

Pinsker's inequality with the constant $1/2$ is optimal in the following sense: there exist distributions where the ratio $D(P\Vert Q)/\delta (P,Q{)}^2$ is arbitrarily close to 2. For binary distributions with $p=1/2$ and $q=1/2+ϵ$:

$$\frac{D(p\Vert q)}{2{\delta }^2}=\frac{d(1/2\Vert 1/2+ϵ)}{2(ϵ/2{)}^2}=\frac{2{ϵ}^2+O({ϵ}^4)}{{ϵ}^2/2}\to 4\ne 2,$$

so the constant 2 in $D\ge 2{\delta }^2$ is not achieved but is the best universal constant (the infimum of $D/(2{\delta }^2)$ over all distribution pairs is 1, achieved in the limit as $\delta \to 0$ for uniform distributions).

Fedotov, Harremoes, and Topsoe (2003) proved a family of refinements. One particularly useful form is:

$$D(P\Vert Q)\ge \frac{2{\delta }^2}{1-2\delta +2\delta \log \delta +2(1-\delta )\log (1-\delta )},$$

for $\delta \in (0,1/2]$, where $\delta =\delta (P,Q)$. This is tighter than Pinsker for large divergences.

The Bretagnolle-Huber inequality

An alternative to Pinsker that is tighter for large divergences:

$$\delta (P,Q)\le \min \left(\sqrt{D(P\Vert Q)/2},1-\frac{1}{2}\exp (-D(P\Vert Q))\right).$$

The second bound $1-\exp (-D)/2$ is always at least as good as Pinsker for $D\ge 2$ and provides a meaningful bound even when $D$ is large. This inequality is derived from the fact that for any event $A$, $P(A)+Q(A^c)\ge \exp (-D(P\Vert Q))$, which follows from the data processing inequality applied to the indicator of $A$.

Generalized Fano inequalities

The standard Fano inequality can be sharpened in several directions.

Theorem (Han's inequality for conditional entropy). For random variables $X_1,\dots ,X_n$ and a random variable $Y$:

$$H(X_1,\dots ,X_n|Y)\le \frac{1}{n-1}\sum _{i=1}^{n}H(X_1,\dots ,X_{i-1},X_{i+1},\dots ,X_n|Y).$$

This follows from the subadditivity of conditional entropy and the same argument used for the Han inequality in 46.01.01, applied conditionally.

Corollary (Fano for list decoding). If $\hat{X}$ is an estimate that produces a list $L$ of size $\ell$ (i.e., $\hat{X}$ returns a set of $\ell$ candidates), and $P_e=\mathrm{Pr}[X\notin L]$, then

$$H(X|Y)\le H_b(P_e)+P_e{\log }_2(|\mathcal{X}|-\ell )+(1-P_e){\log }_2\ell .$$

This generalizes Fano by replacing the single guess with a list and is used in the list-decoding analysis of channel codes.

The data processing inequality and contiguity

The data processing inequality for KL divergence has a powerful converse-like refinement in terms of contiguity.

Definition (Contiguity). A sequence of distributions $P_n$ is contiguous with respect to $Q_n$ (written $P_n◃Q_n$) if for any sequence of events $A_n$, $Q_n(A_n)\to 0$ implies $P_n(A_n)\to 0$.

The connection to KL divergence: if $D(P_n\Vert Q_n)\to 0$, then $P_n◃Q_n$ and $Q_n◃P_n$ (mutual contiguity). This follows from Pinsker's inequality: $D(P_n\Vert Q_n)\to 0$ implies $\delta (P_n,Q_n)\to 0$, which implies $|P_n(A_n)-Q_n(A_n)|\to 0$ for all events, hence contiguity. The reverse does not hold: contiguity does not imply vanishing KL divergence.

Maximal coupling and the total-variation identity

There is a probabilistic characterisation of total variation distance that is often more useful than the definition.

Theorem (Maximal coupling). For any two distributions $P$ and $Q$ on a common measurable space, there exists a joint distribution $\pi$ on $\mathcal{X}\times \mathcal{X}$ with marginals $P$ and $Q$ such that

$$\pi \{(x,y):x=y\}=1-\delta (P,Q).$$

Equivalently, $\delta (P,Q)={\inf }_{\pi }{\mathrm{Pr}}_{(X,Y)\sim \pi }[X\ne Y]$, where the infimum is over all couplings $\pi$ of $P$ and $Q$. The infimum is achieved by the maximal coupling.

This identity is the reason Pinsker's inequality is useful in concentration arguments: a bound on KL divergence translates via Pinsker to a bound on total variation, which translates via maximal coupling to a bound on the probability that two coupled processes diverge.

Synthesis. The log-sum inequality builds toward every comparison theorem in information theory by serving as the single algebraic fact from which Gibbs' inequality, the data processing inequality, and joint convexity of KL divergence all derive. It appears again in the channel coding converse 46.03.01 where Fano's inequality transforms information bounds into error-probability bounds. The central insight is that Pinsker's inequality is the bridge between the information-theoretic world of KL divergence and the probabilistic world of total variation distance; this is exactly the tool that makes it possible to translate abstract divergence bounds into concrete probability guarantees. Putting these together: the log-sum inequality generates all basic information inequalities; subadditivity constrains how uncertainty distributes across variables; Fano translates conditional entropy into error bounds; and Pinsker connects divergence to distance. The bridge is that these four results form a closed loop — divergence bounds distance, distance bounds coupling error, coupling error bounds estimation accuracy, and estimation accuracy feeds back into conditional entropy via Fano.

Full proof set Master

Proposition (Gibbs' inequality from the log-sum inequality). For probability distributions $p$ and $q$ on $\mathcal{X}$:

$$D(p\Vert q)\ge 0,$$

with equality iff $p=q$.

Proof. Apply the log-sum inequality with $a_i=p(x_i)$ and $b_i=q(x_i)$ for $i=1,\dots ,n=|\mathcal{X}|$:

$$\sum _{i=1}^{n}p(x_i)\log \frac{p(x_i)}{q(x_i)}\ge \left(\sum _{i}p(x_i)\right)\log \frac{{\sum }_{i}p(x_i)}{{\sum }_{i}q(x_i)}=1\cdot \log \frac{1}{1}=0.$$

Equality holds iff $p(x_i)/q(x_i)$ is constant for all $i$ with $p(x_i)+q(x_i)>0$. Since both are probability distributions summing to 1, this constant must be 1, giving $p=q$. $\square$

Proposition (Data processing for KL implies data processing for mutual information). If $X\to Y\to Z$ is a Markov chain, then $I(X;Z)\le I(X;Y)$.

Proof. Let $P_{XZ}$ and $P_{XY}$ denote the joint distributions. The Markov condition means $P_{Z|XY}=P_{Z|Y}$. Now:

$$I(X;Z)=D(P_{XZ}\Vert P_X\otimes P_Z).$$

Define the function $f:(x,y)\mapsto (x,g(y))$ where $g$ is the channel from $Y$ to $Z$. The pushforward of $P_{XY}$ under $f$ is $P_{XZ}$, and the pushforward of $P_X\otimes P_Y$ is $P_X\otimes P_Z$ (since $Z$ depends only on $Y$). By the data processing inequality for KL divergence:

$$D(P_{XZ}\Vert P_X\otimes P_Z)\le D(P_{XY}\Vert P_X\otimes P_Y)=I(X;Y).\square$$

Proposition (Subadditivity implies the Han inequality). For any $n$ random variables $X_1,\dots ,X_n$:

$$H(X_1,\dots ,X_n)\le \frac{1}{n-1}\sum _{i=1}^{n}H(X_1,\dots ,X_{i-1},X_{i+1},\dots ,X_n).$$

Proof. For each $i$, subadditivity applied to the $n-1$ variables excluding $X_i$ gives:

$$H(X_1,\dots ,X_{i-1},X_{i+1},\dots ,X_n)\le \sum _{j\ne i}H(X_j).$$

Also, the chain rule gives $H(X_{1:n})=H(X_i)+H(X_{-i}|X_i)\le H(X_i)+H(X_{-i})$, so $H(X_i)\ge H(X_{1:n})-H(X_{-i})$.

Summing over $i$: ${\sum }_{i}H(X_i)\ge n\cdot H(X_{1:n})-{\sum }_{i}H(X_{-i})$.

By subadditivity, ${\sum }_{i}H(X_i)\ge H(X_{1:n})$, so $n\cdot H(X_{1:n})-{\sum }_{i}H(X_{-i})\le {\sum }_{i}H(X_i)$.

Rearranging: $(n-1)H(X_{1:n})\le {\sum }_{i}H(X_{-i})$, giving $H(X_{1:n})\le \frac{1}{n-1}{\sum }_{i}H(X_{-i})$. $\square$

Proposition (Fano's inequality, sharp form for uniform prior). If $X$ is uniform on $\{1,\dots ,m\}$ and $\hat{X}$ is any estimator based on $Y$, then:

$$P_e\ge 1-\frac{I(X;Y)+1}{{\log }_{2}m}.$$

Proof. Since $X$ is uniform, $H(X)={\log }_{2}m$. By Fano's inequality:

$$H(X|Y)\le H_b(P_e)+P_e{\log }_2(m-1).$$

Since $I(X;Y)=H(X)-H(X|Y)={\log }_{2}m-H(X|Y)$, we get:

$$I(X;Y)\ge {\log }_{2}m-H_b(P_e)-P_e{\log }_2(m-1).$$

Since $H_b(P_e)\le 1$ and ${\log }_2(m-1)\le {\log }_{2}m$:

$$I(X;Y)\ge {\log }_{2}m-1-P_e{\log }_{2}m=(1-P_e){\log }_{2}m-1.$$

Rearranging: $P_e\ge 1-(I(X;Y)+1)/{\log }_{2}m$. $\square$

Connections Master

• 46.01.01 — Entropy, conditional entropy, and the chain rule are the direct prerequisites; subadditivity $H(X_{1:n})\le \sum H(X_i)$ follows from the chain rule plus conditioning-reduces-entropy.
• 46.01.02 — KL divergence and mutual information are defined here; Pinsker's inequality bounds total variation by $\sqrt{D/2}$, and the data processing inequality for KL divergence uses the log-sum inequality.
• 46.02.01 — The AEP uses subadditivity to prove the entropy rate exists for stationary sources and Pinsker to control the deviation of empirical distributions from the true distribution.
• 46.03.01 — The channel coding converse uses Fano's inequality to translate mutual information bounds into lower bounds on error probability for rates above capacity.
• 46.04.01 — Hypothesis testing uses the data processing inequality for KL divergence to bound the performance of statistical tests based on processed data, and Pinsker connects information divergences to the total-variation test criterion.
• 37.07.05 — Sanov's theorem in large deviations identifies $D(q\Vert p)$ as the rate function; Pinsker's inequality provides the continuity bound needed to show the rate function is well-behaved.

Historical & philosophical context Master

The log-sum inequality is implicit in the Gibbs inequality used by Shannon (1948) and was made explicit by Kullback and Leibler (1951) as the algebraic foundation for the non-negativity of KL divergence. Its name comes from the observation that the logarithm of a sum is bounded by the weighted sum of logarithms, a fact that sits between Jensen's inequality and the convexity of the log-partition function.

Pinsker's inequality was first proved by Mark Pinsker in 1960 and published in Doklady Akademii Nauk SSSR (volume 160, 194-197) with an English translation in Soviet Mathematics Doklady (volume 6, 88-91) in 1965. Pinsker was working on information stability of random variables and needed a concrete connection between the information-theoretic divergence $D(P\Vert Q)$ and the probability-theoretic distance $\delta (P,Q)$. The original proof used the binary reduction argument (partition the alphabet into two sets where $P$ and $Q$ differ maximally, then reduce to the Bernoulli case) and established the constant $1/2$ as optimal. Kemperman (1967, On the optimal rate of transmitting information) and Csiszar (1967, Information-type measures of difference of probability distributions) independently proved the same result in greater generality, showing that the constant holds for arbitrary measurable spaces.

Imre Csiszar's 1967 paper is notable for introducing the $f$-divergence framework, which unified KL divergence, total variation distance, and the chi-squared divergence under a single family $D_f(P\Vert Q)=\int f(dP/dQ)dQ$ for convex $f$ with $f(1)=0$. Within this framework, Pinsker's inequality becomes a statement about the relationship between the KL divergence ($f(t)=t\log t$) and the total variation ($f(t)=|t-1|/2$): one $f$-divergence dominates another. The question of which $f$-divergences dominate which others was subsequently studied extensively by Liese and Vajda in their 1987 monograph Convex Statistical Distances.

Fano's inequality appears in Robert Fano's 1952 MIT technical report Class Notes for Transmission of Information (course 6.574) and was published in his 1961 textbook Transmission of Information (MIT Press/Wiley). Fano was motivated by the problem of computing the channel capacity of the binary symmetric channel and needed a tool to convert bounds on mutual information into bounds on the probability of decoding error. The inequality was used in the original proof of the converse to Shannon's channel coding theorem and remains the standard tool for converse arguments in network information theory.

The maximal coupling characterisation of total variation was developed by several authors independently. The existence of an optimal coupling achieving $\mathrm{Pr}[X\ne Y]=\delta (P,Q)$ was shown by Doeblin (1938) for discrete distributions and extended to the general case by Goldstein (1979) and Thorisson (2000, Coupling, Stationarity, and Regeneration, Springer). The coupling perspective gives Pinsker's inequality a probabilistic reading: if the KL divergence between $P$ and $Q$ is small, then there exists a coupling in which $P$ and $Q$ agree with high probability.

Bibliography Master

@book{cover-thomas2006,
  author    = {Cover, T. M. and Thomas, J. A.},
  title     = {Elements of Information Theory},
  edition   = {2nd},
  publisher = {Wiley},
  year      = {2006},
}
@book{csiszar-korner2011,
  author    = {Csisz{\'a}r, I. and K{\"o}rner, J.},
  title     = {Information Theory: Coding Theorems for Discrete Memoryless Systems},
  edition   = {2nd},
  publisher = {Cambridge University Press},
  year      = {2011},
}
@article{pinsker1964,
  author  = {Pinsker, M. S.},
  title   = {Information and Information Stability of Random Variables and Processes},
  journal = {Problems of Information Transmission},
  year    = {1964},
  note    = {Originally published in Russian by Nauka, Moscow; the inequality $\delta^2 \leq D/2$ first appeared in Doklady Akademii Nauk SSSR 160 (1965) 194-197},
}
@article{fedotov-harremoes-topsoe2003,
  author  = {Fedotov, A. A. and Harremoes, P. and Tops{\o}e, F.},
  title   = {Refinements of {Pinsker's} Inequality},
  journal = {IEEE Transactions on Information Theory},
  volume  = {49},
  number  = {6},
  pages   = {1491--1498},
  year    = {2003},
}
@article{csiszar1967,
  author  = {Csisz{\'a}r, I.},
  title   = {Information-Type Measures of Difference of Probability Distributions and Indirect Observations},
  journal = {Studia Scientiarum Mathematicarum Hungarica},
  volume  = {2},
  pages   = {299--318},
  year    = {1967},
}
@book{fano1961,
  author    = {Fano, R. M.},
  title     = {Transmission of Information: A Statistical Theory of Communications},
  publisher = {MIT Press and Wiley},
  year      = {1961},
}
@article{kullback-leibler1951,
  author  = {Kullback, S. and Leibler, R. A.},
  title   = {On Information and Sufficiency},
  journal = {Annals of Mathematical Statistics},
  volume  = {22},
  pages   = {79--86},
  year    = {1951},
}
@book{liese-vajda1987,
  author    = {Liese, F. and Vajda, I.},
  title     = {Convex Statistical Distances},
  publisher = {Teubner},
  year      = {1987},
}
@book{thorisson2000,
  author    = {Thorisson, H.},
  title     = {Coupling, Stationarity, and Regeneration},
  publisher = {Springer},
  year      = {2000},
}