Differential Entropy, the Maximum-Entropy Gaussian, and the Entropy-Power Inequality

Intuition Beginner

Shannon entropy counts bits of surprise for discrete outcomes — coin flips, die rolls, letters of the alphabet. But many real-world signals are continuous. A voltage on a wire can take any value in an interval, a temperature reading is a real number, and an audio waveform varies smoothly over time. To measure uncertainty in these continuous settings, we need a continuous version of entropy.

Differential entropy replaces the sum in Shannon entropy with an integral. Instead of summing over discrete outcomes weighted by their probability, you integrate over all real values weighted by the probability density. The formula is almost the same — you just swap the sum for an integral and the probability mass function for a probability density function.

There is a catch. Differential entropy can be negative. A discrete random variable always has non-negative entropy, but a continuous one concentrated tightly around a single point can have negative differential entropy. This is unsettling at first, but it makes sense once you think about what the continuous version measures: it measures uncertainty relative to the coordinate system, not in absolute bits. The quantity that is always non-negative is mutual information, which compares two distributions and cancels the coordinate dependence.

The Gaussian distribution plays a starring role. Among all continuous distributions with a fixed variance, the Gaussian has the largest differential entropy. This means the Gaussian is the most uncertain, the hardest to predict, the most spread out in an information-theoretic sense. If you know only the variance of a signal, the safest assumption is that it is Gaussian, because that assumption preserves the most uncertainty.

The entropy-power inequality is the deepest result in this unit. It says that when you add two independent continuous random variables together, the resulting variable is at least as uncertain as a certain average of the two originals would suggest. More precisely, the "entropy power" of the sum is at least the sum of the entropy powers of the parts. This inequality underpins the proof that Gaussian noise is the worst-case noise for channel capacity.

Visual Beginner

Figure: three probability density functions plotted on the same axes, all with variance 1. A tall narrow spike (concentrated distribution, low entropy), a medium bell curve (moderate entropy), and the standard Gaussian (maximum entropy). Below each, the differential entropy value: negative for the spike, positive but suboptimal for the medium curve, and exactly $h=\frac{1}{2}\log (2\pi e)\approx 1.42$ nats for the Gaussian.

Distribution	Variance	Differential entropy (nats)	Notes
Uniform on $[0,1]$	$1/12$	$0$	Boundary case — neither positive nor negative
Uniform on $[0,a]$	$a^2/12$	$\log a$	Positive when $a>1$, negative when $a<1$
Standard Gaussian $N(0,1)$	$1$	$\frac{1}{2}\log (2\pi e)\approx 1.42$	Maximum for variance 1
Gaussian $N(0,{\sigma }^2)$	${\sigma }^2$	$\frac{1}{2}\log (2\pi e{\sigma }^2)$	Grows as $\sigma$ grows
Point mass at 0	$0$	$-\infty$	No uncertainty at all

The pattern: spreading a distribution out (increasing variance) increases differential entropy. For a given variance, the Gaussian is the spread-optimiser.

Worked example Beginner

Example 1: Uniform distribution on $[0,a]$.

A continuous random variable $X$ is uniform on the interval $[0,a]$, meaning its density is $1/a$ on that interval and zero everywhere else. What is its differential entropy?

Think of it this way: the density is constant at $1/a$, so every point in $[0,a]$ is equally likely (per unit length). The information content at each point is $\log a$, because the density is $1/a$ and the surprise is $\log (1/(1/a))=\log a$. The average is just $\log a$ itself, since the density is constant.

In nats: $h(X)=\log a$. In bits: $h(X)={\log }_{2}a$.

When $a=2$, the entropy is 1 bit — the same as a fair coin flip, because the interval has length 2 and the density is $1/2$. When $a=1$, the entropy is 0. When $a=1/2$, the entropy is $\log (1/2)=-1$ bit, which is negative. A distribution crammed into a tiny interval has negative differential entropy.

Example 2: Standard Gaussian $N(0,1)$.

The standard Gaussian has density $f(x)=\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2}$. Its differential entropy is a clean formula: $h=\frac{1}{2}\log (2\pi e)\approx 1.42$ nats, or about 2.05 bits. This is the maximum differential entropy achievable by any distribution with variance 1. Every other distribution with the same variance has lower entropy — it is more predictable, more concentrated, or more skewed.

Check your understanding Beginner

Formal definition Intermediate+

Let $X$ be a continuous random variable with probability density function $f(x)$ supported on $\mathbb{R}$ (or a subset thereof).

Definition (Differential entropy). The differential entropy of $X$ is

$$h(X)=-{\int }_{-\infty }^{\infty }f(x)\log f(x)dx=\mathbb{E}[-\log f(X)].$$

The integral is taken in the Lebesgue sense. The logarithm may be natural (giving entropy in nats) or base-2 (giving bits). Throughout this unit we use natural log unless otherwise noted.

Definition (Joint and conditional differential entropy). For continuous random variables $X$ and $Y$ with joint density $f(x,y)$ and marginal densities $f_X(x)$, $f_Y(y)$:

$$h(X,Y)=-\iint f(x,y)\log f(x,y)dxdy,$$ $$h(X\mid Y)=-\iint f(x,y)\log f(x\mid y)dxdy=h(X,Y)-h(Y).$$

These definitions mirror the discrete case exactly, with sums replaced by integrals and probability mass functions replaced by densities.

Definition (Entropy power). The entropy power of a continuous random variable $X$ with differential entropy $h(X)$ is

$$N(X)=\frac{1}{2\pi e}{e}^{2h(X)}.$$

The entropy power has units of variance. For a Gaussian $N(0,{\sigma }^2)$, one can verify that $N(X)={\sigma }^2$: the entropy power equals the actual variance. For non-Gaussian distributions, $N(X)\le \mathrm{Var}(X)$, with equality iff $X$ is Gaussian. The entropy power is the variance of the Gaussian that would have the same differential entropy as $X$.

Counterexamples to common slips

• Differential entropy is not the limit of discrete entropy. Quantising a continuous variable with bin width $\Delta$ gives discrete entropy $H(X^{\Delta })\approx h(X)-\log \Delta$. As $\Delta \to 0$ the discrete entropy diverges to $+\infty$; the $-\log \Delta$ term cancels the divergence, leaving a finite differential entropy. The two quantities differ by an infinite constant.

• Negative entropy does not mean negative uncertainty. Differential entropy measures uncertainty relative to the Lebesgue measure. A negative value means the distribution is more concentrated than the uniform distribution on a unit interval, not that uncertainty is impossible.

• The entropy-power inequality is not subadditivity of entropy. The EPI says $e^{2h(X+Y)}\ge e^{2h(X)}+e^{2h(Y)}$, which is weaker than $h(X+Y)\ge h(X)$ (the latter is false in general). The exponential transformation is load-bearing.

Key theorem with proof Intermediate+

Theorem (Gaussian maximum entropy). Let $X$ be a continuous random variable with variance ${\sigma }^2$. Then

$$h(X)\le \frac{1}{2}\log (2\pi e{\sigma }^2),$$

with equality if and only if $X\sim N(\mu ,{\sigma }^2)$ for some $\mu$.

Proof. Let $f$ be the density of $X$ and let $g$ be the density of $N(\mu ,{\sigma }^2)$, where $\mu =\mathbb{E}[X]$ and ${\sigma }^2=\mathrm{Var}(X)$. Compute the KL divergence:

$$D(f\Vert g)=\int f(x)\log \frac{f(x)}{g(x)}dx\ge 0.$$

Expanding the logarithm:

$$D(f\Vert g)=-h(X)-\int f(x)\log g(x)dx.$$

Now evaluate the second integral. Since $g(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-(x-\mu {)}^2/(2{\sigma }^2)}$, we have $\log g(x)=-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{(x-\mu {)}^2}{2{\sigma }^2}$. Therefore:

$$\int f(x)\log g(x)dx=-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}\mathbb{E}[(X-\mu {)}^2]=-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{1}{2}.$$

So $D(f\Vert g)=-h(X)+\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{1}{2}=-h(X)+\frac{1}{2}\log (2\pi e{\sigma }^2)\ge 0$, giving $h(X)\le \frac{1}{2}\log (2\pi e{\sigma }^2)$.

Equality holds iff $D(f\Vert g)=0$, which happens iff $f=g$ almost everywhere, i.e., $X$ is Gaussian. $\square$

Theorem (Entropy-power inequality). Let $X$ and $Y$ be independent continuous random variables with densities. Then

$$e^{2h(X+Y)}\ge e^{2h(X)}+e^{2h(Y)},$$

with equality if and only if $X$ and $Y$ are Gaussian with proportional variances.

Proof sketch. The standard proof proceeds via the de Bruijn identity and the Fisher information inequality. Define $X_t=X+\sqrt{t}Z$ where $Z\sim N(0,1)$ is independent of $X$. The de Bruijn identity states that

$$\frac{d}{dt}h(X_t)=\frac{1}{2}J(X_t),$$

where $J(X_t)=\int \frac{(f_t^{\prime }(x){)}^2}{f_t(x)}dx$ is the Fisher information of $X_t$ with density $f_t$. The Fisher information inequality for the sum of independent variables gives $J(X+Y)\le {\left(\frac{1}{J(X)}+\frac{1}{J(Y)}\right)}^{-1}$ when $X$ and $Y$ are independent. Integrating the de Bruijn identity from $t=0$ to $t=\infty$ (where $X_t$ approaches a Gaussian) and using the Fisher information inequality at each step yields the EPI. Equality requires both $X$ and $Y$ to be Gaussian. $\square$

Bridge. The maximum-entropy property of the Gaussian builds toward the Gaussian channel capacity in 46.03.01, where the optimal input distribution for a power-constrained channel is Gaussian precisely because it maximises differential entropy. The entropy-power inequality appears again in the proof of the converse of the Gaussian channel coding theorem 46.03.01, where it establishes the foundational reason that no coding scheme can exceed $C=\frac{1}{2}\log (1+P/N)$. The EPI generalises to vector-valued random variables and is dual to the Brunn-Minkowski inequality in convex geometry, putting these together reveals that information-theoretic and geometric uncertainty are two facets of the same mathematical structure.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib has the infrastructure for continuous probability distributions — MeasureTheory.Integral, ProbabilityTheory.ProbabilityDistribution, and ProbabilityTheory.Gaussian with the Gaussian density — but no definition of differential entropy as a functional $h:\text{Density}\to \mathbb{R}$. The max-entropy property (Gaussian maximises $h$ under variance constraint) would require defining KL divergence for continuous distributions and proving $D(f\Vert g)\ge 0$ in the Lebesgue integral setting. The entropy-power inequality would further need the de Bruijn identity linking Fisher information and entropy under Gaussian perturbation, plus the Fisher information inequality for sums. None of these are in Mathlib. A Codex.InformationTheory.DifferentialEntropy module defining $h$ on absolutely continuous distributions and proving the max-entropy property via KL divergence would be the load-bearing first step; this unit ships without it.

Advanced results Master

The de Bruijn identity and its role in proving the EPI

The de Bruijn identity provides the key analytical link between Fisher information and differential entropy. Let $X$ be a random variable with density $f$, and let $Z\sim N(0,I)$ be independent of $X$. Define the heat flow $X_t=X+\sqrt{t}Z$ with density $f_t$. Then:

$$\frac{d}{dt}h(X_t)=\frac{1}{2}J(X_t),$$

where $J(X)=\int \frac{(f^{\prime }(x){)}^2}{f(x)}dx$ is the Fisher information (the second derivative of the KL divergence). The identity is proved by differentiating $h(X_t)=-\int f_t\log f_t$ under the integral sign and using the fact that $f_t$ satisfies the heat equation $\frac{\partial f_t}{\partial t}=\frac{1}{2}\Delta f_t$.

The de Bruijn identity is the engine of the EPI proof. The strategy is:

1. Express $h(X_t)$ in terms of the Fisher information via de Bruijn.
2. Use the Fisher information inequality $J(X+Y)\le {\left(\frac{1}{J(X)}+\frac{1}{J(Y)}\right)}^{-1}$ for independent $X,Y$.
3. Integrate from $t=0$ (the original variables) to $t=\infty$ (where everything is Gaussian and the inequality is tight).

The Fisher information inequality itself follows from the convolution identity for Fisher information and the Cauchy-Schwarz inequality. This three-step structure — de Bruijn, Fisher inequality, integration — is the template for many information-theoretic inequalities.

Relation to the Gaussian channel capacity

Shannon's formula for the capacity of the additive white Gaussian noise (AWGN) channel is

$$C=\frac{1}{2}\log \left(1+\frac{P}{N}\right)\text{ bits per channel use},$$

where $P$ is the signal power constraint and $N$ is the noise variance. This is derived as follows. The channel output is $Y=X+Z$ where $Z\sim N(0,N)$ and the input satisfies $\mathbb{E}[X^2]\le P$. The mutual information is

$$I(X;Y)=h(Y)-h(Y|X)=h(Y)-h(Z).$$

Since $h(Z)=\frac{1}{2}\log (2\pi eN)$ is fixed, maximising $I(X;Y)$ is equivalent to maximising $h(Y)$. The variance of $Y$ is $\mathrm{Var}(Y)=\mathrm{Var}(X)+N\le P+N$. By the max-entropy theorem, $h(Y)$ is maximised when $Y$ is Gaussian, which happens when $X$ is Gaussian. This gives $h(Y)=\frac{1}{2}\log (2\pi e(P+N))$ and

$$C=\frac{1}{2}\log (2\pi e(P+N))-\frac{1}{2}\log (2\pi eN)=\frac{1}{2}\log \left(1+\frac{P}{N}\right).$$

The EPI enters the converse proof: it shows that no input distribution can achieve a higher rate, because the entropy power of $Y=X+Z$ is bounded by the sum of entropy powers of $X$ and $Z$.

Generalised EPI and Brunn-Minkowski connection

The EPI has a deep geometric interpretation. The Brunn-Minkowski inequality in convex geometry states that for compact sets $A,B\subseteq {\mathbb{R}}^n$:

$$\mathrm{Vol}(A+B{)}^{2/n}\ge \mathrm{Vol}(A{)}^{2/n}+\mathrm{Vol}(B{)}^{2/n},$$

where $A+B=\{a+b:a\in A,b\in B\}$ is the Minkowski sum. The EPI is the information-theoretic analogue: replace volume by entropy power and Minkowski sum by convolution of densities. Dembo, Cover, and Thomas (1991) showed that the EPI and Brunn-Minkowski inequality are equivalent in a precise sense — each can be derived from the other via a limiting argument that approximates distributions by indicator functions of sets.

The generalised EPI for $n$-dimensional random vectors states:

$$e^{2h(\mathbf{X}+\mathbf{Y})/n}\ge e^{2h(\mathbf{X})/n}+e^{2h(\mathbf{Y})/n},$$

where $h(\mathbf{X})$ is the joint differential entropy of the random vector $\mathbf{X}\in {\mathbb{R}}^n$.

Maximum entropy under other constraints

The Gaussian is the maximum-entropy distribution under a variance constraint. Other constraints yield other extremal distributions:

Constraint	Maximum-entropy distribution
Support on $[a,b]$	Uniform on $[a,b]$
Variance ${\sigma }^2$	Gaussian $N(\mu ,{\sigma }^2)$
Mean $\mu$ (positive support)	Exponential with rate $1/\mu$
Mean $\mu$ and variance ${\sigma }^2$ on $\mathbb{R}$	Gaussian $N(\mu ,{\sigma }^2)$

Each of these is proved by the same KL divergence argument: if $g$ is the extremal distribution and $f$ is any other distribution satisfying the same constraint, then $D(f\Vert g)\ge 0$ expands to give $h(f)\le h(g)$. The constraints determine $g$; the non-negativity of KL divergence does the rest.

Synthesis. Differential entropy extends Shannon's uncertainty measure from discrete to continuous distributions via integration; the Gaussian is the maximum-entropy distribution under a variance constraint, which is the foundational reason it appears as the optimal input in Gaussian channel coding 46.03.01; the entropy-power inequality $e^{2h(X+Y)}\ge e^{2h(X)}+e^{2h(Y)}$ constrains how uncertainty combines under addition and is dual to the Brunn-Minkowski inequality in convex geometry; and the de Bruijn identity linking Fisher information and differential entropy via the heat equation is the central insight that makes the EPI provable, generalising to vector-valued random variables and to the full family of information-theoretic inequalities in the Dembo-Cover-Thomas framework.

Full proof set Master

Proposition (Scaling of differential entropy). For any continuous random variable $X$ with differential entropy $h(X)$ and any constant $a\ne 0$:

$$h(aX)=h(X)+\log |a|.$$

Proof. Let $f$ be the density of $X$ and $g$ be the density of $Y=aX$. Then $g(y)=\frac{1}{|a|}f\left(\frac{y}{a}\right)$. Compute:

$$\begin{array}{rl}h(Y)& =-\int g(y)\log g(y)dy\\ & =-\int \frac{1}{|a|}f\left(\frac{y}{a}\right)\log \left[\frac{1}{|a|}f\left(\frac{y}{a}\right)\right]dy.\end{array}$$

Substituting $x=y/a$ (so $dy=|a|dx$ and $y/a=x$):

$$\begin{array}{rl}h(Y)& =-\int f(x)[\log f(x)+\log (1/|a|)]dx\\ & =-\int f(x)\log f(x)dx-\log (1/|a|)\int f(x)dx\\ & =h(X)+\log |a|.\square \end{array}$$

Proposition (Differential entropy of the Gaussian). If $X\sim N(\mu ,{\sigma }^2)$, then $h(X)=\frac{1}{2}\log (2\pi e{\sigma }^2)$.

Proof. The density is $f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-(x-\mu {)}^2/(2{\sigma }^2)}$. Compute $\log f(x)=-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{(x-\mu {)}^2}{2{\sigma }^2}$. Then:

$$\begin{array}{rl}h(X)& =-\int f(x)\log f(x)dx\\ & =\frac{1}{2}\log (2\pi {\sigma }^2)\int f(x)dx+\frac{1}{2{\sigma }^2}\int (x-\mu {)}^{2}f(x)dx\\ & =\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{{\sigma }^2}{2{\sigma }^2}\\ & =\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{1}{2}\\ & =\frac{1}{2}\log (2\pi e{\sigma }^2).\square \end{array}$$

Note that the mean $\mu$ does not appear in the final expression: differential entropy depends on the spread of the distribution, not its location.

Proposition (Differential entropy of the uniform distribution). If $X$ is uniform on $[a,b]$, then $h(X)=\log (b-a)$.

Proof. The density is $f(x)=\frac{1}{b-a}$ for $x\in [a,b]$ and $0$ elsewhere. Then:

$$h(X)=-{\int }_a^b\frac{1}{b-a}\log \frac{1}{b-a}dx=\log (b-a)\cdot \frac{1}{b-a}{\int }_a^{b}dx=\log (b-a).\square$$

Proposition (Max-entropy under support constraint). Among all continuous random variables supported on $[a,b]$, the uniform distribution maximises differential entropy.

Proof. Let $f$ be any density supported on $[a,b]$ and let $u(x)=\frac{1}{b-a}$ be the uniform density. Compute the KL divergence:

$$D(f\Vert u)={\int }_a^{b}f(x)\log \frac{f(x)}{1/(b-a)}dx=-h(X)+\log (b-a)\ge 0.$$

Therefore $h(X)\le \log (b-a)=h(U)$, where $U$ is uniform on $[a,b]$. Equality holds iff $f=u$ a.e., i.e., $X$ is uniform. $\square$

Proposition (EPI implies Gaussian capacity converse). Let $Y=X+Z$ where $Z\sim N(0,N)$ and $\mathrm{Var}(X)\le P$. The EPI implies $I(X;Y)\le \frac{1}{2}\log (1+P/N)$.

Proof. We have $I(X;Y)=h(Y)-h(Z)$. Since $Z$ is Gaussian, $h(Z)=\frac{1}{2}\log (2\pi eN)$. By the EPI applied to $X$ and $Z$:

$$e^{2h(Y)}=e^{2h(X+Z)}\ge e^{2h(X)}+e^{2h(Z)}=e^{2h(X)}+2\pi eN.$$

By the max-entropy theorem under variance constraint, $h(X)\le \frac{1}{2}\log (2\pi eP)$, so $e^{2h(X)}\le 2\pi eP$. Therefore:

$$e^{2h(Y)}\ge 2\pi eP+2\pi eN=2\pi e(P+N).$$

Taking logs: $h(Y)\ge \frac{1}{2}\log (2\pi e(P+N))$. But also $h(Y)\le \frac{1}{2}\log (2\pi e(P+N))$ by the max-entropy theorem (since $\mathrm{Var}(Y)\le P+N$). So:

$$I(X;Y)=h(Y)-h(Z)\le \frac{1}{2}\log (2\pi e(P+N))-\frac{1}{2}\log (2\pi eN)=\frac{1}{2}\log (1+P/N).\square$$

Connections Master

• 46.01.01 — Shannon entropy is the discrete precursor; differential entropy replaces sums by integrals and pmfs by densities, and shares the same chain rule and conditioning structure.
• 46.01.02 — Mutual information for continuous variables is $I(X;Y)=h(X)-h(X|Y)$, built from differential entropy; the KL divergence proof of the max-entropy theorem is a direct application of the divergence defined there.
• 46.03.01 — Gaussian channel capacity $C=\frac{1}{2}\log (1+P/N)$ is derived from the max-entropy property of the Gaussian; the EPI proves the converse bound.
• 02.01.01 — Real analysis and Lebesgue integration provide the measure-theoretic foundation for the integral $h(X)=-\int f\log f$ and the dominated convergence arguments in the de Bruijn identity.
• 37.01.01 — Probability spaces, density functions, and the concept of absolutely continuous distributions are the prerequisites for defining differential entropy.
• 46.01.05 — Entropy rates for continuous-time processes extend differential entropy to stationary processes; the max-entropy property under spectral density constraints is the frequency-domain generalisation.

Historical & philosophical context Master

Claude Shannon introduced differential entropy in sections 23-25 of his 1948 paper "A Mathematical Theory of Communication" (Bell System Technical Journal 27, 623-656). He needed a continuous version of entropy to analyse the capacity of the Gaussian channel, which models the fundamental limits of radio, telephone, and optical communication. Shannon stated the entropy-power inequality without full proof, noting only that it "can be proven by a variational argument" and that equality holds for Gaussians. The gap in his proof stayed open for over a decade.

The first complete proof of the EPI was given by A. J. Stam in 1959 ("Some Inequalities Satisfied by the Quantities of Information of Fisher and Shannon," Information and Control 2, 101-112). Stam's insight was to use the de Bruijn identity to connect differential entropy to Fisher information, then prove a Fisher information inequality for sums of independent random variables, and finally integrate back to obtain the EPI. This proof strategy — link entropy to Fisher information via the heat equation, prove the inequality at the Fisher level, and integrate — has become the standard template.

The connection to the Brunn-Minkowski inequality was made explicit by Dembo, Cover, and Thomas in their 1991 paper "Information Theoretic Inequalities" (IEEE Transactions on Information Theory 37(6), 1501-1518). They showed that the EPI, the Brunn-Minkowski inequality, and several other classical inequalities are all instances of a single convexity property, unifying information theory and convex geometry.

The maximum-entropy principle has a broader philosophical significance. E. T. Jaynes argued in 1957 ("Information Theory and Statistical Mechanics," Physical Review 106, 620-630) that the max-entropy distribution is the most honest assignment of probabilities given limited knowledge: it assumes nothing beyond the stated constraints. Statistical mechanics (the Maxwell-Boltzmann distribution), thermodynamics (the Gibbs entropy), and Bayesian inference (the max-ent prior) all use this principle. In information theory, the max-entropy property of the Gaussian is not merely a mathematical fact but an operational one: it tells you the worst-case noise and the best-case signal for channel coding.

Bibliography Master

@article{shannon1948,
  author  = {Shannon, C. E.},
  title   = {A Mathematical Theory of Communication},
  journal = {Bell System Technical Journal},
  volume  = {27},
  pages   = {379--423 and 623--656},
  year    = {1948},
}
@book{cover-thomas2006,
  author    = {Cover, T. M. and Thomas, J. A.},
  title     = {Elements of Information Theory},
  edition   = {2nd},
  publisher = {Wiley},
  year      = {2006},
}
@article{stam1959,
  author  = {Stam, A. J.},
  title   = {Some Inequalities Satisfied by the Quantities of Information of {Fisher} and {Shannon}},
  journal = {Information and Control},
  volume  = {2},
  pages   = {101--112},
  year    = {1959},
}
@article{dembo-cover-thomas1991,
  author  = {Dembo, A. and Cover, T. M. and Thomas, J. A.},
  title   = {Information Theoretic Inequalities},
  journal = {IEEE Transactions on Information Theory},
  volume  = {37},
  number  = {6},
  pages   = {1501--1518},
  year    = {1991},
}
@article{jaynes1957,
  author  = {Jaynes, E. T.},
  title   = {Information Theory and Statistical Mechanics},
  journal = {Physical Review},
  volume  = {106},
  pages   = {620--630},
  year    = {1957},
}