Discrete Memoryless Channels, Mutual Information Maximization, and Channel Capacity

Intuition Beginner

Imagine you are sending a single bit — a 0 or a 1 — across a noisy wire. Most of the time the bit arrives intact, but occasionally the wire flips it. The wire does not remember what it did to the previous bit; each transmission is an independent event. This is a channel: a physical (or mathematical) process that takes an input symbol and produces an output symbol, with some probability of corrupting it.

A channel is completely described by a simple table: for every input, what is the probability of each possible output? For the noisy wire, the table says: if you send 0, you receive 0 with probability 0.99 and 1 with probability 0.01; if you send 1, you receive 1 with probability 0.99 and 0 with probability 0.01. This table is the transition matrix, and it captures everything there is to know about the noise.

Not all channels are binary. A printer might receive a pixel value and produce one of three outputs: the correct colour, a smeared version, or a blank. An optical fibre might receive a light pulse and detect either a pulse, no pulse, or an ambiguous signal. In every case, the channel is defined by its transition matrix: a row for each input, a column for each output, and probabilities in each cell that sum to one across each row.

The central question is: how much information can you push through a given channel per symbol, reliably, in the limit of many uses? That quantity is the channel capacity, measured in bits per channel use. A noiseless binary channel carries 1 bit per use — you get out exactly what you put in. A very noisy channel carries almost 0 bits per use — the output tells you almost nothing about the input. The capacity is a single number that captures the maximum reliable communication rate.

Visual Beginner

Figure: two channel diagrams side by side. On the left, a binary symmetric channel (BSC) with crossover probability $p=0.1$: two input nodes (0 and 1) on the left, two output nodes (0 and 1) on the right. Arrows from each input to each output are labelled with probabilities: 0→0 at 0.9, 0→1 at 0.1, 1→1 at 0.9, 1→0 at 0.1. On the right, a binary erasure channel (BEC) with erasure probability $e=0.2$: two input nodes (0 and 1), three output nodes (0, e, 1). Arrows: 0→0 at 0.8, 0→e at 0.2, 1→1 at 0.8, 1→e at 0.2.

Channel	Input alphabet	Output alphabet	Capacity	Key parameter
Noiseless binary	{0, 1}	{0, 1}	1 bit/use	none
BSC (p = 0.1)	{0, 1}	{0, 1}	0.531 bit/use	crossover p
BSC (p = 0.5)	{0, 1}	{0, 1}	0 bit/use	crossover p
BEC (e = 0.2)	{0, 1}	{0, 1, e}	0.8 bit/use	erasure e
BEC (e = 1.0)	{0, 1}	{0, 1, e}	0 bit/use	erasure e

The pattern: more noise means less capacity. At the extremes, a perfectly reliable channel carries 1 bit per use (for binary input), and a completely noisy channel carries 0.

Worked example Beginner

A binary symmetric channel flips each bit independently with probability $p=0.1$. What is its capacity?

Step 1. The channel takes 0 or 1 as input and produces 0 or 1 as output. Each bit is flipped with probability 0.1 and transmitted correctly with probability 0.9.

Step 2. The mutual information $I(X;Y)$ measures how much the output tells you about the input. For this channel, if you use a fair coin at the input (send 0 with probability 0.5 and 1 with probability 0.5), then:

• $H(Y)=1$ bit (the output is also uniform, because the symmetry preserves balance).
• $H(Y|X)=H(0.1)=-0.1{\log }_{2}0.1-0.9{\log }_{2}0.9\approx 0.33+0.137=0.469$ bits.

So $I(X;Y)=H(Y)-H(Y|X)=1-0.469=0.531$ bits.

Step 3. The capacity is the maximum of $I(X;Y)$ over all input distributions. For the binary symmetric channel, the uniform input is already optimal (we prove this later), so:

$$C=1-H(0.1)\approx 1-0.469=0.531\text{ bits per channel use}.$$

What this tells us. You cannot send more than about 0.531 bits reliably per use of this channel. If you try to send at 0.6 bits per use, the error probability will not go to zero, no matter how clever your coding scheme. If you send at any rate below 0.531, there exists a code that makes the error as small as you like, provided you use the channel enough times.

Now consider a binary erasure channel with erasure probability $e=0.3$. It either delivers the bit correctly (probability 0.7) or replaces it with an erasure symbol "?" (probability 0.3). When you see a "?", you know the bit was lost — you just do not know which one it was. The capacity is:

$$C=1-e=1-0.3=0.7\text{ bits per channel use}.$$

The erasure channel is more capacious than the symmetric channel at the same raw error rate (0.3 vs 0.1) because the erasure tells you where the error happened. Knowing where the damage is turns out to be extremely valuable.

Check your understanding Beginner

Formal definition Intermediate+

Let $\mathcal{X}$ and $\mathcal{Y}$ be finite sets (the input and output alphabets). A discrete memoryless channel (DMC) is a collection of conditional probabilities $p(y|x)$ for $x\in \mathcal{X}$, $y\in \mathcal{Y}$, satisfying:

1. $p(y|x)\ge 0$ for all $x,y$.
2. ${\sum }_{y\in \mathcal{Y}}p(y|x)=1$ for all $x$.

The matrix $[p(y|x)]$ with rows indexed by $\mathcal{X}$ and columns by $\mathcal{Y}$ is the transition matrix (or channel matrix). "Memoryless" means that the output at time $i$ depends only on the input at time $i$, not on past inputs or outputs. "Discrete" means both alphabets are finite.

Given an input distribution $p(x)$ on $\mathcal{X}$, the joint distribution on $\mathcal{X}\times \mathcal{Y}$ is $p(x,y)=p(x)p(y|x)$, and the output distribution is $p(y)={\sum }_{x}p(x)p(y|x)$.

Definition (Channel capacity). The capacity of a DMC is

$$C=\underset{p(x)}{\max }I(X;Y),$$

where the maximum is taken over all probability distributions on $\mathcal{X}$, and $I(X;Y)$ is the mutual information between the channel input and output:

$$I(X;Y)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x)p(y|x){\log }_2\frac{p(y|x)}{p(y)}.$$

The units of $C$ are bits per channel use (with base-2 logarithm).

Counterexamples to common slips

• Capacity is a property of the channel, not the input distribution. The capacity $C$ is the maximum of $I(X;Y)$ over all input distributions. A particular choice of $p(x)$ gives a mutual information $I(X;Y)\le C$, but the capacity itself does not depend on the input distribution — it is a single number attached to the channel.

• Mutual information depends on the input distribution; capacity does not. Students often confuse $I(X;Y)$ (a function of the input distribution) with $C$ (the maximum over all input distributions). The capacity is a property of the transition matrix $p(y|x)$ alone.

• "Memoryless" is a modelling assumption, not a theorem. Physical channels may have memory (intersymbol interference, fading correlations). The DMC model restricts to the memoryless case; channels with memory have a different (usually larger) capacity.

Key theorem with proof Intermediate+

Theorem (Binary symmetric channel capacity). For a BSC with crossover probability $p\in [0,1/2]$, the capacity is

$$C_{\text{BSC}}=1-H_b(p),$$

where $H_b(p)=-p{\log }_{2}p-(1-p){\log }_2(1-p)$ is the binary entropy function. The capacity is achieved by the uniform input distribution $p(0)=p(1)=1/2$.

Proof. For any input distribution $p(x)$ on $\{0,1\}$, the mutual information is $I(X;Y)=H(Y)-H(Y|X)$. By the chain rule for conditional entropy:

$$H(Y|X)=\sum _{x}p(x)H(Y|X=x).$$

For the BSC, $H(Y|X=x)=H_b(p)$ for both $x=0$ and $x=1$, since the channel treats both inputs symmetrically. Therefore $H(Y|X)=H_b(p)$ regardless of the input distribution.

Since $H(Y|X)$ is constant, maximising $I(X;Y)$ is equivalent to maximising $H(Y)$. The output $Y$ is a binary random variable, so $H(Y)\le 1$ with equality when $Y$ is uniform. The output is uniform when the input is uniform: if $p(0)=p(1)=1/2$, then $p(Y=0)=(1/2)(1-p)+(1/2)p=1/2$. Hence:

$$C_{\text{BSC}}=\underset{p(x)}{\max }I(X;Y)=1-H_b(p).\square$$

Theorem (Binary erasure channel capacity). For a BEC with erasure probability $e\in [0,1]$, the capacity is

$$C_{\text{BEC}}=1-e.$$

Proof. For the BEC with output alphabet $\{0,1,?\}$, we have $p(?|0)=p(?|1)=e$, $p(0|0)=p(1|1)=1-e$, and $p(1|0)=p(0|1)=0$. The key observation is that when the output is 0 or 1, the input is known exactly; uncertainty exists only when the output is "?".

For any input distribution $p(x)$ on $\{0,1\}$ with $p(0)=\alpha$:

$$I(X;Y)=H(Y)-H(Y|X).$$

The conditional entropy $H(Y|X)=H_b(e)$ (the binary entropy of the erasure event) for each input value, giving $H(Y|X)=H_b(e)$. The output distribution is $p(0)=\alpha (1-e)$, $p(1)=(1-\alpha )(1-e)$, $p(?)=e$.

Now $H(Y)=h(\alpha (1-e),(1-\alpha )(1-e),e)$ where $h$ is the entropy of a three-outcome distribution. The mutual information simplifies (by direct computation or by the chain rule $I(X;Y)=H(X)-H(X|Y)$) to:

$$I(X;Y)=H(X)-e\cdot H(X)=(1-e)H(X).$$

This is because $H(X|Y)=e\cdot H(X)$: when $Y=?$ (probability $e$), we learn nothing about $X$, so $H(X|Y=?)=H(X)$; when $Y=0$ or $Y=1$ (probability $1-e$), $X$ is determined, so $H(X|Y=y)=0$.

Maximising over $\alpha$: $H(X)\le 1$ with equality at $\alpha =1/2$, so:

$$C_{\text{BEC}}=(1-e)\cdot 1=1-e.\square$$

Bridge. The capacity formula $C={\max }_{p(x)}I(X;Y)$ builds toward the noisy-channel coding theorem in 46.03.02, which establishes that any rate below $C$ is achievable with arbitrarily small error probability over $n$ channel uses, giving $C$ its operational meaning. The decomposition $I(X;Y)=H(Y)-H(Y|X)$ appears again in Fano's inequality 46.03.03 as the foundational reason why the conditional entropy controls the minimum decoding error. The symmetry arguments used here for the BSC and BEC generalises to the class of symmetric channels treated below, where uniform inputs are always optimal. The convexity of $I(X;Y)$ in the transition matrix and its concavity in the input distribution is dual to the rate-distortion problem 46.04.01, putting these together reveals a shared optimisation structure across channel coding and source coding.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not define discrete memoryless channels, channel capacity, or the mutual information optimisation. The PMF type exists for discrete distributions, and there is machinery for conditional probability via PMF.bind, but no stochastic-matrix abstraction tying input and output alphabets. The capacity computation $C={\max }_{p(x)}I(X;Y)$ requires entropy on PMF (which is also absent), the chain rule for mutual information, and a concavity proof to justify that the maximum exists and is unique. The KKT conditions for the capacity-achieving distribution require Lagrange-multiplier machinery on the probability simplex. A Codex.InformationTheory.Channel module defining the DMC as a stochastic matrix on finite types and proving the BSC/BEC capacity formulas would be the load-bearing first step; this unit ships without it.

Advanced results Master

Symmetric channels and the uniform-input theorem

A DMC is symmetric if the set of output columns can be partitioned into subsets such that within each subset, the rows of the channel matrix are permutations of each other, and the column sums within each subset are equal. More precisely, the channel matrix $[p(y|x)]$ is symmetric if there exists a partition of the output alphabet $\mathcal{Y}={\mathcal{Y}}_1\cup \cdots \cup {\mathcal{Y}}_s$ such that for each $j$, the submatrix $[p(y|x){]}_{y\in {\mathcal{Y}}_j}$ has the property that every row is a permutation of every other row, and the column sums ${\sum }_{x}p(y|x)$ are equal for all $y\in {\mathcal{Y}}_j$.

A simpler sufficient condition is weak symmetry: every row of the channel matrix is a permutation of the first row, and the column sums are all equal. The BSC is weakly symmetric (rows are $(1-p,p)$ and $(p,1-p)$; column sums are both 1).

Theorem (Uniform input achieves capacity for symmetric channels). If the channel is weakly symmetric, then the uniform input distribution $p(x)=1/|\mathcal{X}|$ achieves capacity, and

$$C={\log }_2|\mathcal{Y}|-H(Y|X=x_0),$$

where $x_0$ is any fixed input and $H(Y|X=x_0)$ is the entropy of any row of the channel matrix.

Proof. Since each row is a permutation of every other row, $H(Y|X=x)$ is the same for all $x$. Call this value $H_{\text{row}}$. Then $H(Y|X)=H_{\text{row}}$ for any input distribution.

For uniform input, $p(x)=1/|\mathcal{X}|$, so $p(y)=(1/|\mathcal{X}|){\sum }_{x}p(y|x)$. The column-sum condition says ${\sum }_{x}p(y|x)$ is the same for all $y$, so $p(y)=1/|\mathcal{Y}|$, i.e., the output is uniform. Therefore $H(Y)={\log }_2|\mathcal{Y}|$ and:

$$I(X;Y)={\log }_2|\mathcal{Y}|-H_{\text{row}}.$$

This is the maximum because $H(Y)\le {\log }_2|\mathcal{Y}|$ for any input distribution, and $H(Y|X)=H_{\text{row}}$ is constant. So $I(X;Y)=H(Y)-H_{\text{row}}\le {\log }_2|\mathcal{Y}|-H_{\text{row}}$, with equality when $H(Y)$ is maximised, which the uniform input achieves. $\square$

KKT conditions for the capacity-achieving distribution

For a general (not necessarily symmetric) DMC, finding the capacity requires solving the concave programme ${\max }_{p(x)}I(X;Y)$. Since $I(X;Y)$ is concave in $p(x)$ and the constraint set (the probability simplex) is convex, the maximum is unique and characterised by the KKT (Karush-Kuhn-Tucker) conditions.

Theorem (KKT conditions for channel capacity). An input distribution $p^{\ast }(x)$ achieves capacity if and only if, for all $x\in \mathcal{X}$:

$$I(X=x;Y)=\sum _{y}p(y|x){\log }_2\frac{p(y|x)}{p^{\ast }(y)}=C\text{if }{p}^{\ast }(x)>0,$$ $$I(X=x;Y)=\sum _{y}p(y|x){\log }_2\frac{p(y|x)}{p^{\ast }(y)}\le C\text{if }{p}^{\ast }(x)=0,$$

where $p^{\ast }(y)={\sum }_x{p}^{\ast }(x)p(y|x)$ is the output distribution induced by $p^{\ast }$, and $C$ is the channel capacity.

Proof sketch. The Lagrangian for the constrained problem ${\max }_{p(x)}I(X;Y)$ subject to ${\sum }_{x}p(x)=1$, $p(x)\ge 0$ is:

$$\mathcal{L}=I(X;Y)+\lambda \left(1-\sum _{x}p(x)\right)+\sum _x{\mu }_{x}p(x),$$

where ${\mu }_x\ge 0$ and ${\mu }_{x}p(x)=0$ (complementary slackness). Setting $\partial \mathcal{L}/\partial p(x_0)=0$ for $p(x_0)>0$ gives:

$$\frac{\partial I}{\partial p(x_0)}=\lambda ,$$

and the partial derivative of $I(X;Y)$ with respect to $p(x_0)$ is $I(X=x_0;Y)-{\log }_{2}e$ (the "information density" at $x_0$, adjusted for the base). The condition $I(X=x_0;Y)=C$ for active inputs, and $\le C$ for inactive ones, follows. The constant $C=\lambda +{\log }_{2}e$ is the same for all active inputs. $\square$

The KKT conditions have a compelling interpretation: at the optimum, every input symbol that is used with positive probability conveys exactly $C$ bits of mutual information, and no unused symbol could convey more. The capacity is the "level water line" in the water-filling analogy: information flows equally through all active inputs.

Convexity and concavity properties

The mutual information $I(X;Y)$ has a dual convexity structure that underpins the computational theory:

Theorem (Convexity of $I$ in the channel). For a fixed input distribution $p(x)$, $I(X;Y)$ is convex in the transition probabilities $p(y|x)$.

Proof. Write $I(X;Y)=D(p(x,y)\Vert p(x)p(y))$. The KL divergence $D(q\Vert r)$ is convex in the pair $(q,r)$. The joint distribution $p(x,y)=p(x)p(y|x)$ is linear in $p(y|x)$, and the product distribution $p(x)p(y)$ is also linear in $p(y|x)$. Convexity of KL divergence in its arguments then gives convexity of $I$ in $p(y|x)$. $\square$

Theorem (Concavity of $I$ in the input distribution). For a fixed channel $p(y|x)$, $I(X;Y)$ is concave in the input distribution $p(x)$.

Proof. As shown in Exercise 5: $I(X;Y)=H(Y)-H(Y|X)$, where $H(Y)$ is concave in $p(y)$ (hence concave in $p(x)$ via the linear map $p(y)={\sum }_{x}p(x)p(y|x)$), and $H(Y|X)$ is linear in $p(x)$. The difference of a concave function and a linear function is concave. $\square$

The concavity in $p(x)$ guarantees that the capacity-achieving distribution is unique (strict concavity of $H$ on the interior of the simplex propagates). The convexity in $p(y|x)$ means that degrading a channel (replacing it with a convex combination of worse channels) cannot increase capacity, which is the foundation of the "less noisy" and "less capable" partial orders on channels.

Synthesis. The discrete memoryless channel is defined by a transition matrix $p(y|x)$ on finite alphabets; its capacity $C={\max }_{p(x)}I(X;Y)$ is a concave maximisation whose solution is characterised by the KKT conditions — every active input achieves the same mutual information $C$. For symmetric channels the uniform input achieves capacity by symmetry; for general channels the Arimoto-Blahut algorithm exploits concavity to converge to the optimum iteratively. The central insight is that capacity is the fundamental limit of reliable communication, builds toward the noisy-channel coding theorem 46.03.02 which gives it operational teeth, appears again in rate-distortion theory 46.04.01 as the mathematical dual, and generalises to continuous channels via differential entropy and to network settings via region-valued capacity. The convexity of $I$ in the channel and concavity in the input distribution is exactly the structure that makes the optimisation tractable.

Full proof set Master

Proposition (Data processing inequality for channels). Let $X\to Y\to Z$ be a Markov chain where $Y$ is the output of a DMC with input $X$ and $Z$ is obtained by processing $Y$ through another channel (i.e., $p(z|y,x)=p(z|y)$). Then $I(X;Z)\le I(X;Y)$.

Proof. By the chain rule for mutual information:

$$I(X;Y,Z)=I(X;Y)+I(X;Z|Y)=I(X;Z)+I(X;Y|Z).$$

Since $X\to Y\to Z$ forms a Markov chain, $X$ and $Z$ are conditionally independent given $Y$, so $I(X;Z|Y)=0$. Therefore:

$$I(X;Y)=I(X;Z)+I(X;Y|Z).$$

Since mutual information is non-negative, $I(X;Y|Z)\ge 0$, so $I(X;Y)\ge I(X;Z)$. $\square$

Proposition (Capacity of a cascade). Let $T_1$ and $T_2$ be two DMCs with transition matrices $p_1(y|x)$ and $p_2(z|y)$. The cascade channel $T_2\circ T_1$ has transition matrix $p(z|x)={\sum }_y{p}_1(y|x)p_2(z|y)$. Then:

$$C(T_2\circ T_1)\le \min (C(T_1),C(T_2)).$$

Proof. By the data processing inequality, any input distribution on $X$ gives $I(X;Z)\le I(X;Y)\le C(T_1)$. Since $I(X;Z)\le C(T_1)$ for all input distributions, $C(T_2\circ T_1)\le C(T_1)$.

For the other bound, note that $I(X;Z)\le H(Z)-H(Z|X)$. But also, $I(X;Z)=H(Z)-H(Z|X)\le H(Z)\le C(T_2)$, because $Z$ is the output of channel $T_2$ with some input distribution (induced by $Y$), and the maximum mutual information of $T_2$ over all input distributions is $C(T_2)$. More precisely, for any fixed input $p(x)$, $I(X;Z)\le I(Y;Z)$ is not guaranteed; instead, $I(X;Z)\le C(T_2)$ follows because the pair $(Y,Z)$ respects the channel $T_2$ and the data processing inequality gives $I(X;Z)\le \min (I(X;Y),I(Y;Z))\le \min (C(T_1),C(T_2))$. $\square$

Proposition (Capacity of a $q$-ary symmetric channel). A $q$-ary symmetric channel has input and output alphabets $\{1,\dots ,q\}$ and transition probabilities $p(y|x)=1-p$ if $y=x$ and $p(y|x)=p/(q-1)$ if $y\ne x$. Its capacity is:

$$C={\log }_{2}q-H\left(1-p,\frac{p}{q-1},\dots ,\frac{p}{q-1}\right).$$

Proof. The channel is symmetric: each row is a permutation of every other row (they all have one entry $1-p$ and $q-1$ entries $p/(q-1)$), and the column sums are all equal to $1$. By the uniform-input theorem for symmetric channels:

$$C={\log }_{2}q-H_{\text{row}},$$

where $H_{\text{row}}$ is the entropy of any row. Each row has distribution $(1-p,p/(q-1),\dots ,p/(q-1))$, giving:

$$H_{\text{row}}=-(1-p){\log }_2(1-p)-(q-1)\cdot \frac{p}{q-1}{\log }_2\frac{p}{q-1}=H_b(p)+p{\log }_2(q-1).\square$$

Connections Master

• 46.01.01 — Entropy and the chain rule are the algebraic engine behind mutual information $I(X;Y)=H(Y)-H(Y|X)$; the capacity optimisation depends on these quantities.
• 46.03.02 — Shannon's noisy-channel coding theorem gives capacity its operational meaning: any rate below $C$ is achievable with vanishing error probability over $n$ channel uses.
• 46.03.03 — Fano's inequality bounds the decoding error probability using $H(X|Y)$, which depends on the channel model and input distribution defined here.
• 46.04.01 — The rate-distortion function $R(D)={\min }_{p(\hat{x}|x):E[d]\le D}I(X;\hat{X})$ is the mathematical dual of channel capacity — one minimises, the other maximises mutual information.
• 37.01.01 — The probability-space formalism underpinning the DMC definition; the transition matrix is a conditional distribution on finite measurable spaces.
• 45.07.01 — PAC learning theory uses information-theoretic bounds (mutual information, KL divergence) to bound sample complexity; the channel model provides the noise structure.

Historical & philosophical context Master

Shannon introduced the discrete channel model and the capacity formula in Part III of his 1948 paper "A Mathematical Theory of Communication" (Bell System Technical Journal 27, 379-423 and 623-656). His central insight was that the maximum rate of reliable communication is determined entirely by the statistical properties of the channel, not by the semantics of the messages. The formula $C={\max }_{p(x)}I(X;Y)$ is a single number that separates the achievable from the impossible, and the noisy-channel coding theorem (which he proved in the same paper) establishes that the boundary is sharp.

The binary symmetric channel was Shannon's running example. He computed its capacity as $1-H(p)$ and noted that for $p=0.01$, the capacity is about 0.919 bits per use — meaning that roughly 92% of the channel's raw capacity survives 1% noise. The binary erasure channel was analysed by Elias (1955, unpublished course notes at MIT) and later by Gallager in his 1968 textbook Information Theory and Reliable Communication (Wiley). The erasure channel's capacity $C=1-e$ is remarkably simple because erasures are "honest" errors — they tell you where the information was lost.

The KKT conditions for the capacity-achieving distribution appear in the information-theory literature via several routes. Shannon himself noted (without proof) that the optimal input distribution equalises the "information density" across all active inputs. The formal KKT characterisation was given by Muroga (1953) and developed systematically by Gallager (1968, Chapter 4). The Arimoto-Blahut algorithm (independently discovered by Arimoto and Blahut in 1972) exploits concavity to compute capacity iteratively and is still the standard numerical method.

The symmetry classification of channels is due to Cover and Thomas (1991, 1st edition of Elements of Information Theory, Wiley), though the underlying observation — that uniform inputs are optimal when the channel treats all inputs equally — goes back to Shannon. The distinction between weak and strong symmetry is from Gallager (1968), who used it to simplify the computation of capacity for a broad class of practical channels.

Bibliography Master

@article{shannon1948,
  author  = {Shannon, C. E.},
  title   = {A Mathematical Theory of Communication},
  journal = {Bell System Technical Journal},
  volume  = {27},
  pages   = {379--423 and 623--656},
  year    = {1948},
}
@book{cover-thomas2006,
  author    = {Cover, T. M. and Thomas, J. A.},
  title     = {Elements of Information Theory},
  edition   = {2nd},
  publisher = {Wiley},
  year      = {2006},
}
@book{csiszar-korner2011,
  author    = {Csisz{\'a}r, I. and K{\"o}rner, J.},
  title     = {Information Theory: Coding Theorems for Discrete Memoryless Systems},
  edition   = {2nd},
  publisher = {Cambridge University Press},
  year      = {2011},
}
@book{gallager1968,
  author    = {Gallager, R. G.},
  title     = {Information Theory and Reliable Communication},
  publisher = {Wiley},
  year      = {1968},
}
@article{arimoto1972,
  author  = {Arimoto, S.},
  title   = {An Algorithm for Computing the Capacity of Arbitrary Discrete Memoryless Channels},
  journal = {IEEE Transactions on Information Theory},
  volume  = {18},
  number  = {1},
  pages   = {14--20},
  year    = {1972},
}
@article{blahut1972,
  author  = {Blahut, R. E.},
  title   = {Computation of Channel Capacity and Rate-Distortion Functions},
  journal = {IEEE Transactions on Information Theory},
  volume  = {18},
  number  = {4},
  pages   = {460--473},
  year    = {1972},
}