The Gaussian Channel: Capacity, Bandwidth, and Water-Filling

Intuition Beginner

Imagine sending a signal over a copper wire. You transmit a voltage, but the receiver does not see exactly what you sent. Thermal noise in the wire adds a random fluctuation. This is the Gaussian channel: the output equals the input plus random noise.

The strength of your signal relative to the noise determines how much information you can send. If you shout over a quiet room, the listener hears every word. If you whisper next to a jackhammer, almost nothing gets through. The ratio of signal power to noise power is called the signal-to-noise ratio (SNR).

Shannon discovered that the maximum rate at which you can send data reliably (with errors vanishing to zero) is $C=\frac{1}{2}{\log }_2(1+\mathrm{SNR})$ bits per channel use. Doubling the power does not double the capacity; it adds a constant. The logarithm means that pushing more power gives diminishing returns.

For a channel with bandwidth $W$ Hertz, the capacity becomes $C=W{\log }_2(1+P/(N{W}_0))$ bits per second, where $P$ is the signal power and $N_0$ is the noise spectral density. More bandwidth gives linearly more capacity, while more power gives only logarithmic improvement. This is why modern communication systems (WiFi, 5G) use wide bandwidths rather than extreme power.

When you have multiple independent channels with different noise levels, you allocate more power to the cleaner channels and less to the noisier ones. The optimal strategy, called water-filling, imagines pouring water (power) into a vessel whose bottom is shaped by the noise levels. The water level equalises across channels, and any channel where the noise exceeds the water level gets no power at all.

Visual Beginner

SNR (linear)	SNR (dB)	Capacity (bits/use)
1	0	0.50
3	4.8	1.00
10	10	1.73
100	20	3.46
1000	30	5.00
10000	40	6.64

Figure: water-filling for three parallel Gaussian channels with noise variances $N_1=1$, $N_2=3$, $N_3=6$, total power $P=12$. The water level $\mu =5$. Allocated powers: $P_1=4$, $P_2=2$, $P_3=0$. Channel 3 is too noisy to receive any power. The total capacity is $\frac{1}{2}{\log }_2(5)+\frac{1}{2}{\log }_2(5/3)+0=1.16+0.37=1.53$ bits per use.

Worked example Beginner

A radio channel has bandwidth 1 MHz, signal power 1 watt, and noise spectral density $N_0=10^{-6}$ watts/Hz. What is the channel capacity?

Step 1. Compute the total noise power in the band: $N=N_0\times W=10^{-6}\times 10^6=1$ watt.

Step 2. Compute the SNR: $P/N=1/1=1$ (0 dB).

Step 3. Apply the formula: $C=W{\log }_2(1+P/N)=10^6\times {\log }_2(2)=10^6$ bits per second = 1 Mbps.

If we double the power to 2 watts: $C=10^6\times {\log }_2(3)=10^6\times 1.585=1.585$ Mbps. Doubling power only gave a 58.5% increase in capacity. The logarithm is harsh.

Check your understanding Beginner

Formal definition Intermediate+

Definition (AWGN channel). The additive white Gaussian noise channel is defined by:

$$Y=X+Z,Z\sim \mathcal{N}(0,N),$$

where $X$ is the input with average power constraint $\mathbb{E}[X^2]\le P$, and $Z$ is independent of $X$.

Definition (Capacity of the AWGN channel). The capacity of the AWGN channel with power constraint $P$ and noise variance $N$ is:

$$C=\underset{p(x):\mathbb{E}[X^2]\le P}{\max }I(X;Y)=\frac{1}{2}{\log }_2\left(1+\frac{P}{N}\right)\text{ bits per channel use}.$$

Definition (Band-limited Gaussian channel). A band-limited channel with bandwidth $W$ Hz, power constraint $P$, and two-sided noise spectral density $N_0/2$ has capacity:

$$C=W{\log }_2\left(1+\frac{P}{N_{0}W}\right)\text{ bits per second}.$$

Definition (Parallel Gaussian channels and water-filling). Given $k$ independent Gaussian channels $Y_i=X_i+Z_i$ with $Z_i\sim \mathcal{N}(0,N_i)$ and total power constraint ${\sum }_{i=1}^k\mathbb{E}[X_i^2]\le P$, the capacity is:

$$C=\underset{\sum P_i\le P}{\max }\sum _{i=1}^k\frac{1}{2}{\log }_2\left(1+\frac{P_i}{N_i}\right),$$

with optimal power allocation given by water-filling: $P_i^{\ast }=(\mu -N_i{)}^{+}$ where $\mu$ is chosen to satisfy ${\sum }_i{P}_i^{\ast }=P$.

Key theorem with proof Intermediate+

Theorem (AWGN capacity). The capacity of the channel $Y=X+Z$, $Z\sim \mathcal{N}(0,N)$, with power constraint $\mathbb{E}[X^2]\le P$ is:

$$C=\frac{1}{2}{\log }_2\left(1+\frac{P}{N}\right).$$

The maximum is achieved by Gaussian input $X^{\ast }\sim \mathcal{N}(0,P)$.

Proof. The mutual information is $I(X;Y)=h(Y)-h(Y|X)=h(Y)-h(Z)$. Since $h(Z)=\frac{1}{2}{\log }_2(2\pi eN)$ is fixed, maximising $I(X;Y)$ is equivalent to maximising $h(Y)$.

Given the power constraint $\mathbb{E}[Y^2]=\mathbb{E}[(X+Z{)}^2]=\mathbb{E}[X^2]+N\le P+N$, the differential entropy $h(Y)$ is maximised by a Gaussian distribution (the maximum-entropy property), giving $h(Y)\le \frac{1}{2}{\log }_2(2\pi e(P+N))$.

Equality holds when $Y\sim \mathcal{N}(0,P+N)$, which occurs when $X\sim \mathcal{N}(0,P)$ (since the sum of independent Gaussians is Gaussian). Therefore:

$$C=h(Y^{\ast })-h(Z)=\frac{1}{2}{\log }_2(2\pi e(P+N))-\frac{1}{2}{\log }_2(2\pi eN)=\frac{1}{2}{\log }_2\left(\frac{P+N}{N}\right)=\frac{1}{2}{\log }_2\left(1+\frac{P}{N}\right).\square$$

Theorem (Water-filling for parallel channels). The optimal power allocation for $k$ independent Gaussian channels with noise variances $N_1\le N_2\le \cdots \le N_k$ and total power $P$ is:

$$P_i^{\ast }=(\mu -N_i{)}^{+}=\max (0,\mu -N_i),$$

where the water level $\mu$ satisfies ${\sum }_{i=1}^k(\mu -N_i{)}^{+}=P$.

Proof. The Lagrangian for the constrained optimisation is:

$$\mathcal{L}=\sum _i\frac{1}{2}{\log }_2\left(1+\frac{P_i}{N_i}\right)-\lambda \left(\sum _i{P}_i-P\right).$$

Taking the derivative with respect to $P_i$:

$$\frac{\partial \mathcal{L}}{\partial P_i}=\frac{1}{2\ln 2}\cdot \frac{1}{N_i+P_i}-\lambda =0⟹P_i=\frac{1}{2\lambda \ln 2}-N_i=\mu -N_i,$$

where $\mu =1/(2\lambda \ln 2)$. The constraint $P_i\ge 0$ gives $P_i=(\mu -N_i{)}^{+}$. The KKT conditions 44.02.01 guarantee that this is optimal. $\square$

Bridge. The Gaussian channel capacity builds toward the capacity-achieving codes in 46.08.04 where LDPC, turbo, and polar codes are evaluated by their gap to this formula; this appears again in 46.02.05 as the source-coding dual where the same logarithmic structure governs the rate-distortion function; the foundational reason Gaussian input is optimal is the maximum-entropy property of the Gaussian distribution 46.01.04; the central insight is that water-filling equalises the marginal benefit of power across all channels, and putting these together gives the information-theoretic analogue of economic resource allocation under a budget constraint.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib has no formalisation of the Gaussian channel model, the capacity formula, or the water-filling optimisation. The proof that Gaussian input maximises mutual information under a power constraint requires the maximum-entropy property of Gaussians (which Mathlib has in a limited form for real-valued distributions) and the entropy power inequality. The parallel-channel decomposition and the KKT-based water-filling proof require constrained optimisation over real-valued functions, which Mathlib's optimisation library does not support in the needed generality. The band-limited channel derivation needs Fourier analysis and the sampling theorem. This unit ships without a lean_module.

Advanced results Master

The sphere-packing interpretation

The capacity formula has a beautiful geometric interpretation. In $n$ dimensions, the received vector $Y^n$ lies near the transmitted codeword $X^n$ in a sphere of radius $\sqrt{nN}$ (by the law of large numbers, the noise power concentrates). The transmitted codewords must be centres of non-overlapping spheres of radius $\sqrt{nN}$ in a sphere of radius $\sqrt{n(P+N)}$ (the total power constraint). The number of non-overlapping spheres that fit is approximately the ratio of volumes:

$$M\approx \frac{\text{Vol}(n,\sqrt{n(P+N)})}{\text{Vol}(n,\sqrt{nN})}={\left(\frac{P+N}{N}\right)}^{n/2},$$

giving rate $R=\frac{1}{n}{\log }_{2}M\approx \frac{1}{2}{\log }_2(1+P/N)=C$. This is not a proof (the spheres overlap at their boundaries), but it captures the essential geometry.

The Gaussian channel with coloured noise

When the noise is not white but has a non-flat power spectral density $S_Z(f)$, the capacity is found by water-filling in the frequency domain:

$$C={\int }_0^W\max \left(0,\frac{1}{2}{\log }_2\frac{\mu }{S_Z(f)}\right)df,$$

where $\mu$ is the water level chosen to satisfy ${\int }_0^W\max (0,\mu -S_Z(f))df=P$. This generalises the parallel-channel result to a continuum of sub-channels, one for each frequency. The optimal input power spectral density is $S_X(f)=(\mu -S_Z(f){)}^{+}$.

This result connects to the rate-distortion water-filling 46.02.05 by the observation that the same variational problem arises in both contexts: allocate a fixed resource (power in channel coding, rate in source coding) across parallel sub-problems with varying quality (noise levels in channels, variances in sources).

The information bottleneck and the Gaussian case

The information bottleneck method asks: given $X$ and a relevant variable $Y$, find a compressed representation $\hat{X}$ that preserves information about $Y$ while minimising the description length of $\hat{X}$. For Gaussian $X,Y$, the optimal $\hat{X}$ is a Gaussian test channel that admits a water-filling solution analogous to the rate-distortion case. The information bottleneck curve in the Gaussian case is parametrically identical to the Gaussian rate-distortion curve, establishing a deep connection between compression, prediction, and channel capacity.

Fading channels

When the channel gain $h$ varies randomly (as in wireless communication), the channel becomes $Y=hX+Z$ where $h$ is a random fading coefficient. The ergodic capacity (averaged over fading realisations known at the receiver) is:

$$C_{\text{ergodic}}={\mathbb{E}}_h\left[\frac{1}{2}{\log }_2\left(1+\frac{|h{|}^{2}P}{N}\right)\right].$$

For Rayleigh fading ($h$ complex Gaussian), this integral evaluates to $C_{\text{ergodic}}=\frac{1}{\ln 2}{e}^{N/P}{E}_1(N/P)$ where $E_1$ is the exponential integral. The outage capacity (the rate achievable with probability $1-p_{\text{out}}$) is determined by the tail of the fading distribution.

Synthesis. The Gaussian channel capacity $C=\frac{1}{2}\log (1+P/N)$ is the central formula of continuous-alphabet information theory; it builds toward the code constructions in 46.08.04 that approach this limit; the central insight is that Gaussian input achieves capacity because of the maximum-entropy property, and this generalises to the water-filling principle for parallel channels where power is allocated to equalise marginal benefit; putting these together, the Gaussian capacity connects source coding 46.02.05, channel coding, and the Blahut-Arimoto computation 46.04.04 through the shared logarithmic structure of mutual information; the foundational reason water-filling is optimal is the KKT stationarity condition that equalises the derivative of the objective across all active channels.

Full proof set Master

Proposition (High-SNR approximation). For the AWGN channel, when $P/N\gg 1$, the capacity satisfies $C\approx \frac{1}{2}{\log }_2(P/N)$, and each doubling of power adds exactly $\frac{1}{2}$ bit per channel use.

Proof. $C=\frac{1}{2}{\log }_2(1+P/N)\approx \frac{1}{2}{\log }_2(P/N)$ when $P/N\gg 1$. If power is doubled from $P$ to $2P$:

$$C_{\text{new}}-C_{\text{old}}=\frac{1}{2}{\log }_2\frac{1+2P/N}{1+P/N}=\frac{1}{2}{\log }_2\left(1+\frac{P/N}{1+P/N}\right)\approx \frac{1}{2}$$

when $P/N\gg 1$, since the fraction inside the log approaches 1. More precisely, $C_{2P}-C_P=\frac{1}{2}{\log }_2(2+N/P)-\frac{1}{2}{\log }_2(1+N/P)\to \frac{1}{2}{\log }_{2}2=\frac{1}{2}$ as $P/N\to \infty$. $\square$

Proposition (Entropy power inequality implies Gaussian optimality). If $X$ and $Z$ are independent with $\mathbb{E}[X^2]\le P$ and $Z\sim \mathcal{N}(0,N)$, then $h(X+Z)\le \frac{1}{2}\log (2\pi e(P+N))$, with equality iff $X\sim \mathcal{N}(0,P)$.

Proof. By the entropy power inequality 46.01.04:

$$2^{2h(X+Z)}\ge 2^{2h(X)}+2^{2h(Z)}.$$

Since $h(X)\le \frac{1}{2}\log (2\pi eP)$ (maximum entropy under variance $P$) and $h(Z)=\frac{1}{2}\log (2\pi eN)$:

$$2^{2h(X+Z)}\ge 2\pi eP+2\pi eN=2\pi e(P+N).$$

Therefore $h(X+Z)\ge \frac{1}{2}\log (2\pi e(P+N))$. Wait — this gives a lower bound. The upper bound follows from the maximum-entropy property: since $\text{Var}(X+Z)\le P+N$, the entropy $h(X+Z)\le \frac{1}{2}\log (2\pi e(P+N))$. Equality iff $X+Z$ is Gaussian with variance $P+N$, which occurs iff $X\sim \mathcal{N}(0,P)$. $\square$

Connections Master

• 46.03.01 — The discrete channel capacity is generalised to continuous alphabets by the Gaussian channel; the same max-mutual-information principle governs both.
• 46.01.04 — Differential entropy, the maximum-entropy property of Gaussians, and the entropy power inequality are the tools used to prove the capacity formula.
• 46.02.05 — The Gaussian rate-distortion function $R(D)=\frac{1}{2}\log ({\sigma }^2/D)$ is the source-coding dual of the channel capacity; water-filling appears in both.
• 44.02.01 — The KKT conditions and Lagrangian duality provide the optimisation framework for water-filling.
• 46.08.04 — LDPC, turbo, and polar codes are designed to approach the Gaussian channel capacity; the gap to capacity is the figure of merit.

Historical & philosophical context Master

Shannon derived the band-limited Gaussian channel capacity in his 1949 paper "Communication in the Presence of Noise" (Proceedings of the IRE 37(1):10-21). The derivation used the sampling theorem (each band-limited signal of bandwidth $W$ can be represented by $2W$ samples per second) and the maximum-entropy argument. The paper also introduced the geometric sphere-packing interpretation of the capacity formula.

The water-filling solution for parallel Gaussian channels was derived independently by several researchers in the 1960s. The earliest complete treatment is in Gallager's 1968 textbook Information Theory and Reliable Communication (Wiley, Chapter 8), which also contains the rigorous proof of achievability via random coding and the converse via the entropy power inequality.

The extension to fading channels has been central to wireless communication theory since the 1990s. The ergodic capacity formula for Rayleigh fading channels was derived by Ozarow, Shamai, and Wyner in 1994 ("Information Theoretic Considerations for Cellular Mobile Radio," IEEE Trans. IT 40(3):747-763). The outage capacity formulation is due to Cover and its collaborators in the early 1990s.

The connection between water-filling in channel coding and reverse water-filling in rate-distortion (the information-theoretic duality) was recognised by Shannon in his 1959 paper and has been developed extensively by Berger, Dubois, and others. The duality is not merely formal: it reflects a deep symmetry between the source and channel coding problems that is made explicit by the lossy source-channel separation theorem.

Bibliography Master

@article{shannon1949,
  author  = {Shannon, C. E.},
  title   = {Communication in the Presence of Noise},
  journal = {Proceedings of the IRE},
  volume  = {37},
  number  = {1},
  pages   = {10--21},
  year    = {1949},
}
@book{gallager1968,
  author    = {Gallager, R. G.},
  title     = {Information Theory and Reliable Communication},
  publisher = {Wiley},
  year      = {1968},
}
@book{cover-thomas2006,
  author    = {Cover, T. M. and Thomas, J. A.},
  title     = {Elements of Information Theory},
  edition   = {2nd},
  publisher = {Wiley},
  year      = {2006},
}
@article{ozarow-shamai-wyner1994,
  author  = {Ozarow, L. H. and Shamai, S. and Wyner, A. D.},
  title   = {Information Theoretic Considerations for Cellular Mobile Radio},
  journal = {IEEE Transactions on Information Theory},
  volume  = {40},
  number  = {3},
  pages   = {747--763},
  year    = {1994},
}
@book{pinsker1964,
  author    = {Pinsker, M. S.},
  title     = {Information and Information Stability of Random Variables and Processes},
  publisher = {Holden-Day},
  year      = {1964},
}