Gambling, the Kelly Criterion, and the Doubling Rate: Entropy as Growth

Intuition Beginner

Imagine a horse race with $m$ horses. Horse $x$ wins with probability $p(x)$. The track offers odds $o(x)$-to-1: bet $1 on horse $x$, and if it wins you get back $o(x)$; if it loses, you lose your dollar.

You have $1 to spread across all horses. You put fraction $b(x)$ on horse $x$, with all fractions summing to 1. If horse $x$ wins, your new wealth is $b(x)\cdot o(x)$. You just multiplied your money by the factor $b(x)\cdot o(x)$.

Now repeat the race over and over, reinvesting everything each time. After $n$ races your total wealth factor is the product of $n$ random multipliers. Products of random numbers grow or shrink exponentially, and the exponential rate is what matters in the long run. That rate is called the doubling rate $W$: the number of times your wealth doubles per race, on average. Positive $W$ means you get richer exponentially. Negative $W$ means you go broke.

The first surprise: the best strategy is proportional betting, $b(x)=p(x)$. Bet on each horse in proportion to its probability of winning. This is optimal regardless of the odds, though the value of the strategy depends on the odds ^{[source pending]}.

Under "fair odds," where $o(x)=1/p(x)$, proportional betting yields a doubling rate of exactly zero: your wealth stays flat on average. The track is fair, so no strategy can extract a positive growth rate. Under "super-fair" odds, where the track pays out more than the fair level for some horses, proportional betting exploits the edge and yields a positive $W$.

The second surprise comes from side information. Suppose you receive a tip $Y$ correlated with the outcome $X$ — say, a weather report that shifts the probabilities. You can now bet conditionally: choose $b(x|y)$ depending on the signal. The increase in doubling rate from using the tip equals exactly the mutual information $I(X;Y)$ ^{[source pending]}.

If the tip tells you the outcome perfectly, $I(X;Y)=H(X)$ and your doubling rate jumps by the full entropy. If the tip is noise, $I(X;Y)=0$ and it adds nothing. Mutual information is not just an abstract measure of statistical dependence — it is the exact monetary value of a tip, measured in growth-rate units.

Why does proportional betting win? If you put too little on a likely horse and too much on an unlikely one, you lose a little on most races (when the likely horse wins) and gain a lot on rare races (when the longshot wins). The arithmetic works against you because you are multiplying your wealth each round. A single zero wipes you out, but even a near-zero outcome sets you back enormously.

Proportional betting avoids near-zero outcomes: you always have something on the winning horse, and the fraction scales with how likely that horse was to begin with. The log of your wealth is what matters, not the wealth itself, because logs turn products into sums and the law of large numbers kicks in.

This framework extends beyond horse racing. Any repeated favorable bet — stock markets, venture capital, insurance underwriting — has the same structure. You have a bankroll, you allocate it across outcomes, and you reinvest. The Kelly criterion tells you how to allocate optimally: in proportion to your beliefs about the probabilities. The result is that your wealth grows at the fastest possible exponential rate, and that rate is governed by information-theoretic quantities.

The gambling framework gives information theory an economic interpretation. Entropy measures the uncertainty in the race. Mutual information measures how much a correlated signal reduces that uncertainty and translates it into extra growth. These are theorems, not analogies.

Theory Intermediate

Formal definitions: the horse race model and the doubling rate

Fix a finite alphabet $\mathcal{X}=\{1,2,\dots ,m\}$. The race outcome is a random variable $X\sim p(x)$, a probability mass function on $\mathcal{X}$. The track offers odds $o(x)>0$ for each horse $x\in \mathcal{X}$: a bet of $1 on horse $x$ returns $o(x)$ifhorse$x$ wins and $0 otherwise.

A betting strategy is a vector $\mathbf{b}=(b(1),b(2),\dots ,b(m))$ with $b(x)\ge 0$ for all $x$ and ${\sum }_{x}b(x)=1$. The gambler invests their entire bankroll each round, distributing it across the $m$ horses.

If horse $x$ wins, the gambler's wealth is multiplied by $b(x)\cdot o(x)$. After $n$ independent races with outcomes $X_1,X_2,\dots ,X_n$, the total wealth factor is:

$$S_n=\prod _{i=1}^{n}b(X_i)\cdot o(X_i)$$

Taking logarithms and dividing by $n$:

$$\frac{1}{n}\log S_n=\frac{1}{n}\sum _{i=1}^n\log (b(X_i)\cdot o(X_i))$$

By the strong law of large numbers (prerequisite 37.02.01), the sample average converges almost surely to its expectation:

$$\frac{1}{n}\log S_n⟶W(\mathbf{b},p)\text{a.s. as }n\to \infty$$

where the doubling rate is defined as:

$$W(\mathbf{b},p)={\mathbb{E}}_{X\sim p}\left[\log \right(b(X)\cdot o(X)\left)\right]=\sum _{x}p(x)\log (b(x)\cdot o(x))$$

All logarithms are natural unless otherwise noted. The doubling rate is measured in nats per race. Wealth behaves as $S_n\approx e^{nW}$ in the long run: positive $W$ yields exponential growth, negative $W$ yields exponential decay ^{[source pending]}.

The doubling rate decomposes as:

$$W(\mathbf{b},p)=\underset{\text{depends on strategy}}{\underbrace{\sum _{x}p(x)\log b(x)}}+\underset{\text{depends on odds}}{\underbrace{\sum _{x}p(x)\log o(x)}}$$

The second term is a constant with respect to $\mathbf{b}$. The first term is where the gambler's choice matters. By the strict concavity of $\log$, this term satisfies ${\sum }_{x}p(x)\log b(x)\le \log {\sum }_{x}p(x)b(x)=0$, with equality if and only if $b=p$. The strategy-dependent term is always non-positive, and it attains its maximum value of zero precisely at proportional betting. This observation is the core of the optimality proof.

2. Optimal betting: the Kelly strategy

Theorem (Kelly optimality). For any odds $o(x)>0$, the optimal doubling rate is achieved by proportional betting $b^{\ast }(x)=p(x)$:

$$W^{\ast }(p)=\underset{\mathbf{b}}{\max }W(\mathbf{b},p)=\sum _{x}p(x)\log p(x)\cdot o(x)$$

Proof. Maximize ${\sum }_{x}p(x)\log b(x)$ subject to ${\sum }_{x}b(x)=1$ and $b(x)\ge 0$. (The $\sum p(x)\log o(x)$ term is constant in $\mathbf{b}$ and does not affect the optimization.)

Method 1: Lagrange multipliers. The Lagrangian is $\mathcal{L}={\sum }_{x}p(x)\log b(x)+\lambda (1-{\sum }_{x}b(x))$. Setting $\partial \mathcal{L}/\partial b(x)=p(x)/b(x)-\lambda =0$ gives $b(x)=p(x)/\lambda$. Summing: $1=(1/\lambda ){\sum }_{x}p(x)=1/\lambda$, so $\lambda =1$ and $b^{\ast }(x)=p(x)$. The objective is strictly concave in $\mathbf{b}$, so the stationary point is the unique global maximum.

Method 2: KL divergence. For any strategy $\mathbf{b}$:

$$W^{\ast }-W(\mathbf{b},p)=\sum _{x}p(x)\log \frac{p(x)}{b(x)}=D(p\Vert \mathbf{b})\ge 0$$

where $D(p\Vert b)$ is the KL divergence between the true distribution $p$ and the betting strategy $\mathbf{b}$ (viewed as a distribution on $\mathcal{X}$). Equality holds if and only if $b(x)=p(x)$ for all $x$. The gap between optimal and any other strategy is exactly the KL divergence $D(p\Vert b)$.

Substituting $b^{\ast }=p$ gives:

$$W^{\ast }(p)=\sum _{x}p(x)\log p(x)\cdot o(x)=-H(X)+\sum _{x}p(x)\log o(x)$$

where $H(X)=-{\sum }_{x}p(x)\log p(x)$ is the Shannon entropy ^{[source pending]}.

Corollary (Fair odds). When the odds are fair, $o(x)=1/p(x)$:

$$W^{\ast }(p)=\sum _{x}p(x)\log \frac{p(x)}{p(x)}=\sum _{x}p(x)\log 1=0$$

Under fair odds, no strategy — including proportional betting — can extract a positive growth rate. The track is efficient: the expected return on any bet equals the amount wagered. Proportional betting is still optimal, but the optimum is zero.

Corollary (Super-fair odds). When $o(x)=c/p(x)$ for some constant $c>1$ (the track overpays every horse by the same factor):

$$W^{\ast }(p)=\sum _{x}p(x)\log \frac{c\cdot p(x)}{p(x)}=\log c$$

The growth rate depends only on the overpayment factor $c$, not on the distribution $p$. This makes sense: every horse is overpaid by the same multiplicative factor, so the edge is uniform regardless of which horse wins.

Corollary (General super-fair). More generally, $W^{\ast }(p)>0$ if and only if ${\sum }_{x}p(x)\log o(x)>H(X)$, equivalently ${\sum }_{x}p(x)\log (p(x)o(x))>0$. A positive growth rate requires that the odds sufficiently exceed the fair-odds level $1/p(x)$ on average.

3. Side information and the mutual-information gain

Suppose the gambler observes a side signal $Y$, jointly distributed with $X$ according to $p(x,y)$. Given $Y=y$, the gambler adopts a conditional betting strategy $b(x|y)$. The conditional doubling rate is:

$$W(\mathbf{b}(\cdot |Y),p)=\sum _{y}p(y)\sum _{x}p(x|y)\log b(x|y)\cdot o(x)$$

Maximizing independently for each $y$ yields the conditional optimal strategy $b^{\ast }(x|y)=p(x|y)$, by the same argument as the unconditional case.

Theorem (Value of side information). The increase in doubling rate from using side information $Y$ equals the mutual information:

$$\Delta W=W_{\text{cond}}^{\ast }-W_{\text{uncond}}^{\ast }=I(X;Y)$$

where $W_{\text{uncond}}^{\ast }={\sum }_{x}p(x)\log p(x)o(x)$ and $W_{\text{cond}}^{\ast }={\sum }_{x,y}p(x,y)\log p(x|y)\cdot o(x)$.

Proof.

$$\Delta W=\sum _{x,y}p(x,y)\log p(x|y)o(x)-\sum _{x}p(x)\log p(x)o(x)$$ $$=\sum _{x,y}p(x,y)\log \frac{p(x|y)\cdot o(x)}{p(x)\cdot o(x)}=\sum _{x,y}p(x,y)\log \frac{p(x|y)}{p(x)}$$ $$=\sum _{x,y}p(x,y)\log p(x|y)-\sum _{x,y}p(x,y)\log p(x)=-H(X|Y)+H(X)=I(X;Y)$$

The odds $o(x)$ cancel because they depend only on $x$, not on $y$. The result holds for any odds structure ^{[source pending]}.

Interpretation. This is the second major operational meaning of mutual information (the first being channel capacity, unit 46.03.01). The mutual information $I(X;Y)$ is the increase in exponential growth rate of wealth attributable to the signal $Y$. For fair odds, the unconditional doubling rate is zero and the conditional doubling rate is $I(X;Y)$: the gambler's wealth grows at a rate determined by how informative $Y$ is about $X$. A perfect signal ($I=H(X)$) gives the gambler full knowledge and the maximum possible edge. A useless signal ($I=0$) adds no growth.

Key result: optimal betting and worked examples

Worked example. A binary horse race: $X\in \{1,2\}$ with $p(1)=p(2)=1/2$, odds $o(1)=3$, $o(2)=3/2$.

Without side information. The optimal strategy is $b^{\ast }(1)=b^{\ast }(2)=1/2$ (proportional betting). The doubling rate:

$$W^{\ast }=\frac{1}{2}\log \frac{1}{2}\cdot 3+\frac{1}{2}\log \frac{1}{2}\cdot \frac{3}{2}=\frac{1}{2}\log \frac{3}{2}+\frac{1}{2}\log \frac{3}{4}=\frac{1}{2}\log \frac{9}{8}\approx 0.0589\text{ nats}$$

This is positive: the track overpays relative to fair odds on average. (Fair odds would be $o(1)=2$, $o(2)=2$; here $o(1)=3>2$ and $o(2)=3/2<2$, but the excess on horse 1 more than compensates.)

With side information. Suppose $Y\in \{0,1\}$ satisfies $\mathrm{Pr}(Y=X)=0.8$. Then $p(1|0)=0.8$, $p(2|0)=0.2$ and symmetrically for $Y=1$. The mutual information:

$$I(X;Y)=H(X)-H(X|Y)=1-H_b(0.2)=1-(-0.2\log 0.2-0.8\log 0.8)\approx 1-0.5004=0.4996\text{ bits}$$

Converting to nats: $I(X;Y)\approx 0.3466$ nats. The conditional optimal doubling rate:

$$W_{\text{cond}}^{\ast }=W_{\text{uncond}}^{\ast }+I(X;Y)\approx 0.0589+0.3466=0.4055\text{ nats}$$

Side information nearly septuples the growth rate. The conditional gambler exploits the correlation by betting $b^{\ast }(1|0)=0.8$, $b^{\ast }(2|0)=0.2$, concentrating wealth on the likely winner.

4. The gambling-compression duality

The horse race has a precise duality with lossless source coding (unit 46.02.02). Given a uniquely decodable code with codeword lengths $\ell (x)$, the Kraft inequality guarantees ${\sum }_x{2}^{-\ell (x)}\le 1$. Define a "gambling strategy" $b(x)=2^{-\ell (x)}$ (renormalized to sum to 1 if necessary) and odds $o(x)=2^{\ell (x)}$. Then $b(x)\cdot o(x)=1$ for all $x$, and the doubling rate is $W={\sum }_{x}p(x)\log 1=0$.

The duality becomes meaningful with a different mapping. Let the code lengths be $\ell (x)=\lceil -{\log }_{2}p(x)\rceil$. The associated strategy bets $b(x)=2^{-\ell (x)}$ and the odds are determined by the channel or track. The expected code length ${\sum }_{x}p(x)\ell (x)$ is minimized by the same distribution $p$ that maximizes the doubling rate. Both problems optimize over the simplex: the Huffman code minimizes $\sum p(x)\ell (x)$ subject to Kraft, while Kelly maximizes $\sum p(x)\log b(x)o(x)$ subject to $\sum b(x)=1$ ^{[source pending]}.

This duality is not merely structural. Cover and Thomas show that the optimal doubling rate for the horse race with fair odds can be expressed as $W^{\ast }=H(X)-H(X)=0$ when the gambler uses the code-length strategy. The surplus growth under super-fair odds corresponds to the redundancy of a suboptimal code. Gambling and compression are two faces of the same optimization problem over the probability simplex.

A deeper version of the duality emerges from the Kraft inequality. Given a uniquely decodable code with lengths $\ell (x)$, define a gambling strategy $b(x)=2^{-\ell (x)}/Z$ where $Z={\sum }_x{2}^{-\ell (x)}\le 1$. The corresponding odds are $o(x)=2^{\ell (x)}\cdot Z$. Then the doubling rate is:

$$W=\sum _{x}p(x)\log \frac{2^{-\ell (x)}}{Z}\cdot 2^{\ell (x)}\cdot Z=\sum _{x}p(x)\log 1=0$$

Every code induces a fair-gambling game with zero doubling rate. The code length $\ell (x)$ plays the role of the log-odds. The optimality of the Shannon code ($\ell (x)=-\log p(x)$) in compression corresponds to the optimality of proportional betting ($b(x)=p(x)$) in gambling: both minimize the KL divergence to the true distribution.

This duality has computational consequences. The Blahut-Arimoto algorithm (unit 46.04.04) iterates between optimizing the input distribution and the test channel to compute channel capacity. An analogous algorithm computes the log-optimal portfolio by iterating between the portfolio weights and the dual variables. The convergence guarantees are the same because the underlying convex optimization structure is identical.

5. The Kelly criterion for continuous outcomes

The horse race extends to the stock market. Consider $m$ assets with price relatives $\mathbf{X}=(X_1,\dots ,X_m)$ drawn i.i.d. from a distribution $F$ on ${\mathbb{R}}_{+}^m$. The component $X_i$ is the ratio of closing to opening price for asset $i$.

A portfolio is $\mathbf{b}=(b_1,\dots ,b_m)$ with $b_i\ge 0$, ${\sum }_i{b}_i=1$. The one-period wealth factor is ${\mathbf{b}}^T\mathbf{X}={\sum }_i{b}_i{X}_i$. The doubling rate is:

$$W(\mathbf{b},F)={\mathbb{E}}_F[\log ({\mathbf{b}}^T\mathbf{X})]$$

The log-optimal portfolio $b^{\ast }$ maximizes $W(\mathbf{b},F)$. Since $\log$ is concave and ${\mathbf{b}}^T\mathbf{X}$ is linear in $\mathbf{b}$, the composition $W$ is concave in $\mathbf{b}$. The maximum exists and first-order conditions are sufficient.

Theorem (Log-optimal portfolio conditions). The portfolio $b^{\ast }$ is log-optimal if and only if:

$$\mathbb{E}\left[\frac{X_i}{{\mathbf{b}}^{\text{astT}}\mathbf{X}}\right]=1\text{for all }i\text{ with }{b}_i^{\ast }>0$$ $$\mathbb{E}\left[\frac{X_i}{{\mathbf{b}}^{\text{astT}}\mathbf{X}}\right]\le 1\text{for all }i\text{ with }{b}_i^{\ast }=0$$

The condition states that at the optimum, reallocating wealth from any active asset to any other active asset does not improve expected log-return. This is a stationarity condition for the concave program ^{[source pending]}.

The horse race is a special case: define $\mathbf{X}$ so that $X_x=o(x)$ when horse $x$ wins and $X_{x^{\prime }}=0$ for $x^{\prime }\ne x$. Then ${\mathbf{b}}^T\mathbf{X}=b(x)o(x)$ when horse $x$ wins, and $W={\sum }_{x}p(x)\log b(x)o(x)$.

Example (Two-asset market). Two assets with price relatives $X_1$ and $X_2$ jointly distributed as: $(X_1,X_2)=(2,1/2)$ with probability $1/2$ and $(1/2,2)$ with probability $1/2$. By symmetry, the log-optimal portfolio is $b^{\ast }=(1/2,1/2)$. The doubling rate:

$$W^{\ast }=\frac{1}{2}\log (\frac{1}{2}\cdot 2+\frac{1}{2}\cdot \frac{1}{2})+\frac{1}{2}\log (\frac{1}{2}\cdot \frac{1}{2}+\frac{1}{2}\cdot 2)=\log (5/4)\approx 0.223\text{ nats}$$

A portfolio all in one asset gives $W=\frac{1}{2}\log 2+\frac{1}{2}\log (1/2)=0$. Diversification improves the growth rate from zero to $\log (5/4)$.

A less obvious strategy, $b=(3/4,1/4)$, yields:

$$W=\frac{1}{2}\log (\frac{3}{4}\cdot 2+\frac{1}{4}\cdot \frac{1}{2})+\frac{1}{2}\log (\frac{3}{4}\cdot \frac{1}{2}+\frac{1}{4}\cdot 2)=\frac{1}{2}\log (13/8)+\frac{1}{2}\log (7/8)$$ $$=\frac{1}{2}\log (91/64)\approx \frac{1}{2}\log 1.422\approx 0.175\text{ nats}$$

This is lower than $W^{\ast }=0.223$. The symmetric portfolio dominates the asymmetric one, consistent with the symmetry of the return distribution. The log-optimal portfolio matches the symmetry of the market.

Cover's universal portfolio achieves the optimal doubling rate without knowing $F$. At time $t$, the universal investor allocates wealth proportional to the accumulated wealth of each constant-rebalanced portfolio under the observed returns. Formally, let $\mathbf{b}$ be a constant-rebalanced portfolio and define $S_n(\mathbf{b})={\prod }_{i=1}^n{\mathbf{b}}^T{\mathbf{X}}_i$. The universal portfolio at time $n+1$ invests:

$${\mathbf{b}}_{n+1}^{\text{univ}}=\frac{{\int }_{\mathbf{b}}\mathbf{b}\cdot S_n(\mathbf{b})d\mathbf{b}}{{\int }_{\mathbf{b}}{S}_n(\mathbf{b})d\mathbf{b}}$$

where the integrals are over the $(m-1)$-dimensional simplex. This is a Bayesian mixture with uniform prior on the simplex, where the "posterior" weight of each portfolio equals its accumulated wealth. The resulting wealth satisfies:

$$S_n^{\text{univ}}={\int }_{\mathbf{b}}{S}_n(\mathbf{b})d\mathbf{b}\ge \frac{1}{(n+1{)}^{m-1}}{S}_n(b^{\ast })$$

The inequality follows from the fact that the integral of the maximum is bounded below by the maximum divided by the "effective number" of portfolios. Taking logs: $\frac{1}{n}\log (S_n^{\text{univ}}/S_n(b^{\ast }))\ge -\frac{(m-1)\log (n+1)}{n}\to 0$. The universal strategy achieves the same doubling rate as the optimal strategy that knows $F$ in advance, with polynomial convergence in $n$ and exponential dependence on $m$ ^{[source pending]}.

The universal portfolio has a natural interpretation in the language of prediction with expert advice. The "experts" are the constant-rebalanced portfolios, and the loss function is the log-loss. The universal strategy is the Bayesian mixture, and the regret bound is $\frac{(m-1)\log (n+1)}{n}$ per round. This connects Kelly betting to the online learning literature, where the same mixture strategy (with different loss functions) appears as the "exponentiated gradient" or "Hedge" algorithm.

6. Exercises

Exercise 1 (Doubling rate computation). A 3-horse race has $p(1)=1/2$, $p(2)=1/3$, $p(3)=1/6$ and odds $o(1)=2.5$, $o(2)=4$, $o(3)=7$. Compute the optimal strategy $b^{\ast }$, the optimal doubling rate $W^{\ast }$, and the doubling rate of the uniform strategy $b=(1/3,1/3,1/3)$.

Exercise 2 (KL divergence proof of optimality). Show that for any strategy $\mathbf{b}$, $W^{\ast }-W(\mathbf{b},p)=D(p\Vert b)$. Use this to prove that proportional betting is the unique optimal strategy. Hint: $D(p\Vert b)\ge 0$ with equality iff $b=p$.

Exercise 3 (Fair odds). For a 3-horse race with $p=(1/2,1/3,1/6)$ and fair odds, verify that $W^{\ast }=0$. Then compute $W^{\ast }$ when the odds are $o(x)=2/p(x)$ (uniformly doubled) and verify $W^{\ast }=\log 2$.

Exercise 4 (Side information gain). A binary race has $X\in \{1,2\}$, $p(1)=p(2)=1/2$, and odds $o(1)=3$, $o(2)=3/2$. The side signal $Y$ satisfies $\mathrm{Pr}(Y=X)=0.7$. Compute $I(X;Y)$, the unconditional $W^{\ast }$, and the conditional $W_{\text{cond}}^{\ast }$. Verify $\Delta W=I(X;Y)$.

Exercise 5 (Suboptimal strategies). For the race in Exercise 1, compute $W$ for the strategies: (a) $b=p$ (proportional), (b) $b=(1/3,1/3,1/3)$ (uniform), (c) $b=(0.9,0.05,0.05)$ (concentrated). Rank them and compute the KL divergence $D(p\Vert b)$ for each.

Exercise 6 (Numerical simulation). Simulate 10,000 rounds of a 3-horse race with $p=(0.5,0.3,0.2)$ and odds $o=(2.2,3.5,6)$. Compare the empirical $\frac{1}{n}\log S_n$ for proportional betting, uniform betting, and a concentrated strategy. Verify convergence to the theoretical doubling rates.

Exercise 7 (Continuous portfolio). Two assets with $(X_1,X_2)=(3,2/3)$ w.p. $1/3$ and $(2/3,3/2)$ w.p. $2/3$. Find the log-optimal portfolio and doubling rate. Hint: take the derivative of $W(b)$ with respect to $b_1$ and set it to zero.

Exercise 8 (General odds, general proof). Prove that for arbitrary positive odds $o(x)$, the conditional Kelly strategy $b^{\ast }(x|y)=p(x|y)$ achieves $\Delta W=I(X;Y)$. Show explicitly that the odds cancel in the difference $W_{\text{cond}}^{\ast }-W_{\text{uncond}}^{\ast }$.

Connections Master

Relationships across the curriculum

Mutual information (46.01.02). The identity $\Delta W=I(X;Y)$ provides the second major operational interpretation of mutual information. The first is channel capacity: $C={\max }_{p(x)}I(X;Y)$ is the maximum reliable communication rate over a channel. The gambling interpretation says $I(X;Y)$ is the maximum increase in wealth growth rate from observing a correlated signal. Both involve maximizing mutual information over a choice variable (input distribution vs. conditional betting strategy), but the domains differ: communication vs. investment.

The structural parallel runs deeper. In channel coding, the encoder chooses $p(x)$ to maximize $I(X;Y)$. In gambling, the gambler chooses $b(x|y)=p(x|y)$ (the posterior) and the growth gain is $I(X;Y)$ for free — the signal $Y$ comes from nature. The maximization in the gambling case is immediate (bet the posterior), and the substantive content is the interpretation of the resulting quantity.

Source coding (46.02.02). The duality between entropy as a compression limit and entropy as a growth rate is one of the deepest connections in information theory. Shannon's source coding theorem says the minimum expected code length for $X$ is $H(X)$ bits. The Kelly theorem says the optimal doubling rate with fair odds is determined by $H(X)$. Both arise from the same functional ${\sum }_{x}p(x)\log p(x)$.

The duality is constructive. A prefix code with lengths $\ell (x)$ induces a gambling strategy $b(x)=2^{-\ell (x)}$. The expected log-wealth under this strategy with odds $o(x)=2^{\ell (x)}$ is $\sum p(x)\log b(x)o(x)=\sum p(x)\log 1=0$. A code that is nearly optimal ($\ell (x)\approx -\log p(x)$) corresponds to proportional betting. A suboptimal code ($\ell (x)\gg -\log p(x)$ for some $x$) corresponds to under-betting on likely outcomes, and the penalty in both contexts equals the KL divergence $D(p\Vert b)$ ^{[source pending]}.

Slepian-Wolf coding (46.05.01). Slepian-Wolf codes compress $X$ to rate $H(X|Y)$ when the decoder has access to side information $Y$, achieving the full joint entropy $H(X,Y)$ across both encoder and decoder. The gambling analog: the gambler with side information $Y$ achieves a doubling rate that is $I(X;Y)$ higher than without. In both cases, side information reduces the "cost" of uncertainty by exactly $I(X;Y)$. In compression, the saved bits equal $I(X;Y)$. In gambling, the gained growth rate equals $I(X;Y)$.

Large deviations (37.07.02). The probability of ruin under Kelly betting decays exponentially. The rate function is computed from Cramer's theorem applied to $\frac{1}{n}{\sum }_i\log b(X_i)o(X_i)$. Specifically, $\mathrm{Pr}(S_n\le ϵ)=\mathrm{Pr}(\frac{1}{n}{\sum }_i\log b(X_i)o(X_i)\le \frac{\log ϵ}{n})$. By the large-deviations principle, this probability decays as $e^{-n{I}_W(\delta )}$ where $I_W$ is the Fenchel-Legendre transform of the cumulant generating function of $\log b(X)o(X)$ and $\delta$ is the deviation of the sample average from its mean $W^{\ast }$. The Kelly strategy, by maximizing $W^{\ast }$, simultaneously pushes the "center" of the distribution of $\frac{1}{n}\log S_n$ to the right, making large negative deviations less probable for a given tail probability.

The precise relationship is as follows. Define the cumulant generating function $\Lambda (\theta )=\log \mathbb{E}[e^{\theta \log b(X)o(X)}]$. By the Gartner-Ellis theorem, the probability that $\frac{1}{n}\log S_n\le a$ for $a<W^{\ast }$ decays as:

$$\mathrm{Pr}\left(\frac{1}{n}\log S_n\le a\right)\asymp e^{-nI(a)}$$

where $I(a)={\sup }_{\theta }[\theta a-\Lambda (\theta )]$ is the rate function. For the Kelly strategy with $b=p$ and fair odds, $\log b(X)o(X)=\log 1=0$ deterministically, so $\Lambda (\theta )=0$ and $I(a)=0$ for $a=0$, $\infty$ otherwise. There is no fluctuation — wealth is constant. Under super-fair odds, the fluctuations are non-negligible, and the rate function depends on the distribution of $p(x)o(x)$.

This connects to a general principle: the Kelly strategy minimizes the probability of falling behind any competitor by a given factor over any time horizon. Breiman (1961) proved that for any fixed wealth target $C$, the Kelly strategy minimizes the expected time to reach $C$. The large-deviations perspective shows that Kelly also minimizes the exponential rate at which the probability of ruin decays ^{[source pending]}.

Rate-distortion (46.02.05). Both $R(D)$ and $W(\mathbf{b},p)$ are solutions to convex optimization problems involving $\mathbb{E}[\log ]$ functionals. The rate-distortion function minimizes $I(X;\hat{X})$ over conditional distributions satisfying an expected distortion constraint. The doubling rate maximizes $\mathbb{E}[\log b(X)o(X)]$ over strategies on the simplex. Both involve Lagrange multipliers, both have information-theoretic interpretations of their optimizers, and both connect to the duality between compression and estimation.

The parallel extends to the computational level. The rate-distortion function $R(D)$ is parameterized by a Lagrange multiplier $s$: $R(D_s)={\min }_{p(\hat{x}|x)}[I(X;\hat{X})-s\mathbb{E}[d(X,\hat{X})]]$. The doubling rate under general odds can be similarly parameterized by introducing a "risk-aversion" parameter $\alpha$: $W_{\alpha }={\max }_{\mathbf{b}}\frac{1}{\alpha }\mathbb{E}[\log ({\mathbf{b}}^T\mathbf{X}{)}^{\alpha }]$. For $\alpha =1$, this is the standard Kelly criterion. For $\alpha \to 0$, it approaches the entropy-like functional $\mathbb{E}[{\mathbf{b}}^T\mathbf{X}-1]$, the expected simple return. The family interpolates between growth-optimality ($\alpha =1$) and return-optimality ($\alpha =0$), paralleling the rate-distortion tradeoff between rate ($R$) and distortion ($D$).

Hypothesis testing (46.04.01). The optimal gambling strategy under uncertain odds involves a hypothesis-testing problem. If the true distribution is either $p$ or $q$, the gambler must decide which model to use for betting. The Kelly strategy under model uncertainty reduces to a likelihood-ratio test: bet according to the posterior-weighted average $\bar{p}(x)=\pi \cdot p(x)+(1-\pi )\cdot q(x)$ where $\pi$ is the posterior probability of $p$. The growth rate penalty from model uncertainty equals the Chernoff information between $p$ and $q$, connecting gambling to the large-deviations theory of hypothesis testing (unit 46.04.02).

Advanced topics

Competitive optimality. Cover (1991) proved that the log-optimal portfolio is competitively optimal in a multiplicative sense. For any constant-rebalanced portfolio $\mathbf{b}$, the wealth ratio $S_n(b^{\ast })/S_n(\mathbf{b})$ is a submartingale: $\mathbb{E}[S_{n+1}(b^{\ast })/S_{n+1}(\mathbf{b})\mid {\mathcal{F}}_n]\ge S_n(b^{\ast })/S_n(\mathbf{b})$. The log-optimal portfolio does not merely win on average; it dominates every competitor pathwise in this multiplicative expectation. This result strengthens Kelly from an expected-value criterion to a competitive guarantee.

Universal portfolios. Cover's universal portfolio (1986) achieves $W^{\ast }$ without knowing $F$. The strategy maintains a wealth-weighted mixture over all constant-rebalanced portfolios, with each portfolio's weight proportional to its accumulated wealth under the observed returns. This is a Bayesian mixture with uniform prior on the simplex. The wealth ratio satisfies $S_n^{\text{univ}}/S_n(b^{\ast })\ge (n+1{)}^{-(m-1)}$, giving $\frac{1}{n}\log (S_n^{\text{univ}}/S_n(b^{\ast }))\ge -\frac{(m-1)\log (n+1)}{n}\to 0$. The universal strategy achieves the optimal doubling rate asymptotically, with polynomial convergence in $n$ and exponential dependence on $m$.

Log-optimality and repeated games. The Kelly strategy is the solution to a minimax problem over nature's choice of distribution. Consider a two-player repeated game where the gambler chooses $\mathbf{b}$ and nature chooses the outcome $x$. The gambler wants to maximize $\mathbb{E}[\log {\mathbf{b}}^T\mathbf{X}]$ while nature is adversarial. The minimax strategy for the gambler is the log-optimal portfolio. This connects Kelly betting to online learning and regret minimization: the universal portfolio is the "follow the leader" strategy for the log-loss game.

Fractional Kelly and drawdown control. The fractional Kelly strategy ${\mathbf{b}}_f=f\cdot {\mathbf{b}}^{\ast }+(1-f)\cdot {\mathbf{b}}_{\text{safe}}$ achieves a growth rate $W_f\approx f\cdot W^{\ast }-\frac{f^2}{2}{\sigma }^2$ where ${\sigma }^2=\text{Var}[\log ({\mathbf{b}}^{\text{astT}}\mathbf{X})]$. The optimal $f$ that maximizes the Sharpe ratio (growth per unit volatility) is $f=W^{\ast }/{\sigma }^2$, which coincides with full Kelly. For drawdown-constrained investors, the relationship between $f$ and the maximum drawdown is $P(\text{drawdown}>d)\approx d^{-2(1-f)/f}$, showing that even small reductions from full Kelly dramatically reduce tail risk.

The fractional Kelly framework can be derived rigorously from the following Taylor expansion. Let $W(f)=\mathbb{E}[\log (f\cdot {\mathbf{b}}^{\text{astT}}\mathbf{X}+(1-f))]$. Expanding around $f=0$:

$$W(f)=f\cdot \mathbb{E}[{\mathbf{b}}^{\text{astT}}\mathbf{X}-1]-\frac{f^2}{2}\text{Var}[{\mathbf{b}}^{\text{astT}}\mathbf{X}]+O(f^3)$$

The first-order term $f\cdot W^{\ast }$ dominates for small $f$, and the second-order correction $-\frac{f^2}{2}{\sigma }^2$ captures the volatility drag. The optimal $f$ from this approximation is $f^{\ast }=W^{\ast }/{\sigma }^2$, the same as full Kelly when ${\mathbf{b}}^{\ast }$ is already the log-optimal portfolio. The fractional Kelly approach is most useful when the gambler's estimate of $p$ is noisy: the estimation error introduces additional variance, and the optimal $f$ is reduced accordingly. This connects to the "estimation error" literature in portfolio theory, where the Black-Litterman model and shrinkage estimators play roles analogous to fractional Kelly in reducing the impact of parameter uncertainty.

The horse race and the channel. Kelly's original insight was that the horse race is dual to the communication channel. Consider a channel with input $Y$ (the side information) and output $X$ (the race outcome). The channel capacity is $C={\max }_{p(y)}I(X;Y)$. The gambler's maximum growth rate with access to $Y$ is $W_{\text{cond}}^{\ast }=W_{\text{uncond}}^{\ast }+I(X;Y)$. If the gambler can choose the distribution of $Y$ (by designing the signal acquisition), they should maximize $I(X;Y)$ — which is exactly the channel capacity problem. The optimal "investigation" of the race outcome is the same as the optimal use of a communication channel. This duality, first noted by Kelly himself, is the reason the paper was titled "A New Interpretation of Information Rate" rather than "A New Theory of Gambling."

Historical & philosophical context Master

In 1956, John Kelly Jr., a physicist at Bell Labs, published "A New Interpretation of Information Rate" in the Bell System Technical Journal ^{[source pending]}. Kelly was directly inspired by Shannon's 1948 paper. He observed that if a gambler receives a noisy signal about a race outcome transmitted through a channel, the capacity of that channel determines the maximum achievable growth rate. The connection was exact: the mathematics of optimal gambling and the mathematics of reliable communication share the same structure. Kelly's paper reportedly drew the attention of Shannon himself, and the story goes that Shannon advised Kelly to tone down the gambling language for the Bell Labs audience.

Ed Thorp, a mathematician at MIT and later UC Irvine, operationalized Kelly's theory. His 1961 paper "A Favorable Strategy for Twenty-One" showed that Blackjack card counting creates a sequence of favorable bets where the Kelly strategy applies. Thorp used the Kelly criterion to size bets: increase the wager when the remaining deck is rich in tens and aces, decrease it otherwise. He then extended the approach to portfolio management, running one of the first quantitative hedge funds. His 1969 paper formalized the continuous-outcome extension and established practical viability ^{[source pending]}.

The relationship between Kelly betting and Markowitz mean-variance portfolio theory has been debated for over five decades. Mean-variance optimization maximizes expected return subject to a variance constraint, producing an efficient frontier in the risk-return plane. Kelly betting maximizes $\mathbb{E}[\log \text{wealth}]$, which has no variance constraint. The two approaches can yield radically different portfolios. The Kelly strategy can exhibit variance that a mean-variance investor would find intolerable — in a simple binary bet with a 60% chance of doubling and 40% chance of losing everything, Kelly bets 20% of wealth each round, producing large short-term swings. Conversely, a mean-variance optimum can have negative expected log-wealth, meaning ruin is certain in the long run.

Samuelson (1971, 1979) criticized Kelly betting on the grounds that maximizing $\mathbb{E}[\log S_n]$ does not maximize $\mathbb{E}[S_n]$ or $\mathbb{E}[U(S_n)]$ for concave utility functions other than log. His critique was pointed: in a single-period binary gamble where you win with probability $p>1/2$ and lose with probability $1-p$, the Kelly bet is $b^{\ast }=2p-1$. For $p=0.51$, this means betting 2% of wealth. The expected wealth after one round is maximized by betting everything (all-or-nothing), but this leads to ruin with probability 0.49. Samuelson argued that for any reasonable utility function, the Kelly bet is not optimal. However, in the repeated setting, the argument reverses: any strategy other than Kelly has a strictly lower almost-sure growth rate, and with probability 1, the Kelly bettor eventually surpasses all competitors. This "asymptotic" argument is the basis for Kelly's defenders.

The debate has not been fully resolved. The "fractional Kelly" compromise — betting a fraction $f\in (0,1)$ of the Kelly amount — interpolates between the two objectives. For an investor with constant relative risk aversion $\gamma >1$, the optimal strategy is fractional Kelly with $f=1/\gamma$; full Kelly corresponds to $\gamma =1$ (log utility). In practice, most quantitative funds that use Kelly-inspired position sizing employ fractional Kelly, typically with $f$ between 0.25 and 0.5. The reduction in growth rate is quadratic in the deviation from full Kelly ($W_f\approx f\cdot W^{\ast }-\frac{f^2}{2}{\sigma }^2$), so cutting the Kelly bet in half reduces the growth rate by roughly a quarter while halving the volatility of log-returns.

Bibliography Master

1. Kelly, J. L. (1956). "A New Interpretation of Information Rate." Bell System Technical Journal 35: 917-926. The founding paper that connected Shannon's information theory to proportional gambling and established the identity between growth rate and mutual information gain.

2. Thorp, E. O. (1969). "Optimal Gambling Systems for Favorable Games." Review of the International Statistical Institute 37:3: 273-293. Extended Kelly's discrete framework to continuous settings and demonstrated practical application through Blackjack card counting.

3. Cover, T. M. (1991). "Universal Portfolios." Mathematical Finance 1:1: 1-29. Introduced the universal portfolio strategy that achieves log-optimal growth without prior knowledge of the return distribution.

4. Cover, T. M. & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. Chapters 6 and 16 provide the definitive treatment of gambling as an information-theoretic problem and its extension to portfolio theory.

5. Breiman, L. (1961). "Optimal Gambling Systems for Favorable Games." Proceedings of the 4th Berkeley Symposium 1: 65-78. Proved that the Kelly strategy minimizes the expected time to reach a given wealth target, complementing the growth-rate maximization result.

6. Hakansson, N. H. (1971). "Capital Growth and the Mean-Variance Approach to Portfolio Selection." JFQA 6:1: 517-557. Analyzed the relationship between log-optimal growth and mean-variance efficiency, establishing conditions for agreement.

7. MacLean, L. C., Thorp, E. O., & Ziemba, W. T. (2011). The Kelly Capital Growth Investment Criterion. World Scientific. Comprehensive collection of the major papers on Kelly betting with commentary.

Meta Master

This unit formalized the horse race gambling model, defined the doubling rate $W(\mathbf{b},p)=\mathbb{E}[\log b(X)o(X)]$, proved that proportional betting $b^{\ast }=p$ is the unique optimal strategy via the non-negativity of KL divergence, and established that side information $Y$ increases the doubling rate by exactly $I(X;Y)$ for any odds structure. The extension to continuous asset returns through the log-optimal portfolio and Cover's universal portfolio was developed.

For the information-theory spine, this unit adds the third major operational interpretation of entropy and mutual information: entropy determines the doubling rate structure under general odds, and mutual information is the monetary value of side information. The identity $\Delta W=I(X;Y)$ ties gambling directly to the core information measures (unit 46.01.02), while the gambling-compression duality connects to source coding (unit 46.02.02). The universal portfolio extends the information-theoretic framework to sequential decision-making under uncertainty.