44.07.01 · optimization-control / optimal-control-pontryagin

Pontryagin's Maximum Principle for Constrained Optimal Control

shipped3 tiersLean: none

Anchor (Master): Pontryagin, Boltyanskii, Gamkrelidze & Mishchenko 1962 The Mathematical Theory of Optimal Processes (Interscience/Wiley) Ch. 1-2 (statement and needle-variation proof of the maximum principle, transversality conditions, the constancy of the Hamiltonian); Liberzon 2012 Calculus of Variations and Optimal Control Theory (Princeton University Press) Ch. 4 (a modern proof via needle variations and the adjoint/variational equations); Clarke 2013 Functional Analysis, Calculus of Variations and Optimal Control (Springer GTM 264) Ch. 22 (the nonsmooth maximum principle and the role of the abnormal multiplier)

Intuition Beginner

Imagine flying a small rocket to a fixed point and trying to spend as little fuel as possible. At each instant you choose how hard to burn the engine, but the throttle is not free to take any value: it can sit anywhere from fully off to fully open, and nothing beyond. That cap is the whole story. If the throttle could go arbitrarily large or arbitrarily negative, you could just nudge it to wherever the cost stops changing and call that the best setting. The cap forbids that move, and you need a rule that still works when the best setting is pinned against a limit.

Pontryagin's principle is that rule. It introduces a hidden bookkeeping quantity, one number for each thing you are tracking about the rocket, that records how much a tiny change in your current state is worth to the rest of the journey. Think of it as a running price tag on position and speed. Multiply that price tag by how each throttle setting would push the rocket, add the running fuel cost, and you get a single score for every allowed throttle setting at this instant.

The best choice is the throttle that makes this score as good as it can be among the allowed settings, evaluated separately at each instant. Not the setting where the score is flat, the setting where the score is best inside the cap. When the cap is binding, the best score sits at the edge, full on or full off, never in between.

That edge behaviour is why the principle matters. The old recipe from plain calculus, find where a derivative is zero, only finds flat spots in the interior. Pontryagin's principle keeps working when the answer lives on the boundary of what you are allowed to do.

Visual Beginner

The picture is a single instant in the flight, frozen, showing how the principle scores each allowed throttle setting and picks the best one.

Read the table top to bottom as the comparison the principle runs at one moment in time. You hold the rocket's current state and its price tag fixed. You sweep the throttle across every allowed value. For each value you compute one score. The chosen throttle is the one whose score is smallest (least total cost-rate), and because the throttle is capped, that winner often sits at an endpoint of the allowed range.

allowed throttle	how it pushes the rocket	running cost	combined score
off (low edge)	gentle drift	tiny fuel use	candidate
middle	moderate push	moderate fuel	candidate
full (high edge)	hard push	high fuel	candidate
chosen	the setting with the best combined score	—	winner

The takeaway: the principle turns a hard search over entire flight plans into a simple instant-by-instant contest among the allowed throttle settings, refereed by a price tag that someone hands you. The fact that the winner usually sits at an edge is exactly the feature that ordinary calculus misses.

Worked example Beginner

Take the simplest steering problem. A cart sits on a line at position $x$ , and you control its velocity directly: $x$ changes at the rate $u$ , where the throttle $u$ is capped between $- 1$ and $+ 1$ . You start at $x = 2$ and want to reach $x = 0$ in the least time. The running cost is just $1$ per unit time, since adding up $1$ over the trip gives the total time.

Step 1. Form the score. The price tag on position is a number $p$ . The combined score at an instant is the running cost plus the price tag times the push: $H = 1 + p u$ .

Step 2. Pick the best allowed throttle. To make $H$ smallest, look at the term $p u$ . If $p$ is positive, the smallest $p u$ comes from $u = - 1$ (the low edge). If $p$ is negative, the smallest comes from $u = + 1$ (the high edge). The throttle slams to one edge or the other; it is never in the middle.

Step 3. Find the price tag. The backward rule says the price tag does not change here, because the score $H$ does not depend on position $x$ at all. So $p$ stays constant for the whole trip.

Step 4. Read off the plan. We start at $x = 2$ , to the right of the target, so we must move left: $u = - 1$ . For $u = - 1$ to be the choice, the constant $p$ must be positive. With $u = - 1$ the cart moves left at speed $1$ , covering the distance $2$ in time $2$ .

Step 5. Check. Position goes $2 \to 0$ in $2$ time units at full leftward throttle, then stops. No allowed plan is faster, because $1$ is the top speed.

What this tells us: even with the throttle capped, the best plan is dead simple, run at one edge of the allowed range the whole way. The price tag $p$ never had to be computed exactly; its sign alone decided the direction. This edge-running behaviour, called bang-bang control, is the signature of constrained optimal control.

Check your understanding Beginner

Exercise (easy, multiple choice).

In Pontryagin's principle, how is the best control chosen at each instant?

A. By setting the derivative of the running cost to zero, ignoring any limits. B. By picking the allowed control that makes the combined score (the Hamiltonian) as small as possible. C. By choosing a fresh random control at each instant. D. By keeping the control at the middle of its allowed range.

Hint

The principle compares a single score across all allowed controls and selects the winner, even when the winner sits at an edge.

Answer

B. By picking the allowed control that minimises the combined score. At each instant the principle holds the state and the price tag (costate) fixed and selects the control in the allowed set that makes the Hamiltonian smallest. Feedback-correct: this is a pointwise minimisation over the constraint set, so the answer can land on a boundary. Feedback-wrong: setting a derivative to zero (A) only finds interior flat spots and fails when the optimum is on the boundary; the control is not random (C); and there is no reason it sits in the middle (D) — capped problems often run at an edge.

Exercise (easy, true-false).

True or false: when the allowed control range has hard limits, the optimal control found by Pontryagin's principle often sits at one of those limits rather than strictly inside.

Hint

Recall the cart example, where the throttle ran at $+ 1$ or $- 1$ , never in between.

Answer

True. When the score is minimised over a bounded set, the minimiser is frequently at the boundary of that set, producing bang-bang control that jumps between extremes. Feedback-correct: this is exactly the case ordinary stationarity (derivative equals zero) cannot capture, which is why the principle is stated as a minimisation, not a flat-spot condition. Feedback-wrong: interior optima do occur when the limits are not binding, but the boundary case is the one the principle was built to handle.

Formal definition Intermediate+

Consider the controlled system $\overset{x}{˙} = f (x, u, t)$ of 02.18.04's variational lineage, with state $x (t) \in R^{n}$ , control $u (t) \in U \subseteq R^{m}$ a measurable function taking values in a fixed nonempty constraint set $U$ , and $f : R^{n} \times U \times R \to R^{n}$ continuously differentiable in $(x, t)$ and continuous in $u$ . Fix an initial condition $x (t_{0}) = x_{0}$ and a time interval $[t_{0}, T]$ (with $T$ either fixed or free).

Definition (Bolza optimal-control problem). Among all measurable controls $u : [t_{0}, T] \to U$ with the resulting absolutely continuous state trajectory, minimise the cost

J (u) = φ (x (T)) + \int_{t_{0}}^{T} L (x (t), u (t), t) d t,

subject to $\overset{x}{˙} = f (x, u, t)$ , $x (t_{0}) = x_{0}$ , and the constraint $u (t) \in U$ for almost every $t$ . Here $L$ is the running cost (Lagrangian) and $φ$ the terminal cost. A control attaining the minimum is optimal and denoted $u^{⋆}$ , with optimal trajectory $x^{⋆}$ .

Definition (control Hamiltonian). The Hamiltonian (in normal form, multiplier on the cost equal to one) is

H (x, u, p, t) = L (x, u, t) + p^{T} f (x, u, t),

where $p (t) \in R^{n}$ is the costate (or adjoint) vector, a row of price tags on the state. The sign convention here pairs $H$ with minimisation of $J$ ; Pontryagin's original convention writes $H = p^{T} f - L$ and maximises, whence "maximum principle." The two are mirror images under $p \mapsto - p$ .

Definition (costate / adjoint equation). The costate evolves backward by the linear time-varying ODE

\overset{p}{˙} = - \frac{\partial H}{\partial x} (x^{⋆}, u^{⋆}, p, t) = - (\frac{\partial L}{\partial x})^{T} - (\frac{\partial f}{\partial x})^{T} p,

driven along the optimal trajectory. The matrix $\partial f / \partial x$ is the state Jacobian.

Definition (transversality / terminal conditions). The boundary data for the costate at the free end depend on the endpoint constraints:

Free terminal state, fixed $T$ : $p (T) = (\partial φ / \partial x) (x^{⋆} (T))^{T}$ .
Fixed terminal state $x (T) = x_{T}$ : $p (T)$ is free (determined by the requirement that the trajectory hit $x_{T}$ ).
Free terminal time $T$ : additionally $H (x^{⋆} (T), u^{⋆} (T), p (T), T) = - \partial φ / \partial t$ at $T$ (for autonomous $φ$ this reads $H (T) = 0$ ).

Definition (Pontryagin minimum principle). An optimal pair $(x^{⋆}, u^{⋆})$ admits a costate $p$ solving the adjoint equation with the transversality conditions, such that for almost every $t$ ,

u^{⋆} (t) \in ar g v \in U min H (x^{⋆} (t), v, p (t), t) .

The defining feature is pointwise minimisation over the constraint set $U$ , not the interior stationarity $\partial H / \partial u = 0$ . When $U$ is open and the minimiser is interior, the condition does collapse to $\partial H / \partial u = 0$ ; the principle is strictly stronger when the minimiser sits on $\partial U$ .

Notation: $x$ is the state, $u \in U$ the control, $p$ the costate, $H$ the Hamiltonian, $L$ the running cost, $φ$ the terminal cost, $f$ the dynamics, $\partial f / \partial x$ the state Jacobian; $ar g min_{v \in U}$ the pointwise minimiser over the constraint set.

Counterexamples to common slips

The maximum principle is a necessary, not sufficient, condition. A pair satisfying the adjoint equation, transversality, and pointwise minimisation is an extremal; it need not be optimal. Sufficiency requires extra structure (convexity of $H$ in $(x, u)$ , or verification against the Hamilton-Jacobi-Bellman value function). Treating an extremal as automatically optimal is the most common error.
The pointwise condition is minimisation, not stationarity. Writing $\partial H / \partial u = 0$ as the optimality condition silently assumes the minimiser is interior to $U$ . For a box-constrained $u$ the minimiser is generically on the boundary, where $\partial H / \partial u \neq = 0$ ; the gradient points out of the feasible set. Stationarity is the open-set special case, not the general rule.
The normal multiplier is not guaranteed. In the abnormal case the cost multiplier $p_{0}$ multiplying $L$ vanishes and $H = p^{T} f$ carries no cost information; the extremal is forced entirely by the endpoint geometry. Normality (taking $p_{0} = 1$ , as above) holds under a controllability-type regularity at the endpoint and fails for degenerate problems; assuming normality without checking is a genuine pitfall.

Key theorem with proof Intermediate+

Theorem (Pontryagin maximum principle, free-endpoint Bolza problem; Pontryagin et al. Ch. 1-2 ^{[source pending]}; Liberzon Ch. 4 ^{[source pending]}). Let $u^{⋆}$ be optimal for the problem of minimising $J (u) = φ (x (T)) + \int_{t_{0}}^{T} L (x, u, t) d t$ subject to $\overset{x}{˙} = f (x, u, t)$ , $x (t_{0}) = x_{0}$ , $u (t) \in U$ , with $T$ fixed and $x (T)$ free, under the smoothness hypotheses above. Then there is an absolutely continuous costate $p : [t_{0}, T] \to R^{n}$ satisfying the adjoint equation $\overset{p}{˙} = - (\partial H / \partial x)^{T}$ with terminal condition $p (T) = (\partial φ / \partial x) (x^{⋆} (T))^{T}$ , such that for almost every $t$ the optimal control minimises the Hamiltonian pointwise: $H (x^{⋆} (t), u^{⋆} (t), p (t), t) \leq H (x^{⋆} (t), v, p (t), t)$ for all $v \in U$ .

Proof. The construction is the needle (spike) variation. Fix a Lebesgue point $τ \in (t_{0}, T)$ of $u^{⋆}$ and a value $w \in U$ . For small $ε > 0$ define the perturbed control

u_{ε} (t) = {w, u^{⋆} (t), t \in (τ - ε, τ], otherwise,

a brief spike to $w$ on an interval of length $ε$ , admissible because $w \in U$ . Let $x_{ε}$ be the resulting trajectory. Before $τ - ε$ the trajectory is unchanged. Across the spike, the state at $τ$ shifts by

x_{ε} (τ) - x^{⋆} (τ) = ε [f (x^{⋆} (τ), w, τ) - f (x^{⋆} (τ), u^{⋆} (τ), τ)] + o (ε) =: ε ξ + o (ε),

since on $(τ - ε, τ]$ the dynamics integrate $f (\cdot, w, \cdot)$ instead of $f (\cdot, u^{⋆}, \cdot)$ and the integrand is continuous at the Lebesgue point. The vector $ξ = f (x^{⋆} (τ), w, τ) - f (x^{⋆} (τ), u^{⋆} (τ), τ)$ is the spike direction.

For $t > τ$ the controls agree again, so the deviation $δ x (t) := lim_{ε \to 0} ε^{- 1} (x_{ε} (t) - x^{⋆} (t))$ propagates by the variational equation obtained by linearising $\overset{x}{˙} = f$ about the optimal trajectory:

\dot{δ x} = \frac{\partial f}{\partial x} (x^{⋆}, u^{⋆}, t) δ x, δ x (τ) = ξ .

This is a linear ODE; write its state-transition matrix as $Φ (t, τ)$ , so $δ x (t) = Φ (t, τ) ξ$ for $t \geq τ$ .

Optimality requires $J (u_{ε}) \geq J (u^{⋆})$ for all small $ε > 0$ , hence the first-order change in cost is non-negative:

0 \leq \frac{d}{d ε}_{ε = 0^{+}} J (u_{ε}) = \frac{\partial φ}{\partial x} (x^{⋆} (T)) δ x (T) + \int_{τ}^{T} \frac{\partial L}{\partial x} (x^{⋆}, u^{⋆}, t) δ x (t) d t + ε -spike cost term,

where the explicit spike contributes the running-cost difference $L (x^{⋆} (τ), w, τ) - L (x^{⋆} (τ), u^{⋆} (τ), τ)$ over the length- $ε$ interval.

Now introduce the costate $p$ defined as the solution of the adjoint equation $\overset{p}{˙} = - (\partial L / \partial x)^{T} - (\partial f / \partial x)^{T} p$ with $p (T) = (\partial φ / \partial x) (x^{⋆} (T))^{T}$ . The key identity is that $p^{T} δ x$ accumulates exactly the integrated running-cost sensitivity: differentiating,

\frac{d}{d t} (p^{T} δ x) = \overset{p}{˙}^{T} δ x + p^{T} \dot{δ x} = (- \frac{\partial L}{\partial x} - p^{T} \frac{\partial f}{\partial x}) δ x + p^{T} \frac{\partial f}{\partial x} δ x = - \frac{\partial L}{\partial x} δ x .

Integrating from $τ$ to $T$ and using $p (T) = (\partial φ / \partial x)^{T}$ ,

\frac{\partial φ}{\partial x} δ x (T) + \int_{τ}^{T} \frac{\partial L}{\partial x} δ x d t = p (T)^{T} δ x (T) - \int_{τ}^{T} \frac{d}{d t} (p^{T} δ x) d t - \int_{τ}^{T} \frac{d}{d t} (p^{T} δ x) d t \cdot 0 = p (τ)^{T} δ x (τ) = p (τ)^{T} ξ .

So the entire downstream cost sensitivity collapses to the instantaneous inner product $p (τ)^{T} ξ$ . The inequality becomes

0 \leq [L (x^{⋆} (τ), w, τ) - L (x^{⋆} (τ), u^{⋆} (τ), τ)] + p (τ)^{T} [f (x^{⋆} (τ), w, τ) - f (x^{⋆} (τ), u^{⋆} (τ), τ)],

which is exactly $H (x^{⋆} (τ), w, p (τ), τ) - H (x^{⋆} (τ), u^{⋆} (τ), p (τ), τ) \geq 0$ . Since $w \in U$ was arbitrary and $τ$ ranges over the full-measure set of Lebesgue points, $u^{⋆} (τ)$ minimises $H (x^{⋆} (τ), \cdot, p (τ), τ)$ over $U$ for almost every $τ$ . $□$

Bridge. This needle-variation identity builds toward the Advanced results, where the same costate-pairing argument yields the constancy of the Hamiltonian along autonomous optimal trajectories and the transversality conditions for free terminal time, and it appears again in the linear-quadratic specialisation, where the adjoint equation and the dynamics together form the Hamiltonian system whose stable invariant subspace produces the Riccati matrix of 44.07.02. The foundational reason the principle generalises the Euler-Lagrange equations of 09.02.02 is visible in the proof: when $U$ is open and $u^{⋆}$ interior, the pointwise minimisation $min_{w \in U} H$ reduces to $\partial H / \partial u = 0$ , which combined with the adjoint equation reproduces $\frac{d}{d t} (\partial L / \partial \overset{q}{˙}) = \partial L / \partial q$ under the identification $\overset{q}{˙} = u$ , $p = \partial L / \partial \overset{q}{˙}$ . This is exactly the move from interior stationarity to boundary minimisation; the costate is dual to the variational multiplier, and putting these together, the bridge is the recognition that Euler-Lagrange handles unconstrained interior optima while Pontryagin handles the constrained boundary case by replacing a derivative-zero condition with a genuine minimisation over the admissible set.

Exercises Intermediate+

Exercise 3 (medium, symbolic).

Show that for an unconstrained control ( $U = R^{m}$ ) with interior optimum, Pontryagin's pointwise condition reduces to $\partial H / \partial u = 0$ , and combine it with the adjoint equation to recover the Euler-Lagrange form for the special case $f (x, u) = u$ , $L = L (x, u)$ .

Hint

Interior minimisation of $H$ in $u$ gives $\partial H / \partial u = 0$ . With $f = u$ , identify $p = \partial L / \partial u$ via this stationarity and differentiate.

Answer

With $U = R^{m}$ and an interior minimiser, the first-order condition is $\partial H / \partial u = \partial L / \partial u + p^{T} \partial f / \partial u = 0$ . For $f = u$ , $\partial f / \partial u = I$ , so $\partial L / \partial u + p^{T} = 0$ , i.e. $p = - (\partial L / \partial u)^{T}$ . The adjoint equation reads $\overset{p}{˙} = - (\partial L / \partial x)^{T} - (\partial f / \partial x)^{T} p = - (\partial L / \partial x)^{T}$ (since $f = u$ has $\partial f / \partial x = 0$ ). Substituting $p = - (\partial L / \partial u)^{T}$ gives $- \frac{d}{d t} (\partial L / \partial u)^{T} = - (\partial L / \partial x)^{T}$ , i.e. $\frac{d}{d t} \frac{\partial L}{\partial u} = \frac{\partial L}{\partial x}$ . With $u = \overset{x}{˙} = \overset{q}{˙}$ this is the Euler-Lagrange equation $\frac{d}{d t} \frac{\partial L}{\partial q ˙} - \frac{\partial L}{\partial q} = 0$ . Rubric: full credit for the stationarity step, the costate identification, and the recovery of Euler-Lagrange.

Exercise 4 (medium, symbolic).

Prove that for an autonomous problem ( $f$ and $L$ independent of $t$ ) the Hamiltonian is constant along the optimal trajectory: $\frac{d}{d t} H (x^{⋆} (t), u^{⋆} (t), p (t)) = 0$ where the control is held at its optimal pointwise minimiser.

Hint

Differentiate $H$ along the trajectory; use the dynamics $\overset{x}{˙} = \partial H / \partial p$ , the adjoint $\overset{p}{˙} = - \partial H / \partial x$ , and the envelope fact $\partial H / \partial u \cdot \overset{u}{˙} = 0$ at the minimiser.

Answer

Along the trajectory, $\frac{d H}{d t} = \frac{\partial H}{\partial x} \overset{x}{˙} + \frac{\partial H}{\partial u} \overset{u}{˙} + \frac{\partial H}{\partial p} \overset{p}{˙} + \frac{\partial H}{\partial t}$ . For an autonomous problem $\partial H / \partial t = 0$ . The canonical relations are $\overset{x}{˙} = f = \partial H / \partial p$ and $\overset{p}{˙} = - \partial H / \partial x$ , so $\frac{\partial H}{\partial x} \overset{x}{˙} + \frac{\partial H}{\partial p} \overset{p}{˙} = \frac{\partial H}{\partial x} \frac{\partial H}{\partial p} - \frac{\partial H}{\partial p} \frac{\partial H}{\partial x} = 0$ . The remaining term $\frac{\partial H}{\partial u} \overset{u}{˙}$ vanishes: at an interior minimiser $\partial H / \partial u = 0$ , and at a boundary minimiser $u^{⋆}$ is locally constant on each smooth arc so $\overset{u}{˙} = 0$ there (with the contribution vanishing across the measure-zero switching set). Hence $d H / d t = 0$ and $H$ is constant. Rubric: full credit for the cancellation of the canonical terms and the envelope/boundary argument.

Exercise 6 (medium, symbolic).

Write the adjoint equation and the Hamiltonian for the double integrator $\overset{x}{˙}_{1} = x_{2}$ , $\overset{x}{˙}_{2} = u$ , $u \in [- 1, 1]$ , with $L = 1$ (minimum-time to the origin), and show that the switching function $σ (t) = p_{2} (t)$ governs the control via $u^{⋆} = - sign (p_{2})$ . Show $p_{2}$ is affine in $t$ , so there is at most one switch.

Hint

$H = 1 + p_{1} x_{2} + p_{2} u$ ; the adjoint is $\overset{p}{˙} = - \partial H / \partial x$ . Minimising over $u$ uses only the $p_{2} u$ term.

Answer

$H = 1 + p_{1} x_{2} + p_{2} u$ . The adjoint equations are $\overset{p}{˙}_{1} = - \partial H / \partial x_{1} = 0$ and $\overset{p}{˙}_{2} = - \partial H / \partial x_{2} = - p_{1}$ . So $p_{1}$ is a constant $c$ and $p_{2} (t) = p_{2} (0) - c t$ , affine in $t$ . Minimising $H$ over $u \in [- 1, 1]$ involves only $p_{2} u$ , giving $u^{⋆} = - sign (p_{2})$ , so $p_{2}$ is the switching function. An affine function changes sign at most once, so the optimal control switches between $+ 1$ and $- 1$ at most once: the minimum-time double-integrator solution is bang-bang with a single switch. Rubric: full credit for the Hamiltonian, both adjoint equations, the affine $p_{2}$ , and the at-most-one-switch conclusion.

Exercise 7 (hard, symbolic).

Derive the free-terminal-time transversality condition $H (x^{⋆} (T), u^{⋆} (T), p (T)) = 0$ (for autonomous $φ$ ) by considering a variation of the terminal time $T \mapsto T + δ T$ and requiring stationarity of the cost.

Hint

A change $δ T$ changes the cost by the terminal-cost variation plus the running cost over the extra time $L δ T$ ; combine with the costate pairing to express the total as $H (T) δ T$ .

Answer

Extend the optimal trajectory past $T$ by $δ T$ with the optimal control held. The terminal cost changes by $\frac{\partial φ}{\partial x} \overset{x}{˙} (T) δ T = p (T)^{T} f (x^{⋆} (T), u^{⋆} (T)) δ T$ (using transversality $p (T) = (\partial φ / \partial x)^{T}$ ), and the integral cost gains $L (x^{⋆} (T), u^{⋆} (T)) δ T$ . The total first-order change is $[L (x^{⋆} (T), u^{⋆} (T)) + p (T)^{T} f (x^{⋆} (T), u^{⋆} (T))] δ T = H (x^{⋆} (T), u^{⋆} (T), p (T)) δ T$ . Since $T$ is free, $δ T$ is of arbitrary sign, so stationarity forces the bracket to vanish: $H (x^{⋆} (T), u^{⋆} (T), p (T)) = 0$ . Combined with Exercise 4's constancy, $H \equiv 0$ along the whole autonomous free-time optimum. Rubric: full credit for the two cost-variation pieces, the assembly into $H$ , and the free-sign conclusion.

Exercise 8 (hard, symbolic).

For the linear-quadratic problem $\overset{x}{˙} = A x + B u$ , $L = \frac{1}{2} (x^{T} Q x + u^{T} R u)$ , $U = R^{m}$ , $R ≻ 0$ , $φ = \frac{1}{2} x (T)^{T} S x (T)$ , apply the maximum principle: derive $u^{⋆} = - R^{- 1} B^{T} p$ and show the state-costate pair satisfies the Hamiltonian system $\frac{d}{dt}\begin{psmallmatrix}x\\p\end{psmallmatrix} = \begin{psmallmatrix}A & -BR^{-1}B^{\mathsf T}\\ -Q & -A^{\mathsf T}\end{psmallmatrix}\begin{psmallmatrix}x\\p\end{psmallmatrix}$ , whose matrix is the Hamiltonian of 44.07.02.

Hint

$H = \frac{1}{2} (x^{T} Q x + u^{T} R u) + p^{T} (A x + B u)$ ; interior minimisation gives $u^{⋆}$ , and the adjoint gives the bottom row.

Answer

$H = \frac{1}{2} x^{T} Q x + \frac{1}{2} u^{T} R u + p^{T} (A x + B u)$ . Interior minimisation: $\partial H / \partial u = R u + B^{T} p = 0$ , so $u^{⋆} = - R^{- 1} B^{T} p$ (a minimum since $R ≻ 0$ ). Substituting into the dynamics, $\overset{x}{˙} = A x + B u^{⋆} = A x - B R^{- 1} B^{T} p$ . The adjoint equation is $\overset{p}{˙} = - \partial H / \partial x = - Q x - A^{T} p$ . Stacking,

\frac{d}{d t} (x p) = (A - Q - B R^{- 1} B^{T} - A^{T}) (x p),

exactly the Hamiltonian matrix $H$ of 44.07.02. The ansatz $p = P x$ then yields the Riccati equation. Rubric: full credit for $u^{⋆}$ , both rows of the Hamiltonian system, and identifying $H$ .

Advanced results Master

Theorem (constancy of the Hamiltonian; Pontryagin et al. Ch. 2 ^{[source pending]}). For an autonomous problem ( $f$ , $L$ independent of $t$ ), along any extremal the value $H (x^{⋆} (t), u^{⋆} (t), p (t))$ is constant in $t$ . If in addition the terminal time is free with autonomous terminal cost, that constant is zero.

The constancy is the optimal-control image of energy conservation for autonomous Lagrangian systems 09.02.02: the Hamiltonian plays the role of the conserved energy function $E_{L} = \overset{q}{˙} \partial L / \partial \overset{q}{˙} - L$ , and the costate plays the role of the conjugate momentum. The free-time condition $H \equiv 0$ pins the otherwise-undetermined level set, fixing the duration through the requirement that the marginal value of one more instant of motion be zero.

Theorem (transversality on a terminal manifold; Pontryagin et al. Ch. 2 ^{[source pending]}; Liberzon Ch. 4 ^{[source pending]}). If the terminal state is constrained to a smooth manifold $M = {x : g (x) = 0} \subseteq R^{n}$ rather than fixed or free, then the costate satisfies $p (T) - (\partial φ / \partial x) (x^{⋆} (T))^{T} ⊥ T_{x^{⋆} (T)} M$ , i.e. $p (T) - (\partial φ / \partial x)^{T}$ is a linear combination of the rows of $\partial g / \partial x$ at $x^{⋆} (T)$ . The free-endpoint and fixed-endpoint conditions are the extremes: $M = R^{n}$ gives $p (T) = (\partial φ / \partial x)^{T}$ , and $M = {x_{T}}$ leaves $p (T)$ entirely free. Transversality is the boundary bookkeeping that closes the two-point boundary-value problem formed by the forward state equation and the backward adjoint equation.

Theorem (normal/abnormal form and the multiplier $p_{0}$ ; Clarke Ch. 22 ^{[source pending]}). In full generality the maximum principle carries a scalar multiplier $p_{0} \leq 0$ on the running cost, with the Hamiltonian $H = p_{0} L + p^{T} f$ , and the nontriviality condition $(p_{0}, p (t)) \neq = 0$ for all $t$ . The normal case $p_{0} \neq = 0$ permits the normalisation $p_{0} = - 1$ (or $+ 1$ in the minimisation convention) and recovers the cost-bearing Hamiltonian above; the abnormal case $p_{0} = 0$ produces extremals determined purely by the constraint and endpoint geometry, independent of $L$ . Abnormal extremals are not pathological curiosities: they arise in sub-Riemannian geometry and in problems where the attainable set has empty interior at the endpoint, and ignoring the abnormal branch can miss genuine minimisers.

Theorem (sufficiency via the HJB value function). Suppose $V (x, t)$ is a $C^{1}$ solution of the Hamilton-Jacobi-Bellman equation $- \partial_{t} V = min_{v \in U} [L (x, v, t) + (\partial_{x} V) f (x, v, t)]$ with $V (x, T) = φ (x)$ , and let $u^{⋆} (t)$ achieve the minimum with corresponding trajectory $x^{⋆}$ . Then $u^{⋆}$ is optimal, and $p (t) := (\partial_{x} V (x^{⋆} (t), t))^{T}$ is a costate satisfying the maximum principle. The HJB equation is the sufficient counterpart to the necessary maximum principle: where Pontryagin gives conditions every optimum must satisfy, dynamic programming gives a verification certificate. The costate is the gradient of the value function, $p = \partial_{x} V$ , which is the precise sense in which the price tag of the Beginner intuition is a marginal value.

Synthesis. Pontryagin's principle is one object — an optimal control minimises the Hamiltonian $H = L + p^{T} f$ pointwise over the admissible set $U$ while the costate $p$ runs backward along $\overset{p}{˙} = - \partial H / \partial x$ — and every face of the classical theory falls out of it. This is exactly why the principle generalises the Euler-Lagrange equations of 09.02.02: when $U$ is open and the optimum interior, pointwise minimisation collapses to $\partial H / \partial u = 0$ , the costate becomes the conjugate momentum, and the adjoint equation becomes $\frac{d}{d t} (\partial L / \partial \overset{q}{˙}) = \partial L / \partial q$ . The foundational reason the two theories diverge is that Euler-Lagrange tests a stationary flat spot while Pontryagin tests a genuine minimum, and only the latter survives when the optimum is pinned to a constraint boundary, producing bang-bang control. The central insight is that the needle variation localises the entire downstream cost sensitivity into the single inner product $p (τ)^{T} ξ$ , collapsing an infinite-dimensional search over control functions into an instant-by-instant minimisation refereed by the costate. Putting these together, the costate is dual to the value-function gradient $\partial_{x} V$ of dynamic programming, which is the bridge from the necessary maximum principle to the sufficient HJB equation; the linear-quadratic case makes this explicit, where the state-costate Hamiltonian system carries the matrix $H$ whose stable invariant subspace is the Riccati matrix of 44.07.02.

Full proof set Master

Proposition 1 (needle variation reduces optimality to pointwise Hamiltonian minimisation). Under the smoothness hypotheses, an optimal control $u^{⋆}$ minimises $H (x^{⋆} (t), \cdot, p (t), t)$ over $U$ for almost every $t$ , where $p$ solves the adjoint equation with terminal condition $p (T) = (\partial φ / \partial x)^{T}$ .

Proof. This is the Key theorem. A spike of value $w \in U$ on $(τ - ε, τ]$ at a Lebesgue point $τ$ creates the first-order state shift $ε ξ$ with $ξ = f (x^{⋆} (τ), w, τ) - f (x^{⋆} (τ), u^{⋆} (τ), τ)$ ; the deviation propagates by the variational equation $\dot{δ x} = (\partial f / \partial x) δ x$ . The costate makes $\frac{d}{d t} (p^{T} δ x) = - (\partial L / \partial x) δ x$ , so integrating collapses the downstream cost sensitivity to $p (τ)^{T} ξ$ . Non-negativity of the first-order cost change gives $H (x^{⋆} (τ), w, p (τ), τ) \geq H (x^{⋆} (τ), u^{⋆} (τ), p (τ), τ)$ for every $w \in U$ . $□$

Proposition 2 (the costate identity). Along the optimal trajectory, for the deviation $δ x$ solving the variational equation with $δ x (τ) = ξ$ , $p (τ)^{T} ξ = (\partial φ / \partial x) δ x (T) + \int_{τ}^{T} (\partial L / \partial x) δ x d t$ .

Proof. Differentiate $p^{T} δ x$ : with $\overset{p}{˙} = - (\partial L / \partial x)^{T} - (\partial f / \partial x)^{T} p$ and $\dot{δ x} = (\partial f / \partial x) δ x$ , the $(\partial f / \partial x)$ terms cancel and $\frac{d}{d t} (p^{T} δ x) = - (\partial L / \partial x) δ x$ . Integrate from $τ$ to $T$ : $p (T)^{T} δ x (T) - p (τ)^{T} ξ = - \int_{τ}^{T} (\partial L / \partial x) δ x d t$ . Substituting $p (T) = (\partial φ / \partial x)^{T}$ and rearranging gives the identity. $□$

Proposition 3 (constancy of the Hamiltonian, autonomous case). If $f$ and $L$ do not depend on $t$ , then $t \mapsto H (x^{⋆} (t), u^{⋆} (t), p (t))$ is constant.

Proof. On any open subinterval where $u^{⋆}$ is continuous, $\frac{d H}{d t} = \frac{\partial H}{\partial x} \overset{x}{˙} + \frac{\partial H}{\partial p} \overset{p}{˙} + \frac{\partial H}{\partial u} \overset{u}{˙}$ (no explicit $t$ ). The canonical relations $\overset{x}{˙} = \partial H / \partial p$ , $\overset{p}{˙} = - \partial H / \partial x$ cancel the first two: $\frac{\partial H}{\partial x} \frac{\partial H}{\partial p} - \frac{\partial H}{\partial p} \frac{\partial H}{\partial x} = 0$ . At an interior minimiser $\partial H / \partial u = 0$ ; at a boundary minimiser on a smooth arc $u^{⋆}$ is constant so $\overset{u}{˙} = 0$ . Thus $d H / d t = 0$ on each arc. At a switching instant the function $H (x^{⋆} (\cdot), u^{⋆} (\cdot), p (\cdot))$ is continuous because the pointwise minimum value $min_{v \in U} H (x^{⋆} (t), v, p (t))$ is continuous in $t$ (state and costate are continuous, $U$ fixed). A continuous function constant on each arc of a partition into finitely or countably many arcs is globally constant. $□$

Proposition 4 (free-time condition). If the terminal time is free and $φ$ is autonomous, then $H (x^{⋆} (T), u^{⋆} (T), p (T)) = 0$ .

Proof. Compare the optimum on $[t_{0}, T]$ with the trajectory continued under $u^{⋆}$ to $T + δ T$ . The terminal cost changes to first order by $(\partial φ / \partial x) \overset{x}{˙} (T) δ T = p (T)^{T} f (x^{⋆} (T), u^{⋆} (T)) δ T$ and the running integral by $L (x^{⋆} (T), u^{⋆} (T)) δ T$ , totalling $H (x^{⋆} (T), u^{⋆} (T), p (T)) δ T$ . Since $T$ is free, $δ T$ takes either sign and optimality forces this first-order change to vanish, so the Hamiltonian is zero at $T$ ; by Proposition 3 it is zero throughout. $□$

Proposition 5 (Euler-Lagrange as the open-set interior special case). For $U = R^{m}$ , $f (x, u) = u$ , and an interior optimum, the maximum principle is equivalent to the Euler-Lagrange equation $\frac{d}{d t} \frac{\partial L}{\partial q ˙} - \frac{\partial L}{\partial q} = 0$ with $\overset{q}{˙} = u$ .

Proof. Interior minimisation of $H = L (x, u) + p^{T} u$ gives $\partial L / \partial u + p^{T} = 0$ , so $p = - (\partial L / \partial u)^{T}$ . With $f = u$ , $\partial f / \partial x = 0$ , so the adjoint equation is $\overset{p}{˙} = - (\partial L / \partial x)^{T}$ . Differentiating the costate identification, $\overset{p}{˙} = - \frac{d}{d t} (\partial L / \partial u)^{T}$ , and equating, $\frac{d}{d t} (\partial L / \partial u) = \partial L / \partial x$ . Under $u = \overset{q}{˙}$ , $x = q$ this is the Euler-Lagrange equation. Conversely, any Euler-Lagrange solution defines $p = - \partial L / \partial \overset{q}{˙}$ satisfying both the adjoint equation and interior stationarity, hence the maximum principle for the unconstrained problem. $□$

Connections Master

The maximum principle is the constrained-control generalisation of the Euler-Lagrange equations of 09.02.02: when the constraint set $U$ is open and the optimum is interior, pointwise Hamiltonian minimisation reduces to $\partial H / \partial u = 0$ , the costate becomes the conjugate momentum $p = \partial L / \partial \overset{q}{˙}$ , and the adjoint equation becomes the Euler-Lagrange equation, so Euler-Lagrange is exactly the interior-stationary special case while Pontryagin handles bounded controls whose optima live on the boundary.

The principle rests on and extends the direct method of the calculus of variations of 02.18.04: where the direct method establishes existence of a minimiser by compactness and lower-semicontinuity arguments over a function space, the maximum principle supplies the necessary conditions that any such minimiser must satisfy, and the needle-variation proof is the control-theoretic refinement of the variational (first-variation) argument that the direct method's existence guarantee makes nonvacuous.

The maximum principle produces bang-bang controls whenever the Hamiltonian is affine in the control over a box constraint, the subject of the downstream unit on bang-bang and singular controls (44.07.03): the switching function is the control-coefficient of $H$ , its sign determines the extreme-value control, and singular arcs occur where the switching function vanishes on an interval, the degenerate case the bang-bang theory must separately resolve.

The maximum principle is the necessary-condition counterpart of the Hamilton-Jacobi-Bellman equation and dynamic programming (44.07.04): HJB gives a sufficient verification via the value function $V$ , the costate is the value gradient $p = \partial_{x} V$ evaluated along the optimal trajectory, and the two theories are dual readings of the same optimal-control problem — Pontryagin from the trajectory side, Bellman from the value side.

The linear-quadratic regulator of 44.07.02 is the maximum principle's most important solvable instance: the state-costate pair satisfies the Hamiltonian system with matrix $\mathcal H = \begin{psmallmatrix}A & -BR^{-1}B^{\mathsf T}\\ -Q & -A^{\mathsf T}\end{psmallmatrix}$ , the ansatz $p = P x$ converts the linear two-point boundary-value problem into the Riccati equation, and the stable invariant subspace of $H$ that produces the stabilising Riccati solution is exactly the optimal state-costate subspace selected by the maximum principle.

Historical & philosophical context Master

The maximum principle was formulated by Lev Pontryagin and his students Vladimir Boltyanskii, Revaz Gamkrelidze, and Evgenii Mishchenko at the Steklov Institute in the mid-1950s, motivated by problems of optimal trajectories for aircraft and rockets where the controls were physically bounded and the classical calculus of variations, built for interior optima, did not apply. The decisive new element was the replacement of the Euler-Lagrange stationarity condition by a pointwise maximisation of the Hamiltonian over a constraint set, proved by the needle-variation (spike-variation) technique that perturbs the control on a vanishingly short interval ^{[Pontryagin 1962]}. Boltyanskii is generally credited with the proof and with isolating the role of the abnormal multiplier; the priority among the collaborators was a point of later dispute.

The principle stood beside Richard Bellman's dynamic programming, developed contemporaneously in the United States, as the second pillar of modern optimal control; Bellman's Hamilton-Jacobi-Bellman equation provides the sufficient, value-function side that complements Pontryagin's necessary, trajectory side ^{[Bellman 1957]}. The relation between the costate and the gradient of the value function — the precise statement that $p = \partial_{x} V$ along the optimum — was clarified through the 1960s and is the bridge between the two theories. The modern, fully rigorous treatment encompassing nonsmooth data, the abnormal case, and the role of measurable selections is due to a line of work running through Hestenes, Berkovitz, and Clarke, the last of whom recast the principle in the language of nonsmooth analysis and generalised gradients ^{[Clarke 1983]}.

Bibliography Master

@book{Pontryagin1962,
  author    = {Pontryagin, Lev S. and Boltyanskii, Vladimir G. and Gamkrelidze, Revaz V. and Mishchenko, Evgenii F.},
  title     = {The Mathematical Theory of Optimal Processes},
  publisher = {Interscience Publishers (Wiley)},
  address   = {New York},
  year      = {1962},
  note      = {English translation by K. N. Trirogoff of the 1961 Russian original}
}

@book{Liberzon2012,
  author    = {Liberzon, Daniel},
  title     = {Calculus of Variations and Optimal Control Theory: A Concise Introduction},
  publisher = {Princeton University Press},
  address   = {Princeton, NJ},
  year      = {2012}
}

@book{Bertsekas2017DPv1,
  author    = {Bertsekas, Dimitri P.},
  title     = {Dynamic Programming and Optimal Control, Volume I},
  edition   = {4th},
  publisher = {Athena Scientific},
  address   = {Belmont, MA},
  year      = {2017}
}

@book{Bellman1957DP,
  author    = {Bellman, Richard E.},
  title     = {Dynamic Programming},
  publisher = {Princeton University Press},
  address   = {Princeton, NJ},
  year      = {1957}
}

@book{Clarke1983,
  author    = {Clarke, Frank H.},
  title     = {Optimization and Nonsmooth Analysis},
  publisher = {Wiley-Interscience},
  address   = {New York},
  year      = {1983}
}

@book{Clarke2013,
  author    = {Clarke, Frank H.},
  title     = {Functional Analysis, Calculus of Variations and Optimal Control},
  series    = {Graduate Texts in Mathematics},
  volume    = {264},
  publisher = {Springer},
  address   = {London},
  year      = {2013}
}

@book{AthansFalb1966,
  author    = {Athans, Michael and Falb, Peter L.},
  title     = {Optimal Control: An Introduction to the Theory and Its Applications},
  publisher = {McGraw-Hill},
  address   = {New York},
  year      = {1966}
}

@article{Boltyanskii1960,
  author  = {Boltyanskii, Vladimir G. and Gamkrelidze, Revaz V. and Pontryagin, Lev S.},
  title   = {The Theory of Optimal Processes. I. The Maximum Principle},
  journal = {Izvestiya Akademii Nauk SSSR, Seriya Matematicheskaya},
  volume  = {24},
  year    = {1960},
  pages   = {3--42}
}

Prerequisites

09.02.02
02.18.04

Tier anchors

beginner: Steering a rocket to a target while the throttle can only sit between off and full, where the best plan is read off a hidden bookkeeping variable rather than by nudging the throttle to a flat spot — Liberzon 2012 Calculus of Variations and Optimal Control Theory: A Concise Introduction (Princeton University Press) Ch. 4 (the maximum principle, stated and motivated from the calculus of variations); Bertsekas 2017 Dynamic Programming and Optimal Control Vol. 1 (4th ed., Athena Scientific) Ch. 3 (the continuous-time optimal-control problem and the minimum principle)
intermediate: Liberzon 2012 Calculus of Variations and Optimal Control Theory (Princeton University Press) Ch. 4 (statement of the maximum principle, the Hamiltonian, the adjoint equation, the pointwise maximisation condition, transversality, and the relation to the Euler-Lagrange equations); Bertsekas 2017 Dynamic Programming and Optimal Control Vol. 1 (4th ed., Athena Scientific) §3.3 (the minimum principle, the costate equation, and the boundary conditions)
master: Pontryagin, Boltyanskii, Gamkrelidze & Mishchenko 1962 The Mathematical Theory of Optimal Processes (Interscience/Wiley) Ch. 1-2 (statement and needle-variation proof of the maximum principle, transversality conditions, the constancy of the Hamiltonian); Liberzon 2012 Calculus of Variations and Optimal Control Theory (Princeton University Press) Ch. 4 (a modern proof via needle variations and the adjoint/variational equations); Clarke 2013 Functional Analysis, Calculus of Variations and Optimal Control (Springer GTM 264) Ch. 22 (the nonsmooth maximum principle and the role of the abnormal multiplier)

References

Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V. & Mishchenko, E. F. — The Mathematical Theory of Optimal Processes (Interscience/Wiley, 1962; English translation of the 1961 Russian original) · Ch. 1-2: statement of the maximum principle for the problem of minimising an integral cost subject to dx/dt = f(x,u) with u in an admissible control set U; the Hamiltonian H = p^T f - L (their sign convention), the adjoint system dp/dt = -partial H/partial x, the pointwise maximisation of H over U along the optimal control; transversality conditions for free-endpoint and fixed-endpoint problems; the constancy of H along the optimal trajectory for autonomous problems; the needle-variation (spike-variation) proof.
Liberzon, D. — Calculus of Variations and Optimal Control Theory: A Concise Introduction (Princeton University Press, 2012) · Ch. 4: the maximum principle stated for the Bolza problem min phi(x(T)) + int L(x,u,t) dt s.t. dx/dt = f(x,u,t), u(t) in U; the control Hamiltonian, the costate/adjoint ODE, the pointwise Hamiltonian-minimisation condition, transversality/terminal conditions; the derivation by needle variations and the adjoint (variational) equation; the explicit reduction to the Euler-Lagrange equations when U is open and the optimum is interior.
Bertsekas, D. P. — Dynamic Programming and Optimal Control, Vol. 1 (4th ed., Athena Scientific, 2017) · Ch. 3: the continuous-time deterministic optimal-control problem; the Pontryagin minimum principle, the Hamiltonian H(x,u,p) = L(x,u) + p^T f(x,u), the costate equation, and the boundary/transversality conditions; the connection to the Hamilton-Jacobi-Bellman equation as the sufficiency counterpart.

Estimated time

beginner: 18m
intermediate: 48m
master: 90m