The Hamilton-Jacobi-Bellman Equation and the Verification Theorem

Intuition Beginner

Suppose you are driving across a country and want to reach a city by sunset at the lowest fuel cost. One way to plan is to consider every possible whole route and compare totals. A smarter way is to build a map that labels each town with a single number: the cheapest cost still remaining to finish the trip from that town. If you had that map, planning would be effortless. Standing in any town, you just look one road ahead, add the cost of that road to the labelled number of the town it leads to, and take the road with the smallest total. The map carries all the foresight; your decisions become local.

The Hamilton-Jacobi-Bellman equation is the rule that map must obey. It says the label of a town equals the best you can do over one small step: the cost paid on that step plus the label of wherever the step lands you. Nothing about the distant future appears directly, because the neighbour's label already summarises everything beyond it. This is the same idea as the value tables of discrete dynamic programming, now stated for time that flows continuously rather than in separate stages.

The second half of the story is the most useful. Building the true cost-to-come map is hard. But checking a candidate is comparatively simple. If someone hands you a map and you verify that every town's label really does equal the best one-step-ahead total, then that map is guaranteed to be the true one, and the road it points to from each town is genuinely an optimal plan. This guarantee is the verification theorem: passing the local self-consistency test certifies global optimality.

That contrast matters. An earlier method gives conditions any best plan must satisfy, like a list of clues a culprit must match. Verification does the opposite: it confirms that a proposed plan actually is best.

Visual Beginner

Picture the cost-to-come map as a landscape where height at each point is the cheapest remaining cost from that point. The optimal motion always heads in the direction that descends this landscape fastest while paying the running cost. The map's defining property is purely local: at every point, the height equals the best one-small-step total.

The table below contrasts the two ways of certifying a plan, the necessary-condition method from the previous chapter and the verification method of this unit.

method	what it provides	direction of the guarantee
necessary conditions (maximum principle)	clues every optimum must satisfy	narrows the suspects; does not by itself convict
verification (this unit)	a self-consistent cost map	confirms a candidate is genuinely best

The takeaway: a single stored function of position and time turns global optimisation into a local check, and passing that check at every point is a certificate of optimality.

Worked example Beginner

Take a cart on a line whose position $x$ changes at the rate $u$, with the throttle free to be any number. You start at some position $x$ at time $t$ and must arrive at time $T=1$. The running cost is $\frac{1}{2}{u}^2$ (steering effort), and the finishing cost is $\frac{1}{2}x(1{)}^2$ (you are penalised for ending far from $0$). We want the cheapest plan and its cost-to-come map.

Step 1. Guess the shape of the map. Because every cost here is a square, a sensible guess is that the cheapest remaining cost from position $x$ at time $t$ has the form $V(x,t)=\frac{1}{2}k(t)x^2$ for some number $k(t)$ that depends only on the time left. At the finish, $V(x,1)=\frac{1}{2}{x}^2$, so $k(1)=1$.

Step 2. Write the local rule. From position $x$, choosing throttle $u$ for a small instant costs $\frac{1}{2}{u}^2$ now and moves you to $x+u\cdot (\text{small time})$. The best throttle balances effort against future savings. Working this out for the squared map gives the cheapest throttle as $u=-k(t)x$: steer back toward $0$, harder when the future penalty $k$ is larger.

Step 3. Find $k(t)$. The self-consistency of the map forces $k$ to satisfy $-k^{\prime }(t)=-k(t{)}^2$, that is $k^{\prime }(t)=k(t{)}^2$, together with $k(1)=1$. Solving gives $k(t)=1/(2-t)$.

Step 4. Read off the answer. At the start time $t=0$, the cheapest remaining cost from position $x$ is $V(x,0)=\frac{1}{2}\cdot \frac{1}{2}{x}^2=\frac{1}{4}{x}^2$, and the optimal steering at any moment is $u=-x/(2-t)$.

What this tells us: a single tidy function $V(x,t)=\frac{1}{2}k(t)x^2$ stored the entire plan, and the optimal control came straight out of it as a feedback rule that reacts to the current position. We never searched over whole trajectories; we found the map and let it steer.

Check your understanding Beginner

Formal definition Intermediate+

Consider the controlled system $\dot{x}=f(x,u,t)$ with state $x(s)\in {\mathbb{R}}^n$ and control $u(s)\in U\subseteq {\mathbb{R}}^m$ a measurable function, $f$ continuous and Lipschitz in $x$ uniformly on bounded sets so that trajectories exist and are unique. For an initial pair $(x,t)$ with $t\le T$, an admissible control $u(\cdot )$ on $[t,T]$ generates the trajectory $x^{x,t,u}(\cdot )$ started at $x^{x,t,u}(t)=x$. The cost is the Bolza functional of 44.07.01,

$$J(x,t;u(\cdot ))=\phi (x^{x,t,u}(T))+{\int }_t^{T}L(x^{x,t,u}(s),u(s),s)ds,$$

with running cost $L$ and terminal cost $\phi$, both continuous and bounded below.

Definition (value function). The value function (optimal cost-to-go) is

$$V(x,t)=\underset{u(\cdot )\text{admissible on }[t,T]}{\inf }J(x,t;u(\cdot )),V(x,T)=\phi (x).$$

It records the best achievable remaining cost from each state-time pair; the terminal identity is the boundary condition.

Definition (dynamic-programming principle). For every $h$ with $0\le h\le T-t$,

$$V(x,t)=\underset{u(\cdot )\text{on }[t,t+h]}{\inf }\left\{{\int }_t^{t+h}L(x(s),u(s),s)ds+V(x(t+h),t+h)\right\}.$$

This is the continuous-time principle of optimality: an optimal plan splits into an optimal first leg followed by an optimal continuation, with the value function summarising the continuation. It is the continuous counterpart of the discrete identity $J^{\ast }=T{J}^{\ast }$ of 44.08.02.

Definition (Hamilton-Jacobi-Bellman equation). The control Hamiltonian is

$$\mathcal{H}(x,p,t)=\underset{u\in U}{\min }[L(x,u,t)+p^{\mathsf{T}}f(x,u,t)],$$

the partial minimisation over the admissible controls of the integrand $L+p^{\mathsf{T}}f$ (the same pointwise minimisation that defines the optimal control in 44.07.01, with $p$ the gradient slot). The Hamilton-Jacobi-Bellman equation is the first-order partial differential equation

$$-\frac{\partial V}{\partial t}(x,t)=\underset{u\in U}{\min }[L(x,u,t)+{\nabla }_{x}V(x,t{)}^{\mathsf{T}}f(x,u,t)]=\mathcal{H}(x,{\nabla }_{x}V(x,t),t),$$

with terminal condition $V(x,T)=\phi (x)$. A control $u^{\star }(x,t)\in \arg {\min }_{u\in U}[L+{\nabla }_x{V}^{\mathsf{T}}f]$ is an optimal feedback law: a function of the current state and time rather than of the initial data.

Definition (greedy / feedback minimiser and the costate). Along a trajectory $x^{\star }(\cdot )$ generated by the feedback $u^{\star }$, the costate of 44.07.01 is recovered as $p(s):={\nabla }_{x}V(x^{\star }(s),s)$. The symbols $V$ (value function), ${\nabla }_{x}V$ (state gradient), ${\partial }_{t}V$ (time derivative), $\mathcal{H}$ (the minimised Hamiltonian), $U$ (control-constraint set), and $\phi ,L$ (terminal and running costs) are recorded in _meta/NOTATION.md.

Counterexamples to common slips Intermediate+

• "The value function is always differentiable, so the HJB PDE holds in the classical sense everywhere." False. Even with smooth data, $V$ is generically only Lipschitz: it develops kinks along surfaces where the optimal trajectory is non-unique (the analogue of caustics in the classical Hamilton-Jacobi setting 09.05.02). At such points ${\nabla }_{x}V$ does not exist, and the equation must be interpreted in the viscosity sense 02.18.05. A classical $C^1$ solution may simply fail to exist.

• "A smooth solution of the HJB equation is automatically the value function." It is, but only once the terminal condition $V(\cdot ,T)=\phi$ is imposed and the minimising feedback is admissible. The verification theorem needs both the PDE and the boundary data; the PDE alone has many solutions. This is the continuous shadow of the discrete fact that Bellman's equation $J=TJ$ pins down $J^{\ast }$ only because the contraction gives a unique fixed point 44.08.02.

• "The HJB equation gives necessary conditions, like the maximum principle." The opposite. HJB plus verification is a sufficiency statement: a solution certifies optimality. Pontryagin's principle 44.07.01 is necessity: it lists conditions any optimum must meet. Confusing the two directions is the central conceptual error; the link between them is $p={\nabla }_{x}V$ along the optimal trajectory.

Key theorem with proof Intermediate+

The signature result is the verification theorem: a smooth solution of the HJB equation that meets the terminal condition is the value function, and its pointwise minimiser is an optimal feedback control. This is the sufficiency counterpart to the necessity of the maximum principle 44.07.01, and the continuous-time face of the discrete optimality equation 44.08.02.

Theorem (verification / sufficiency). Let $W\in C^1({\mathbb{R}}^n\times [t_0,T])$ satisfy the Hamilton-Jacobi-Bellman equation

$$-{\partial }_{t}W(x,t)=\underset{u\in U}{\min }[L(x,u,t)+{\nabla }_{x}W(x,t{)}^{\mathsf{T}}f(x,u,t)],W(x,T)=\phi (x),$$

for all $(x,t)$. Suppose there is a measurable feedback $u^{\star }:{\mathbb{R}}^n\times [t_0,T]\to U$ attaining the minimum for every $(x,t)$, and that the closed-loop equation $\dot{x}=f(x,u^{\star }(x,t),t)$, $x(t)=x$, has a solution $x^{\star }(\cdot )$ on $[t,T]$ for each $(x,t)$. Then $W=V$ is the value function, and $u^{\star }$ is an optimal control: $J(x,t;u^{\star })=V(x,t)$ for every $(x,t)$.

Proof. Fix $(x,t)$ and let $u(\cdot )$ be any admissible control on $[t,T]$, with trajectory $x(\cdot )$, $x(t)=x$. Because $W$ is $C^1$, the map $s\mapsto W(x(s),s)$ is absolutely continuous and, by the chain rule,

$$\frac{d}{ds}W(x(s),s)={\partial }_{t}W(x(s),s)+{\nabla }_{x}W(x(s),s{)}^{\mathsf{T}}f(x(s),u(s),s).$$

The HJB equation gives ${\partial }_{t}W(x(s),s)=-{\min }_{v\in U}[L(x(s),v,s)+{\nabla }_x{W}^{\mathsf{T}}f(x(s),v,s)]\ge -[L(x(s),u(s),s)+{\nabla }_x{W}^{\mathsf{T}}f(x(s),u(s),s)]$, since the minimum over $v$ is no larger than the value at $v=u(s)$. Substituting,

$$\frac{d}{ds}W(x(s),s)\ge -L(x(s),u(s),s).$$

Integrate from $t$ to $T$ and use $W(x(T),T)=\phi (x(T))$:

$$\phi (x(T))-W(x,t)\ge -{\int }_t^{T}L(x(s),u(s),s)ds,$$

that is $W(x,t)\le \phi (x(T))+{\int }_t^{T}Lds=J(x,t;u)$. As $u(\cdot )$ was arbitrary, $W(x,t)\le {\inf }_{u}J(x,t;u)=V(x,t)$.

For the reverse inequality, run the closed-loop control $u^{\star }$ with trajectory $x^{\star }(\cdot )$. Now $u^{\star }(x^{\star }(s),s)$ attains the minimum, so the inequality above is an equality at every $s$:

$$\frac{d}{ds}W(x^{\star }(s),s)=-L(x^{\star }(s),u^{\star }(x^{\star }(s),s),s).$$

Integrating from $t$ to $T$ gives $\phi (x^{\star }(T))-W(x,t)=-{\int }_t^{T}Lds$, i.e. $W(x,t)=J(x,t;u^{\star })\ge V(x,t)$ (the cost of a specific admissible control is at least the infimum). Combining, $W(x,t)=V(x,t)$ and the bound is achieved by $u^{\star }$, so $u^{\star }$ is optimal. $\square$

Bridge. The verification theorem is the foundational reason a stored value function certifies optimality: the chain-rule identity $\frac{d}{ds}W(x(s),s)\ge -L$ telescopes the entire trajectory cost into the boundary values, and equality holds exactly along the feedback minimiser. This is exactly the continuous-time image of the discrete fixed-point equation $J^{\ast }=T{J}^{\ast }$ of 44.08.02 — where the discrete operator iterates to a unique fixed point, the continuous problem replaces iteration with a PDE whose terminal-valued solution plays the role of $J^{\ast }$, and the minimised Hamiltonian $\mathcal{H}(x,{\nabla }_{x}V,t)$ is the infinitesimal generator of that operator. It is dual to the necessity of Pontryagin's maximum principle 44.07.01: the central insight is that the costate is the value gradient, $p={\nabla }_{x}V$ along the optimal trajectory, so the trajectory-side necessary conditions and the value-side sufficient conditions are two readings of one problem. The construction generalises the classical Hamilton-Jacobi equation of 09.05.02, to which it reduces when there is no control to minimise over, and it builds toward the viscosity-solution theory of 02.18.05 that rescues the argument when $V$ is merely Lipschitz. Putting these together, the bridge is that the HJB equation appears again in the linear-quadratic regulator 44.07.02 as the quadratic-value reduction to the Riccati equation, where this verification proof becomes a direct, dimension-free certificate of the LQR optimum.

Exercises Intermediate+

Advanced results Master

The continuous theory extends in four directions that parallel the discrete development of 44.08.02: the exact LQR reduction, the relation of HJB to the classical Hamilton-Jacobi equation, the viscosity-solution rescue of the nonsmooth case, and the discrete-to-continuous limit that ties HJB to the Bellman operator.

Theorem 1 (LQR as the quadratic-value HJB solution). For $\dot{x}=Ax+Bu$, $L=\frac{1}{2}(x^{\mathsf{T}}Qx+u^{\mathsf{T}}Ru)$ with $Q⪰0$, $R\succ 0$, and $\phi =\frac{1}{2}{x}^{\mathsf{T}}Sx$ with $S⪰0$, the value function is the quadratic $V(x,t)=\frac{1}{2}{x}^{\mathsf{T}}P(t)x$ where $P(\cdot )$ solves the differential Riccati equation $-\dot{P}=Q+A^{\mathsf{T}}P+PA-PB{R}^{-1}{B}^{\mathsf{T}}P$, $P(T)=S$. The feedback minimiser is the linear law $u^{\star }(x,t)=-R^{-1}{B}^{\mathsf{T}}P(t)x$, and verification is immediate because the candidate is exactly $C^1$ (indeed $C^{\infty }$) wherever $P$ exists. This is the HJB-side derivation of the regulator of 44.07.02, complementary to its Pontryagin-side derivation via the Hamiltonian matrix \mathcal H = \begin{psmallmatrix}A & -BR^{-1}B^{\mathsf T}\\ -Q & -A^{\mathsf T}\end{psmallmatrix}; the Riccati solution is the value-function Hessian, $P(t)={\nabla }_{xx}V$ ^{[Bertsekas Vol. 1 Ch. 3]}.

Theorem 2 (relation to the classical Hamilton-Jacobi equation). When the running cost depends on the control only through $\dot{x}=u$ and $L=L(x,\dot{x},t)$ over an open control set, the minimisation in $\mathcal{H}(x,p,t)={\min }_v[L(x,v,t)+p^{\mathsf{T}}v]$ is the Legendre transform of $L$ in its velocity argument, and the HJB equation $-{\partial }_{t}V=\mathcal{H}(x,{\nabla }_{x}V,t)$ becomes the classical Hamilton-Jacobi equation ${\partial }_{t}V+H_{\mathrm{cl}}(x,{\nabla }_{x}V,t)=0$ of analytical mechanics 09.05.02, with $H_{\mathrm{cl}}=-\mathcal{H}={\max }_v[p^{\mathsf{T}}v-L]$ the Legendre-dual Hamiltonian. HJB is thus the controlled, minimisation-bearing generalisation of Hamilton-Jacobi: the action-as-value-function of mechanics is the no-constraint, single-Lagrangian special case of the optimal cost-to-go ^{[Liberzon Ch. 5]}.

Theorem 3 (viscosity characterisation and uniqueness). Under mild hypotheses (continuity and appropriate growth of $f,L,\phi$, with $U$ compact), the value function $V$ is the unique bounded uniformly continuous viscosity solution of the HJB equation with terminal data $V(\cdot ,T)=\phi$, in the sense of 02.18.05: $V$ is simultaneously a viscosity sub- and supersolution, tested against $C^1$ functions touching $V$ from above and below. The comparison principle for the Hamilton-Jacobi operator delivers uniqueness even where $V$ has corners, so the verification theorem extends: a continuous function is the value function if and only if it is the viscosity solution with the correct boundary condition. The classical $C^1$ verification theorem is the special case in which the viscosity solution happens to be differentiable ^{[Fleming-Soner Ch. 10]}.

Theorem 4 (HJB as the continuous limit of discrete dynamic programming). Discretising time into steps of length $h$ and replacing $\dot{x}=f$ by $x_{k+1}=x_k+hf(x_k,u_k,t_k)$ yields a finite-horizon discrete problem whose Bellman operator $(T_{h}J)(x)={\min }_{u\in U}[hL(x,u,t)+J(x+hf(x,u,t))]$ satisfies $(T_{h}J-J)/h\to {\partial }_{t}V+{\min }_u[L+{\nabla }_x{V}^{\mathsf{T}}f]$ as $h\to 0$ on smooth $J$. Thus the discrete dynamic-programming recursion $J_k=T_h{J}_{k+1}$ of 44.08.02 converges, under consistency and stability, to the HJB equation, and value iteration in discrete time is the explicit time-marching scheme for the HJB PDE. The discount factor $\alpha =e^{-\rho h}$ of the discrete contraction becomes a zeroth-order term $\rho V$ in the infinite-horizon discounted HJB equation $\rho V={\min }_u[L+{\nabla }_x{V}^{\mathsf{T}}f]$, the coercive term that restores well-posedness without a terminal condition ^{[Bardi-Capuzzo-Dolcetta Ch. 3]}.

The curse of dimensionality. The verification theorem is a certificate, not an algorithm: it presupposes a solution of a PDE on ${\mathbb{R}}^n\times [t_0,T]$. Grid-based solution of the HJB equation requires $O(N^n)$ points for $N$ nodes per axis, so the cost grows exponentially in the state dimension $n$ — Bellman's curse of dimensionality. Exact HJB solution is therefore feasible only for small $n$ or for structured cases (LQR, where the value is quadratic and the PDE collapses to the matrix Riccati equation) and motivates approximate dynamic programming, the maximum-principle's trajectory-based methods that avoid the full state-space sweep, and reinforcement-learning value-function approximation ^{[Bertsekas Vol. 1 Ch. 3]}.

Synthesis. The foundational reason continuous-time optimal control reduces to a single equation is that the dynamic-programming principle, applied over an infinitesimal step, collapses the optimisation over all control functions into the local HJB partial differential equation $-{\partial }_{t}V=\mathcal{H}(x,{\nabla }_{x}V,t)$ — the continuous-time image of the discrete fixed point $J^{\ast }=T{J}^{\ast }$ of 44.08.02, with the minimised Hamiltonian $\mathcal{H}$ the infinitesimal generator of the Bellman operator. This is dual to Pontryagin's maximum principle 44.07.01: the central insight is the identity $p={\nabla }_{x}V$ along the optimal trajectory, which makes the trajectory-side necessary conditions and the value-side sufficient conditions two readings of one problem, and the verification theorem is exactly the sufficiency direction the maximum principle cannot supply. The HJB equation generalises the classical Hamilton-Jacobi equation of 09.05.02 — the no-control, single-Lagrangian special case in which $\mathcal{H}$ becomes the Legendre transform and the action of mechanics is the cost-to-go stripped of its minimisation. Putting these together, the smooth verification theorem is the differentiable shadow of the viscosity-solution characterisation of 02.18.05, the precise sense in which the kinked value function is still the unique solution of HJB; the discount term $\rho V$ that makes the infinite-horizon HJB coercive is dual to the contraction modulus $\alpha$ of the discrete Bellman operator. This appears again in the LQR case 44.07.02, where all of it is exact: the value is quadratic, the PDE is the Riccati equation, and the curse of dimensionality is the one obstruction the certificate cannot remove.

Full proof set Master

Proposition 1 (dynamic-programming principle implies HJB for smooth $V$). If the value function $V$ is $C^1$, then $V$ satisfies $-{\partial }_{t}V(x,t)={\min }_{u\in U}[L(x,u,t)+{\nabla }_{x}V(x,t{)}^{\mathsf{T}}f(x,u,t)]$ with $V(\cdot ,T)=\phi$.

Proof. Fix $(x,t)$ and a constant control value $u\in U$ held over $[t,t+h]$, with trajectory $x_u(\cdot )$. The dynamic-programming principle gives $V(x,t)\le {\int }_t^{t+h}L(x_u(s),u,s)ds+V(x_u(t+h),t+h)$. Subtract $V(x,t)$, divide by $h>0$, and let $h\to 0^{+}$; by the $C^1$ chain rule $\frac{1}{h}[V(x_u(t+h),t+h)-V(x,t)]\to {\partial }_{t}V+{\nabla }_x{V}^{\mathsf{T}}f(x,u,t)$ and $\frac{1}{h}{\int }_t^{t+h}Lds\to L(x,u,t)$, yielding $0\le L(x,u,t)+{\partial }_{t}V+{\nabla }_x{V}^{\mathsf{T}}f(x,u,t)$ for every $u$, hence $-{\partial }_{t}V\le {\min }_u[L+{\nabla }_x{V}^{\mathsf{T}}f]$. For the reverse, choose near-optimal controls $u^{\epsilon }(\cdot )$ achieving $V(x,t)+\epsilon h\ge {\int }_t^{t+h}Lds+V(x^{\epsilon }(t+h),t+h)$; the same expansion and a continuity/selection argument as $h,\epsilon \to 0$ give $-{\partial }_{t}V\ge {\min }_u[L+{\nabla }_x{V}^{\mathsf{T}}f]$. Equality follows, and $V(\cdot ,T)=\phi$ is the definition. $\square$

Proposition 2 (verification / sufficiency). Let $W\in C^1$ solve the HJB equation with $W(\cdot ,T)=\phi$, with a measurable feedback minimiser $u^{\star }$ whose closed loop has solutions. Then $W=V$ and $u^{\star }$ is optimal.

Proof. For any admissible $u(\cdot )$ with trajectory $x(\cdot )$, $x(t)=x$, the chain rule gives $\frac{d}{ds}W(x(s),s)={\partial }_{t}W+{\nabla }_x{W}^{\mathsf{T}}f(x(s),u(s),s)$. The HJB equation gives ${\partial }_{t}W=-{\min }_v[L+{\nabla }_x{W}^{\mathsf{T}}f]\ge -[L(x(s),u(s),s)+{\nabla }_x{W}^{\mathsf{T}}f(x(s),u(s),s)]$, so $\frac{d}{ds}W(x(s),s)\ge -L(x(s),u(s),s)$. Integrating over $[t,T]$ and using $W(x(T),T)=\phi (x(T))$ gives $W(x,t)\le \phi (x(T))+{\int }_t^{T}Lds=J(x,t;u)$; infimising over $u$, $W\le V$. Running $u^{\star }$ makes every inequality an equality (the feedback attains the min), so $W(x,t)=J(x,t;u^{\star })\ge V(x,t)$. Hence $W=V$ and $u^{\star }$ is optimal. $\square$

Proposition 3 (costate-gradient identity). For $V\in C^2$ with interior feedback minimiser $u^{\star }$, the function $p(s):={\nabla }_{x}V(x^{\star }(s),s)$ along the optimal trajectory satisfies the adjoint equation $\dot{p}=-{\nabla }_{x}L-({\nabla }_{x}f{)}^{\mathsf{T}}p$ of 44.07.01.

Proof. Differentiating $p(s)={\nabla }_{x}V(x^{\star }(s),s)$ and using ${\dot{x}}^{\star }=f$ gives $\dot{p}={\nabla }_{xx}Vf+{\nabla }_x({\partial }_{t}V)$. Differentiating the HJB identity $-{\partial }_{t}V=L(x,u^{\star },t)+{\nabla }_x{V}^{\mathsf{T}}f(x,u^{\star },t)$ in $x$, the envelope theorem kills the $\partial u^{\star }/\partial x$ terms (because $u^{\star }$ minimises), leaving $-{\nabla }_x({\partial }_{t}V)={\nabla }_{x}L+{\nabla }_{xx}Vf+({\nabla }_{x}f{)}^{\mathsf{T}}{\nabla }_{x}V$. Substituting, the ${\nabla }_{xx}Vf$ terms cancel and $\dot{p}=-{\nabla }_{x}L-({\nabla }_{x}f{)}^{\mathsf{T}}p=-{\nabla }_x[L+p^{\mathsf{T}}f]$. $\square$

Proposition 4 (LQR value is quadratic and solves the Riccati equation). For the LQR data, $V(x,t)=\frac{1}{2}{x}^{\mathsf{T}}P(t)x$ with $P$ the solution of $-\dot{P}=Q+A^{\mathsf{T}}P+PA-PB{R}^{-1}{B}^{\mathsf{T}}P$, $P(T)=S$, and the optimal feedback is $u^{\star }=-R^{-1}{B}^{\mathsf{T}}Px$.

Proof. Insert $W(x,t)=\frac{1}{2}{x}^{\mathsf{T}}P(t)x$ ($P$ symmetric) into HJB. Then ${\nabla }_{x}W=Px$ and ${\partial }_{t}W=\frac{1}{2}{x}^{\mathsf{T}}\dot{P}x$. The inner problem ${\min }_u[\frac{1}{2}{x}^{\mathsf{T}}Qx+\frac{1}{2}{u}^{\mathsf{T}}Ru+x^{\mathsf{T}}P(Ax+Bu)]$ has, since $R\succ 0$, the unique minimiser $u^{\star }=-R^{-1}{B}^{\mathsf{T}}Px$ with minimum $\frac{1}{2}{x}^{\mathsf{T}}Qx+x^{\mathsf{T}}PAx-\frac{1}{2}{x}^{\mathsf{T}}PB{R}^{-1}{B}^{\mathsf{T}}Px$. The HJB equation $-\frac{1}{2}{x}^{\mathsf{T}}\dot{P}x$ equal to this for all $x$, after symmetrising $x^{\mathsf{T}}PAx=\frac{1}{2}{x}^{\mathsf{T}}(PA+A^{\mathsf{T}}P)x$, forces $-\dot{P}=Q+A^{\mathsf{T}}P+PA-PB{R}^{-1}{B}^{\mathsf{T}}P$ with $P(T)=S$ from $W(\cdot ,T)=\phi$. The candidate is $C^1$ where $P$ exists, so Proposition 2 certifies $W=V$ and $u^{\star }$ optimal. $\square$

Proposition 5 (the value function is a viscosity solution). Under continuity and boundedness of the data with $U$ compact, $V$ is a viscosity solution of the HJB equation in the sense of 02.18.05: for every $\varphi \in C^1$ and every interior local maximum (resp. minimum) point $(x_0,t_0)$ of $V-\varphi$, $-{\partial }_t\varphi (x_0,t_0)\le \mathcal{H}(x_0,{\nabla }_x\varphi (x_0,t_0),t_0)$ (resp. $\ge$).

Proof (subsolution direction). Let $\varphi \in C^1$ touch $V$ from above at $(x_0,t_0)$ with $V-\varphi$ having a local maximum there, normalised so $V(x_0,t_0)=\varphi (x_0,t_0)$ and $V\le \varphi$ nearby. Fix $u\in U$ constant and apply the dynamic-programming principle: $V(x_0,t_0)\le {\int }_{t_0}^{t_0+h}Lds+V(x_u(t_0+h),t_0+h)\le {\int }_{t_0}^{t_0+h}Lds+\varphi (x_u(t_0+h),t_0+h)$ for small $h>0$. Subtract $\varphi (x_0,t_0)=V(x_0,t_0)$, divide by $h$, let $h\to 0^{+}$: by the $C^1$ expansion of $\varphi$, $0\le L(x_0,u,t_0)+{\partial }_t\varphi (x_0,t_0)+{\nabla }_x\varphi (x_0,t_0{)}^{\mathsf{T}}f(x_0,u,t_0)$. Minimising over $u$ gives $-{\partial }_t\varphi (x_0,t_0)\le \mathcal{H}(x_0,{\nabla }_x\varphi ,t_0)$, the subsolution inequality. The supersolution direction uses near-optimal controls and a touch from below symmetrically. Uniqueness then follows from the comparison principle for this Hamilton-Jacobi operator 02.18.05. $\square$

Connections Master

• The infinite-horizon discounted dynamic programming of 44.08.02 is this unit's discrete-time origin: the discrete Bellman operator $(TJ)(x)={\min }_u[g+\alpha J(f)]$ becomes, under time-discretisation with step $h$ and discount $\alpha =e^{-\rho h}$, the differential operator of the HJB equation as $h\to 0$, and the unique-fixed-point statement $J^{\ast }=T{J}^{\ast }$ delivered there by the $\alpha$-contraction becomes the terminal-valued HJB solution $V$ here. Value iteration in discrete time is the explicit time-marching scheme for the HJB PDE, and the discount factor that supplied the contraction modulus reappears as the zeroth-order coercive term $\rho V$ in the stationary infinite-horizon equation.

• The classical Hamilton-Jacobi equation of 09.05.02 is the no-control special case of HJB: when there is no minimisation over an admissible set and the running cost is a single Lagrangian, the minimised Hamiltonian $\mathcal{H}(x,p,t)$ becomes the Legendre transform of $L$ and the HJB equation reduces to ${\partial }_{t}V+H_{\mathrm{cl}}(x,{\nabla }_{x}V,t)=0$, with Hamilton's principal function (the action) playing the role of the value function. Optimal control's cost-to-go is thus the controlled generalisation of mechanics' action, and the method of characteristics for the classical equation is the maximum-principle trajectory sweep seen from the value side.

• The viscosity-solution theory of 02.18.05 is what makes the verification theorem survive nonsmoothness: the value function is generically only Lipschitz, developing corners where optimal trajectories bifurcate, so the classical $C^1$ verification of this unit must be replaced by the statement that $V$ is the unique viscosity solution of HJB with the given boundary data. The comparison principle that delivers uniqueness for the Hamilton-Jacobi operator is the weak-solution substitute for the contraction uniqueness of the discrete theory, and the smooth verification theorem is exactly the case where the viscosity solution is differentiable.

• Pontryagin's maximum principle 44.07.01 is the necessary-condition dual of this unit's sufficient verification theorem: the costate is the value-function gradient, $p(s)={\nabla }_{x}V(x^{\star }(s),s)$, so the adjoint equation $\dot{p}=-{\nabla }_{x}H$ is the differentiated HJB equation along the optimal trajectory, and the pointwise Hamiltonian minimisation defining the optimal control is the same inner minimisation that defines the HJB Hamiltonian $\mathcal{H}$. Pontryagin reads the problem from the trajectory; HJB reads it from the value; the identity $p={\nabla }_{x}V$ is the bridge.

• The linear-quadratic regulator of 44.07.02 is the case where the entire HJB apparatus is exact and finite-dimensional: the value function is the quadratic $\frac{1}{2}{x}^{\mathsf{T}}P(t)x$, the HJB PDE collapses to the matrix differential Riccati equation, the value Hessian $P(t)={\nabla }_{xx}V$ is the Riccati matrix, and the optimal feedback $u^{\star }=-R^{-1}{B}^{\mathsf{T}}Px$ is the linear law. LQR is the standard demonstration that verification, when the value happens to be smooth, is a clean and complete optimality certificate immune to the curse of dimensionality.

Historical & philosophical context Master

The continuous-time dynamic-programming equation grew directly out of Richard Bellman's work of the 1950s on the principle of optimality, the same program from which the discrete Bellman equation of 44.08.02 descends ^{[Bellman 1957]}. Bellman recognised that the cost-to-go function of a continuous control problem should satisfy a first-order partial differential equation obtained by applying the principle of optimality over an infinitesimal time step, and that this equation specialises the Hamilton-Jacobi equation of nineteenth-century analytical mechanics — a connection that gives the equation its compound name. The verification theorem, the sufficiency statement that a smooth solution certifies optimality, was the natural complement to the necessity supplied by Pontryagin's contemporaneous maximum principle ^{[Pontryagin 1962]}.

For two decades the equation was treated formally, because the value function is generically not differentiable and no classical solution exists in the regimes where optimal trajectories cross. The resolution came in 1983 with the introduction of viscosity solutions by Michael Crandall and Pierre-Louis Lions, which gave a weak-solution concept under which the value function is the unique solution and the comparison principle holds ^{[Crandall-Lions 1983]}. Wendell Fleming and Halil Mete Soner's monograph and the parallel treatment by Martino Bardi and Italo Capuzzo-Dolcetta established the viscosity characterisation as the standard rigorous foundation, deterministic and stochastic, and connected it to the controlled-diffusion second-order HJB equation. Lions received the Fields Medal in 1994 in part for this work.

Bibliography Master

@book{Bellman1957DP,
  author    = {Bellman, Richard E.},
  title     = {Dynamic Programming},
  publisher = {Princeton University Press},
  address   = {Princeton, NJ},
  year      = {1957}
}

@book{FlemingSoner2006,
  author    = {Fleming, Wendell H. and Soner, Halil Mete},
  title     = {Controlled Markov Processes and Viscosity Solutions},
  edition   = {2nd},
  series    = {Stochastic Modelling and Applied Probability},
  volume    = {25},
  publisher = {Springer},
  address   = {New York},
  year      = {2006}
}

@book{Liberzon2012,
  author    = {Liberzon, Daniel},
  title     = {Calculus of Variations and Optimal Control Theory: A Concise Introduction},
  publisher = {Princeton University Press},
  address   = {Princeton, NJ},
  year      = {2012}
}

@book{BardiCapuzzoDolcetta1997,
  author    = {Bardi, Martino and Capuzzo-Dolcetta, Italo},
  title     = {Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations},
  series    = {Systems \& Control: Foundations \& Applications},
  publisher = {Birkh\"auser},
  address   = {Boston},
  year      = {1997}
}

@book{Bertsekas2017DPv1,
  author    = {Bertsekas, Dimitri P.},
  title     = {Dynamic Programming and Optimal Control, Volume I},
  edition   = {4th},
  publisher = {Athena Scientific},
  address   = {Belmont, MA},
  year      = {2017}
}

@article{CrandallLions1983,
  author  = {Crandall, Michael G. and Lions, Pierre-Louis},
  title   = {Viscosity Solutions of Hamilton-Jacobi Equations},
  journal = {Transactions of the American Mathematical Society},
  volume  = {277},
  number  = {1},
  year    = {1983},
  pages   = {1--42}
}

@book{Pontryagin1962,
  author    = {Pontryagin, Lev S. and Boltyanskii, Vladimir G. and Gamkrelidze, Revaz V. and Mishchenko, Evgenii F.},
  title     = {The Mathematical Theory of Optimal Processes},
  publisher = {Interscience Publishers (Wiley)},
  address   = {New York},
  year      = {1962}
}

@book{YongZhou1999,
  author    = {Yong, Jiongmin and Zhou, Xun Yu},
  title     = {Stochastic Controls: Hamiltonian Systems and HJB Equations},
  publisher = {Springer},
  address   = {New York},
  year      = {1999}
}