The Functional Delta Method and Hadamard Differentiability

Intuition Beginner

You already know the ordinary delta method: feed a single wobbling number into a smooth function, and the output wobbles by the input wobble times the slope at the true value. This unit asks the same question one size up. What if the thing you feed in is not a number but a whole curve — the data's estimated distribution — and the machine you feed it to turns curves into answers? A median is such a machine: hand it a distribution, it returns the middle value. A trimmed mean is another: hand it a distribution, it lops off the tails and averages the rest.

The estimated distribution wobbles around the true one; that wobble is the Brownian-bridge shape from the empirical-process unit. The question is how much the machine's answer wobbles in response. The answer is the same shape of rule: near the true distribution, a smooth machine acts like a fixed linear stretch, so the answer's wobble is that linear stretch applied to the curve's wobble. The slope of one number is replaced by a fixed linear rule that maps a whole wobble-curve to an output wobble.

The subtle part is what smooth should mean for a machine that eats curves. There are several candidate notions of derivative in this setting, and only one of them — the one mathematicians call Hadamard differentiability — is both weak enough to cover the median and quantile machines and strong enough to obey the chain rule that the proof needs. Picking the right notion of smoothness is the whole content.

The one-sentence takeaway: feed the wobbling empirical distribution into a suitably smooth curve-to-answer machine, and the answer wobbles by a fixed linear rule applied to the curve's Brownian-bridge wobble.

Visual Beginner

Picture two stacked panels. The top panel shows the true distribution curve and, jittering around it, the data's estimated curve — the wobble is the pinned-at-both-ends Brownian-bridge shape from the empirical-process unit. The bottom panel shows the machine's output: a single number (say the median) with its own little bell of uncertainty. An arrow carries the whole top wobble through a box labeled "smooth machine" and deposits it as the bottom wobble.

What you feed the machine	What the machine does	What comes out
A curve very close to the true one	Acts almost like a fixed linear rule	Output close to the true answer
The wobble (a whole random curve)	Applies that one fixed linear rule	A scaled, reshaped wobble on the answer
Wobble magnified by the square root of the sample size	Same linear rule, now on the magnified wobble	A stable, non-vanishing output wobble

The thing to watch is that the box applies the same linear rule no matter which particular wobble arrives. That single fixed rule is the functional's derivative. Because the incoming wobble is Gaussian (a Brownian bridge) and the rule is linear, the outgoing wobble is Gaussian too — which is why so many statistics end up approximately normal.

Worked example Beginner

We find how much the sample median wobbles, using the curve-to-answer picture. Take data with a smooth true distribution, and let the true median be the point $m$ where the curve crosses one half. The machine is: hand it a distribution, it returns the level where the curve hits one half.

Step 1. Name the wobble at the crossing. The estimated curve at the point $m$ sits a little above or below one half. Call that vertical gap the wobble of the curve at $m$. From the empirical-process unit, that gap has a known typical size that shrinks like one over the square root of the sample size.

Step 2. Turn a vertical gap into a horizontal shift. The median is where the curve hits one half. If the curve is shifted up by a small vertical gap near $m$, the crossing point slides sideways. How far sideways depends on how steeply the curve is rising at $m$ — its density value there, call it $f(m)$. A steep curve barely shifts; a flat curve shifts a lot.

Step 3. Read off the linear rule. A vertical gap of size $g$ near $m$ moves the crossing by about $g$ divided by the steepness $f(m)$. That division by $f(m)$ is the machine's fixed linear rule for the median.

Step 4. Plug in numbers. Suppose the true density at the median is $f(m)=0.4$, and the curve's vertical wobble at $m$ has typical size $0.05$. Then the median wobbles by about $0.05/0.4=0.125$.

Step 5. What this tells us. The median's wobble is the curve's wobble divided by the density at the median. The flatter the curve at the middle, the shakier the median — exactly why a median is unreliable when data piles up away from its center. The general functional delta method is this same move: a fixed linear rule carries the curve's wobble to the answer's wobble.

Check your understanding Beginner

Formal definition Intermediate+

Throughout, $\mathbb{D}$ and $\mathbb{E}$ are normed (typically Banach) spaces, $\varphi :{\mathbb{D}}_{\varphi }\subseteq \mathbb{D}\to \mathbb{E}$ a map defined on a subset ${\mathbb{D}}_{\varphi }$, and $\theta \in {\mathbb{D}}_{\varphi }$ a fixed point. The leading instance is $\mathbb{D}={\ell }^{\infty }(\mathcal{F})$ or $\mathbb{D}=D[-\infty ,\infty ]$ (cadlag functions under the uniform norm), $\theta =P$ or $\theta =F$ the true distribution, and ${\hat{\theta }}_n=P_n$ or $F_n$ the empirical distribution. Weak convergence $⇝$ in ${\ell }^{\infty }$-type spaces is the Hoffmann-Jorgensen outer-expectation notion of 45.05.03. The development follows van der Vaart ^{[van der Vaart 1998]} Ch. 20 and van der Vaart-Wellner ^{[van der Vaart Wellner 1996]} §3.9.

Definition (Gâteaux, Hadamard, Fréchet differentiability). Let ${\varphi }_{\theta }^{\prime }:\mathbb{D}\to \mathbb{E}$ be a continuous linear map. The map $\varphi$ is, at $\theta$:

• Gâteaux differentiable with derivative ${\varphi }_{\theta }^{\prime }$ if for each fixed $h\in \mathbb{D}$,

$$\frac{\varphi (\theta +th)-\varphi (\theta )}{t}\to {\varphi }_{\theta }^{\prime }(h)(t\to 0);$$

• Hadamard (or compactly) differentiable if the same limit holds uniformly over $h$ in compact sets, equivalently

$$\frac{\varphi (\theta +t{h}_t)-\varphi (\theta )}{t}\to {\varphi }_{\theta }^{\prime }(h)\text{whenever }{h}_t\to h\text{ and }t\to 0;$$

• Fréchet differentiable if $\Vert \varphi (\theta +h)-\varphi (\theta )-{\varphi }_{\theta }^{\prime }(h)\Vert =o(\Vert h\Vert )$ as $\Vert h\Vert \to 0$.

Fréchet implies Hadamard implies Gâteaux; the converses fail. Hadamard differentiability is the differentiability uniform over directions ranging in compact sets — strictly between the pointwise (Gâteaux) and the norm-uniform (Fréchet) notions.

Definition (Hadamard differentiability tangentially to a subspace). When $\varphi$ is defined only on a subset ${\mathbb{D}}_{\varphi }$ and the perturbations $h_t$ must stay inside it, $\varphi$ is Hadamard differentiable at $\theta$ tangentially to a subspace ${\mathbb{D}}_0\subseteq \mathbb{D}$ if there is a continuous linear ${\varphi }_{\theta }^{\prime }:{\mathbb{D}}_0\to \mathbb{E}$ with

$$\frac{\varphi (\theta +t{h}_t)-\varphi (\theta )}{t}\to {\varphi }_{\theta }^{\prime }(h)$$

for every $h\in {\mathbb{D}}_0$ and every sequence $h_t\to h$ with $h\in {\mathbb{D}}_0$ and $\theta +t{h}_t\in {\mathbb{D}}_{\varphi }$. The tangential restriction is essential: the empirical limit $\mathbb{G}$ is supported on the continuous functions ${\mathbb{D}}_0=UC(\mathcal{F},{\rho }_P)$ even though $\varphi$ acts on the larger ${\ell }^{\infty }(\mathcal{F})$, so the derivative need only exist along directions in the support of the limit.

Definition (statistical functional and the plug-in estimator). A statistical functional is a map $\varphi$ from a space of distributions to $\mathbb{E}$; its plug-in estimator is $\varphi ({\hat{\theta }}_n)$. The influence function of $\varphi$ at $\theta$ is the kernel ${\psi }_{\theta }$ representing the Gâteaux derivative along point-mass perturbations,

$${\psi }_{\theta }(x)=\underset{t\downarrow 0}{\lim }\frac{\varphi ((1-t)\theta +t{\delta }_x)-\varphi (\theta )}{t},$$

when it exists; for a Hadamard-differentiable $\varphi$ the derivative acts as ${\varphi }_{\theta }^{\prime }(h)=\int {\psi }_{\theta }dh$ on the centred directions $h$, identifying the derivative with integration against the influence function.

Why Hadamard is the right notion. Fréchet differentiability is too strong: the quantile and inverse-CDF maps, and even simple integration functionals on $D[0,1]$ under the supremum norm, fail to be Fréchet differentiable, because the remainder is not $o(\Vert h\Vert )$ uniformly over all small $h$. Gâteaux differentiability is too weak: it does not by itself yield a chain rule, and it does not transfer through weak convergence (the convergence is only directionwise, not uniform on the compacts where a tight weak limit concentrates). Hadamard differentiability is exactly the notion that (i) holds for the functionals statistics actually uses and (ii) satisfies a chain rule and a delta method, because a tight weak limit concentrates on $\sigma$-compact sets, precisely the sets over which Hadamard differentiability is uniform.

Key theorem with proof Intermediate+

Theorem (the functional delta method). Let $\mathbb{D},\mathbb{E}$ be normed spaces and $\varphi :{\mathbb{D}}_{\varphi }\subseteq \mathbb{D}\to \mathbb{E}$ be Hadamard differentiable at $\theta$ tangentially to a subspace ${\mathbb{D}}_0$, with derivative ${\varphi }_{\theta }^{\prime }:{\mathbb{D}}_0\to \mathbb{E}$. Let ${\hat{\theta }}_n$ be maps with values in ${\mathbb{D}}_{\varphi }$ such that $r_n({\hat{\theta }}_n-\theta )⇝G$ for some sequence $r_n\to \infty$, where $G$ is a tight random element taking values in ${\mathbb{D}}_0$. Then

$$r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))⇝{\varphi }_{\theta }^{\prime }(G),$$

and moreover $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))={\varphi }_{\theta }^{\prime }(r_n({\hat{\theta }}_n-\theta ))+o_{{\mathbb{P}}^{\ast }}(1)$.

Proof. Write $X_n=r_n({\hat{\theta }}_n-\theta )$, so $X_n⇝G$ with $G$ tight in ${\mathbb{D}}_0$. Define the map

$$g_n(h)=r_n(\varphi (\theta +r_n^{-1}h)-\varphi (\theta )),h\in {\mathbb{D}}_n:=\{h:\theta +r_n^{-1}h\in {\mathbb{D}}_{\varphi }\},$$

so that $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))=g_n(X_n)$. The content is that $g_n\to {\varphi }_{\theta }^{\prime }$ in the sense required to pass weak convergence through, namely: if $h_n\to h$ with $h\in {\mathbb{D}}_0$ and $h_n\in {\mathbb{D}}_n$, then $g_n(h_n)\to {\varphi }_{\theta }^{\prime }(h)$. With $t=r_n^{-1}\to 0$ this is the displayed defining limit of Hadamard differentiability tangentially to ${\mathbb{D}}_0$, applied to the sequence $h_n\to h$.

The transfer is the extended continuous-mapping theorem of 45.05.03: if maps $g_n:{\mathbb{D}}_n\to \mathbb{E}$ satisfy $g_n(h_n)\to g(h)$ for every $h_n\to h$ with $h$ in the support of the tight limit $G$, and $X_n⇝G$, then $g_n(X_n)⇝g(G)$. Here $g={\varphi }_{\theta }^{\prime }$ and the support of $G$ lies in ${\mathbb{D}}_0$ where ${\varphi }_{\theta }^{\prime }$ is defined, continuous, and linear. Apply it: $g_n(X_n)⇝{\varphi }_{\theta }^{\prime }(G)$, which is the claimed weak convergence.

For the asymptotic-linearity remainder, the same Hadamard limit gives $g_n(X_n)-{\varphi }_{\theta }^{\prime }(X_n)\to 0$ along every realisation with $X_n\to h\in {\mathbb{D}}_0$; since ${\varphi }_{\theta }^{\prime }$ is continuous and linear, ${\varphi }_{\theta }^{\prime }(X_n)={\varphi }_{\theta }^{\prime }(r_n({\hat{\theta }}_n-\theta ))$, and the difference $g_n(X_n)-{\varphi }_{\theta }^{\prime }(X_n)$ converges to $0$ in outer probability by the portmanteau/continuous-mapping argument applied to the joint convergence $(X_n,g_n(X_n))⇝(G,{\varphi }_{\theta }^{\prime }(G))$. Hence $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))={\varphi }_{\theta }^{\prime }(r_n({\hat{\theta }}_n-\theta ))+o_{{\mathbb{P}}^{\ast }}(1)$. When $G$ is Gaussian and $\mathbb{E}=\mathbb{R}$, the limit ${\varphi }_{\theta }^{\prime }(G)$ is a continuous linear image of a Gaussian, hence Gaussian. $\square$

Bridge. This theorem builds toward every $\sqrt{n}$-limit law for a nonparametric statistic in the chapter, and the same Hadamard-plus-weak-convergence mechanism appears again in the delta method for the bootstrap, where the bootstrapped empirical process replaces $X_n$ and the identical derivative ${\varphi }_{\theta }^{\prime }$ transfers its conditional weak limit. The foundational reason Hadamard differentiability is the load-bearing hypothesis is that a tight weak limit $G$ concentrates on $\sigma$-compact subsets of ${\mathbb{D}}_0$, and Hadamard differentiability is exactly differentiability made uniform over compacts — this is exactly the matching of the regularity of $\varphi$ to the concentration of $G$ that the extended continuous-mapping theorem needs. This generalises the ordinary delta method 45.04.02 from ${\mathbb{R}}^k$ to function spaces: there $\mathbb{D}={\mathbb{R}}^k$, every notion of differentiability coincides (compact sets are bounded-and-closed), the derivative is the Jacobian $\nabla g(\theta )$, and $G=N(0,\Sigma )$; here $\mathbb{D}$ is infinite-dimensional, the three notions genuinely differ, the derivative is a continuous linear operator, and $G$ is the $P$-Brownian bridge $\mathbb{G}$ of 45.05.03. Putting these together, the central insight is that the Donsker weak limit ${\mathbb{G}}_n⇝\mathbb{G}$ supplies the input and Hadamard differentiability supplies the transfer, so a single weak limit on ${\ell }^{\infty }(\mathcal{F})$ delivers the asymptotics of every smooth functional of the data at once; the bridge is the passage from a limit law for the whole empirical distribution to a limit law for any quantity computed from it.

Exercises Intermediate+

Advanced results Master

Theorem 1 (the functional delta method, sharp form). Let $\varphi :{\mathbb{D}}_{\varphi }\subseteq \mathbb{D}\to \mathbb{E}$ be Hadamard differentiable at $\theta$ tangentially to ${\mathbb{D}}_0$, and let $r_n({\hat{\theta }}_n-\theta )⇝G$ with $G$ tight and supported in ${\mathbb{D}}_0$. Then $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))⇝{\varphi }_{\theta }^{\prime }(G)$ with the asymptotic representation $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))={\varphi }_{\theta }^{\prime }(r_n({\hat{\theta }}_n-\theta ))+o_{{\mathbb{P}}^{\ast }}(1)$ ^{[van der Vaart Wellner 1996]}. The Hadamard hypothesis cannot be weakened to Gâteaux: there are Gâteaux-differentiable functionals for which the conclusion fails, because the directionwise limit does not survive the passage to the tight weak limit, which concentrates on compacts and demands the uniformity that Hadamard supplies. With $G=\mathbb{G}$ the $P$-Brownian bridge of 45.05.03 and $\mathbb{E}=\mathbb{R}$, the limit ${\varphi }_{\theta }^{\prime }(\mathbb{G})$ is centred Gaussian, identified through the influence function ${\psi }_{\theta }$ as $\int {\psi }_{\theta }d\mathbb{G}$ with variance ${\mathrm{Var}}_P({\psi }_{\theta })=P{\psi }_{\theta }^2-(P{\psi }_{\theta }{)}^2$.

Theorem 2 (the quantile process). If $F$ has a continuous positive density $f$ on the interval $\{0<F<1\}$, the empirical quantile function $F_n^{-1}$ satisfies, as a process in $p\in [a,b]\subset (0,1)$,

$$\sqrt{n}(F_n^{-1}(p)-F^{-1}(p))⇝-\frac{\mathbb{G}(F^{-1}(p))}{f(F^{-1}(p))}=\frac{\mathbb{B}(p)}{f(F^{-1}(p))},$$

where $\mathbb{G}$ is the $F$-Brownian bridge and $\mathbb{B}$ the standard bridge on $[0,1]$, weak convergence holding in ${\ell }^{\infty }[a,b]$ ^{[van der Vaart 1998]}. The quantile process is thus the empirical-process bridge run through the inverse-map Hadamard derivative $h\mapsto -h\circ F^{-1}/(f\circ F^{-1})$, which is the function-valued generalisation of the Beginner median computation. The endpoints $p\to 0,1$ are excluded because $f\circ F^{-1}\to 0$ there inflates the variance, the analytic source of the difficulty of extreme-quantile estimation.

Theorem 3 (copula functionals, brief). For a bivariate distribution $H$ with continuous margins $F,G$ and copula $C$ defined by $H(x,y)=C(F(x),G(y))$, the empirical copula $C_n$ is a Hadamard-differentiable functional of the empirical joint and marginal distributions, provided the partial derivatives ${\partial }_{1}C,{\partial }_{2}C$ exist and are continuous ^{[van der Vaart Wellner 1996]}. The resulting empirical copula process $\sqrt{n}(C_n-C)⇝{\mathbb{G}}_C$ is a Gaussian process whose covariance combines the bivariate bridge with correction terms from estimating the margins, the corrections being the images of the marginal bridges under ${\partial }_{1}C,{\partial }_{2}C$. This is the delta-method backbone of nonparametric dependence inference.

Theorem 4 (the delta method for the bootstrap). Let $\varphi$ be Hadamard differentiable at $\theta$ tangentially to ${\mathbb{D}}_0$ with a derivative that is continuous on all of ${\mathbb{D}}_0$ (not merely defined there). Suppose the bootstrap is consistent for ${\hat{\theta }}_n$, meaning the conditional law of $r_n({\hat{\theta }}_n^{\ast }-{\hat{\theta }}_n)$ given the data converges weakly to that of $G$, in outer probability. Then the bootstrap is consistent for $\varphi ({\hat{\theta }}_n)$:

$$\underset{h\in B{L}_1}{\sup }\left|\mathbb{E}\right[h(r_n(\varphi ({\hat{\theta }}_n^{\ast })-\varphi ({\hat{\theta }}_n)))\left|\text{data}\right]-\mathbb{E}h({\varphi }_{\theta }^{\prime }(G))|\stackrel{{\mathbb{P}}^{\ast }}{\to }0,$$

the supremum over $1$-Lipschitz functions bounded by $1$ ^{[van der Vaart Wellner 1996]}. The continuity of the derivative on the whole tangent space (not just at $\theta$) is the extra hypothesis the bootstrap needs beyond the ordinary delta method, because the bootstrap perturbs around the random ${\hat{\theta }}_n$ rather than the fixed $\theta$. This is the rigorous justification of the universal recipe "to get a confidence interval for $\varphi (F)$, bootstrap $\varphi (F_n^{\ast })$," and it cross-refers the bootstrap-consistency theory of the bootstrap unit.

Theorem 5 (von Mises expansion and the influence function). A Hadamard-differentiable functional admits the von Mises expansion

$$\varphi (F_n)=\varphi (F)+\int {\psi }_{F}d(F_n-F)+{\mathrm{Rem}}_n=\varphi (F)+\frac{1}{n}\sum _{k=1}^n{\psi }_F(X_k)+o_{\mathbb{P}}(n^{-1/2}),$$

with ${\psi }_F$ the influence function (the Gâteaux derivative along point masses, centred so $P{\psi }_F=0$) and ${\mathrm{Rem}}_n=o_{\mathbb{P}}(n^{-1/2})$ ^{[von Mises 1947]}. The leading term is an average of i.i.d. mean-zero summands, so the central limit theorem gives $\sqrt{n}(\varphi (F_n)-\varphi (F))⇝N(0,{\mathrm{Var}}_P{\psi }_F)$ directly. The influence function is simultaneously the derivative kernel, the asymptotic-variance generator, and (in robust statistics) the measure of an observation's leverage; its boundedness is the robustness criterion, which is why trimming and Winsorising — which bound ${\psi }_F$ — buy robustness at a controlled efficiency cost.

Synthesis. The functional delta method is the organising theorem of nonparametric large-sample theory, and the foundational reason it works is that two ingredients meet exactly: the Donsker weak limit ${\mathbb{G}}_n⇝\mathbb{G}$ of 45.05.03 provides a tight Gaussian input concentrated on a $\sigma$-compact subspace, and Hadamard differentiability provides regularity uniform over precisely those compacts, so the extended continuous-mapping theorem transfers the limit through $\varphi$. This is exactly the matching that fails for the two neighbouring notions of derivative: Fréchet is too strong and excludes the quantile, inverse, and integration maps that statistics lives on, while Gâteaux is too weak and breaks the chain rule and the transfer; Hadamard is the unique notion that both holds for the working functionals and obeys the calculus. The method generalises the ordinary delta method 45.04.02 from ${\mathbb{R}}^k$ to function spaces, and the central insight is that the influence function ${\psi }_F$ is at once the derivative kernel, the asymptotic-variance generator via ${\mathrm{Var}}_P{\psi }_F$, and the robustness diagnostic, so a single object controls limit law, efficiency, and stability. Putting these together, the same derivative ${\varphi }_{\theta }^{\prime }$ that gives the limit law also certifies the bootstrap (Theorem 4), so quantiles, trimmed means, the Mann-Whitney functional, copulas, and $M$-estimators all acquire $\sqrt{n}$-Gaussian limits and valid resampling-based confidence sets in one stroke; the method is dual to the von Mises expansion (Theorem 5), which packages the same first-order linearisation as an i.i.d. average plus a negligible remainder.

Full proof set Master

The functional delta method, the chain rule, the mean functional, the quantile derivative, the trimmed-mean and Mann-Whitney functionals, and the $M$-functional derivative are proved in the Key theorem and Exercises. The remaining Master claims are recorded here.

Proposition 1 (Hadamard differentiability of the inverse/quantile map, uniform version). Let $F$ be a distribution with continuous positive density $f$ on an interval where $0<F<1$, and let ${\mathbb{D}}_0$ be the functions $h$ continuous on $\{F^{-1}(p):p\in [a,b]\}$. Then $\varphi :F\mapsto F^{-1}{|}_{[a,b]}$, as a map into ${\ell }^{\infty }[a,b]$, is Hadamard differentiable at $F$ tangentially to ${\mathbb{D}}_0$ with ${\varphi }_F^{\prime }(h)(p)=-h(F^{-1}(p))/f(F^{-1}(p))$.

Proof. Fix $h_t\to h$ uniformly with $h\in {\mathbb{D}}_0$, and set $F_t=F+t{h}_t$, ${\xi }_t(p)=F_t^{-1}(p)$, $\xi (p)=F^{-1}(p)$. The defining relations are $F(\xi (p))=p$ and $F_t({\xi }_t(p))=p$ (using continuity of $F$ on the relevant range). Subtract and use $F_t=F+t{h}_t$:

$$0=F({\xi }_t(p))-F(\xi (p))+t{h}_t({\xi }_t(p)).$$

By the mean value theorem $F({\xi }_t(p))-F(\xi (p))=f({\zeta }_{t,p})({\xi }_t(p)-\xi (p))$ for ${\zeta }_{t,p}$ between ${\xi }_t(p)$ and $\xi (p)$. Uniform convergence ${\xi }_t\to \xi$ on $[a,b]$ holds because $f$ is bounded below by $c>0$ on the compact range, giving $|{\xi }_t(p)-\xi (p)|\le t\Vert h_t{\Vert }_{\infty }/c\to 0$ uniformly. Hence ${\zeta }_{t,p}\to \xi (p)$ uniformly, $f({\zeta }_{t,p})\to f(\xi (p))$ by continuity, and $h_t({\xi }_t(p))\to h(\xi (p))$ by joint convergence of $h_t\to h$ and ${\xi }_t\to \xi$ with $h$ continuous. Dividing by $tf({\zeta }_{t,p})$,

$$\frac{{\xi }_t(p)-\xi (p)}{t}=-\frac{h_t({\xi }_t(p))}{f({\zeta }_{t,p})}⟶-\frac{h(\xi (p))}{f(\xi (p))}$$

uniformly in $p\in [a,b]$. The convergence is in the sup-norm of ${\ell }^{\infty }[a,b]$, the limit is linear in $h$ and continuous (division by the bounded-below $f\circ \xi$), so $\varphi$ is Hadamard differentiable tangentially to ${\mathbb{D}}_0$ with the stated derivative. Composing with the input $\sqrt{n}(F_n-F)⇝\mathbb{G}$ proves Theorem 2. $\square$

Proposition 2 (Gâteaux is insufficient: a delta-method counterexample). There is a functional $\varphi$ that is Gâteaux differentiable at $\theta$ with continuous linear derivative ${\varphi }_{\theta }^{\prime }$, and inputs $r_n({\hat{\theta }}_n-\theta )⇝G$, for which $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))⇝{\varphi }_{\theta }^{\prime }(G)$ fails.

Proof. Work in $\mathbb{D}={\ell }^{\infty }$-type direction by taking $\mathbb{D}={\mathbb{R}}^2$ with a non-uniform structure mimicked through a sequence: let $\varphi (x,y)=x^3/(x^2+y^2)$ for $(x,y)\ne 0$ and $\varphi (0,0)=0$. Along every fixed line $(x,y)=t(u,v)$, $\varphi (tu,tv)=t{u}^3/(u^2+v^2)$, so the Gâteaux derivative at $0$ exists and equals the linear map ${\varphi }_0^{\prime }(u,v)=u^3/(u^2+v^2)$ — but this is not linear in $(u,v)$, exhibiting the standard failure: Gâteaux derivatives need not be linear. Replacing by a genuinely linear-Gâteaux but non-Hadamard example, take $\varphi (x,y)=x$ if $y=x^2$ and $\varphi (x,y)=0$ otherwise; then $\varphi$ has zero Gâteaux derivative at $0$ along every straight line, yet for the curved approach $(t{h}_t)=(t,t^2)$ with $h_t\to (1,0)$ the difference quotient is $\varphi (t,t^2)/t=1\nrightarrow {\varphi }_0^{\prime }(1,0)=0$. So $\varphi$ is not Hadamard differentiable, and feeding a sequence ${\hat{\theta }}_n-\theta$ that approaches along such a parabola — which a tight non-degenerate $G$ assigns positive mass near — makes $r_n(\varphi ({\hat{\theta }}_n)-\varphi (\theta ))$ fail to converge to ${\varphi }_0^{\prime }(G)=0$. The obstruction is exactly the lack of uniformity over converging direction-sequences that defines Hadamard differentiability. $\square$

Proposition 3 (the bootstrap delta method requires derivative continuity on the tangent space). Under the hypotheses of Theorem 4 — Hadamard differentiability with derivative continuous on ${\mathbb{D}}_0$ and bootstrap consistency for ${\hat{\theta }}_n$ — the bootstrap is consistent for $\varphi ({\hat{\theta }}_n)$, and the continuity of ${\varphi }_{\theta }^{\prime }$ on ${\mathbb{D}}_0$ is not removable.

Proof. By bootstrap consistency for ${\hat{\theta }}_n$, the conditional law of $X_n^{\ast }=r_n({\hat{\theta }}_n^{\ast }-{\hat{\theta }}_n)$ given the data converges to that of $G$ in outer probability (in the bounded-Lipschitz metric). The bootstrap version of the delta method linearises around the random centre ${\hat{\theta }}_n$ rather than $\theta$: a version of Hadamard differentiability uniform in a neighbourhood (supplied by continuity of ${\varphi }_{\cdot }^{\prime }$ on the tangent space) gives $r_n(\varphi ({\hat{\theta }}_n^{\ast })-\varphi ({\hat{\theta }}_n))={\varphi }_{\theta }^{\prime }(X_n^{\ast })+o_{{\mathbb{P}}^{\ast }}(1)$ conditionally, where the remainder is negligible because the centre ${\hat{\theta }}_n\to \theta$ and the derivative varies continuously. Applying the conditional continuous-mapping theorem to the continuous linear ${\varphi }_{\theta }^{\prime }$ transfers the conditional weak limit: the conditional law of ${\varphi }_{\theta }^{\prime }(X_n^{\ast })$ converges to that of ${\varphi }_{\theta }^{\prime }(G)$ in outer probability, which is the displayed bounded-Lipschitz statement. Continuity of ${\varphi }_{\theta }^{\prime }$ across ${\mathbb{D}}_0$ is needed because the bootstrap evaluates the derivative effectively at the moving point ${\hat{\theta }}_n$; a derivative defined only at $\theta$ leaves the linearisation around ${\hat{\theta }}_n$ uncontrolled, and the bootstrap can then be inconsistent even when the ordinary delta method holds, as occurs for non-smooth functionals such as $|\theta |$ at $0$. $\square$

Proposition 4 (von Mises expansion has a negligible remainder for Hadamard functionals). If $\varphi$ is Hadamard differentiable at $F$ tangentially to a subspace containing the support of $\mathbb{G}$, then $\varphi (F_n)=\varphi (F)+\frac{1}{n}{\sum }_k{\psi }_F(X_k)+o_{\mathbb{P}}(n^{-1/2})$ with ${\psi }_F$ the influence function and $P{\psi }_F=0$.

Proof. The functional delta-method representation (Key theorem) gives $\sqrt{n}(\varphi (F_n)-\varphi (F))={\varphi }_F^{\prime }(\sqrt{n}(F_n-F))+o_{{\mathbb{P}}^{\ast }}(1)$. The derivative acts as integration against the influence function, ${\varphi }_F^{\prime }(h)=\int {\psi }_{F}dh$, so ${\varphi }_F^{\prime }(\sqrt{n}(F_n-F))=\sqrt{n}\int {\psi }_{F}d(F_n-F)=\sqrt{n}(\frac{1}{n}{\sum }_k{\psi }_F(X_k)-P{\psi }_F)$. Centring of the influence function, $P{\psi }_F=0$ (the derivative annihilates the constant direction, since $\varphi (F+t\cdot 0)=\varphi (F)$), reduces this to $\frac{1}{\sqrt{n}}{\sum }_k{\psi }_F(X_k)$. Dividing the whole representation by $\sqrt{n}$ yields $\varphi (F_n)-\varphi (F)=\frac{1}{n}{\sum }_k{\psi }_F(X_k)+o_{\mathbb{P}}(n^{-1/2})$, the von Mises expansion. The leading term is a centred i.i.d. average; the central limit theorem then gives the Gaussian limit with variance ${\mathrm{Var}}_P{\psi }_F$, recovering Theorem 5 and tying the functional delta method to the influence-function calculus of robust statistics. $\square$

Connections Master

Donsker classes and the empirical-process weak limit 45.05.03 supply the indispensable input: the functional delta method is a transfer theorem, and the object it transfers is exactly the Donsker convergence ${\mathbb{G}}_n=\sqrt{n}(P_n-P)⇝\mathbb{G}$ in ${\ell }^{\infty }(\mathcal{F})$. The tightness of $\mathbb{G}$ on a $\sigma$-compact tangent subspace is what licenses the extended continuous-mapping step, so this unit is the consumer of the weak limit that unit produces.

The delta method and the second-order delta method 45.04.02 is the finite-dimensional special case: when $\mathbb{D}={\mathbb{R}}^k$ the unit ball is compact, the three notions of differentiability coincide, the Hadamard derivative is the Jacobian, and the functional delta method collapses to the ordinary delta method with sandwich covariance $\nabla g\Sigma \nabla g^{\mathsf{T}}$. This unit is the infinite-dimensional lift of that result.

The bootstrap and its consistency 45.04.10 is where Theorem 4 lives: the delta method for the bootstrap shows that whenever a functional is Hadamard differentiable with a derivative continuous on the tangent space and the bootstrap is consistent for the empirical measure, the bootstrap is automatically consistent for the plug-in functional. This is the theoretical license behind resampling confidence bands for quantiles, trimmed means, and copula functionals, and it is exactly the smoothness condition whose failure (at non-differentiable functionals) explains documented bootstrap inconsistencies.

$U$-statistics and their limit theory 45.05.05 meet this unit at the Mann-Whitney/Wilcoxon functional of Exercise 6: a degree-two $U$-statistic and the corresponding two-sample functional admit the same Gaussian limit, reached either by Hoeffding projection or by the functional delta method, and the influence function of the functional is the projection kernel of the $U$-statistic.

M-estimators and Z-estimators 45.04.04 are recovered through Exercise 8: an estimator defined as the zero of an estimating equation is a Hadamard-differentiable functional of the empirical measure, and its influence function $V^{-1}{\psi }_{\theta (F)}$ and sandwich variance fall out of the implicit-function differentiation, an alternative to the direct stochastic expansion of the score that unit uses.

Historical & philosophical context Master

The idea that a statistic should be viewed as a functional of the underlying distribution, and that its asymptotics should follow from differentiating that functional, originates with Richard von Mises's 1947 Annals paper ^{[von Mises 1947]}, which introduced the expansion now bearing his name and treated "differentiable statistical functions" by a Taylor development in the distribution. Von Mises worked with what is essentially a Gâteaux derivative and a remainder controlled by hand, and the programme stalled on the question of which notion of derivative makes the remainder genuinely negligible.

The resolution that compact (Hadamard) differentiability is the correct notion was reached in the 1970s, principally in James Reeds's 1976 Harvard thesis ^{[Reeds 1976]}, which argued that Fréchet differentiability is too strong for the quantile and related maps while Gâteaux is too weak for a chain rule, and that Hadamard differentiability threads the needle. Luisa Fernholz's 1983 lecture notes ^{[Fernholz 1983]} systematised the von Mises calculus around the Hadamard derivative and the influence function. Richard Gill's 1989 paper ^{[Gill 1989]} extended the compact-differentiability method to the product-integral and survival-analysis functionals central to event-history analysis, and the synthesis in van der Vaart and Wellner's 1996 monograph ^{[van der Vaart Wellner 1996]} and van der Vaart's 1998 textbook ^{[van der Vaart 1998]} fixed the modern statement, including the delta method for the bootstrap. The influence function, introduced by Frank Hampel in the robustness literature as the derivative of a functional along point-mass contamination, coincides with the Hadamard derivative kernel, unifying the robustness and asymptotic-efficiency readings of the same object.

Bibliography Master

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{vanderVaartWellner1996,
  author    = {van der Vaart, Aad W. and Wellner, Jon A.},
  title     = {Weak Convergence and Empirical Processes: With Applications to Statistics},
  publisher = {Springer-Verlag},
  series    = {Springer Series in Statistics},
  address   = {New York},
  year      = {1996}
}

@article{vonMises1947,
  author  = {von Mises, Richard},
  title   = {On the asymptotic distribution of differentiable statistical functions},
  journal = {Annals of Mathematical Statistics},
  volume  = {18},
  number  = {3},
  year    = {1947},
  pages   = {309--348}
}

@phdthesis{Reeds1976,
  author = {Reeds, James A.},
  title  = {On the Definition of von Mises Functionals},
  school = {Harvard University},
  year   = {1976}
}

@book{Fernholz1983,
  author    = {Fernholz, Luisa T.},
  title     = {von Mises Calculus for Statistical Functionals},
  publisher = {Springer-Verlag},
  series    = {Lecture Notes in Statistics},
  volume    = {19},
  address   = {New York},
  year      = {1983}
}

@article{Gill1989,
  author  = {Gill, Richard D.},
  title   = {Non- and semi-parametric maximum likelihood estimators and the von Mises method (Part 1)},
  journal = {Scandinavian Journal of Statistics},
  volume  = {16},
  number  = {2},
  year    = {1989},
  pages   = {97--128}
}

@article{Hampel1974,
  author  = {Hampel, Frank R.},
  title   = {The influence curve and its role in robust estimation},
  journal = {Journal of the American Statistical Association},
  volume  = {69},
  number  = {346},
  year    = {1974},
  pages   = {383--393}
}