The Delta Method and the Second-Order Delta Method

Intuition Beginner

Suppose you have a good estimate of some quantity — the average of a sample, say — and you know roughly how much it wobbles from sample to sample. Now you do not actually want that quantity; you want some function of it. Maybe you measured an average rate but you report its logarithm, or you measured a proportion but you report the odds. The question is simple: if your raw estimate wobbles by a known amount, how much does the transformed estimate wobble?

The answer is the single most useful rule of thumb in large-sample statistics. Near the true value, any smooth function looks like a straight line: zoom in close enough and a curve is indistinguishable from its tangent. A straight line just stretches or shrinks distances by a fixed factor — its slope. So a small wobble in the raw estimate becomes a wobble in the transformed estimate that is larger or smaller by exactly that slope factor. If the slope at the true value is $2$, the transformed estimate wobbles twice as much; if the slope is one half, it wobbles half as much.

This is the delta method. It says: to find the spread of a function of an estimate, multiply the spread of the estimate by the slope of the function at the true value. It is how every standard error of a transformed quantity gets computed.

The one-sentence takeaway: a smooth function near the truth acts like its tangent line, so the wobble of a transformed estimate is the wobble of the original times the slope of the function.

Visual Beginner

Picture the graph of a smooth function, a gentle curve. Mark the true value on the horizontal axis and the matching point on the curve. Draw the tangent line touching the curve there. Now imagine a small bell-shaped spread of possible estimates sitting on the horizontal axis, centered on the true value. Push that spread up to the curve. Because the curve is nearly straight over this small region, the spread arrives on the vertical axis still bell-shaped, but stretched or squeezed by the steepness of the tangent.

Slope of the function at the truth	What happens to the spread	Resulting wobble
Steep (large slope)	Stretched a lot	Transformed estimate wobbles more
Gentle (small slope)	Squeezed	Transformed estimate wobbles less
Slope near zero (flat)	Almost no first-order stretch	The straight-line rule breaks down

The last row is the warning. When the tangent is flat, the function barely moves at first order, and the simple multiply-by-the-slope rule gives zero spread. Something subtler — the curvature, the bend of the graph — then takes over. That refinement is the second-order delta method.

Worked example Beginner

A factory line produces items, and an unknown fraction $p$ are defective. From a large sample you estimate $p$, and your estimate has a known wobble. But you want to report not $p$ but the odds of a defect, the function $g(p)=p/(1-p)$. How much does the odds estimate wobble?

Step 1. The recipe. Use the delta method: the wobble of $g$ of the estimate equals the wobble of the estimate times the slope of $g$ at the true $p$.

Step 2. The slope. For $g(p)=p/(1-p)$, the slope at $p$ is $1/(1-p{)}^2$. (This is the derivative; you can read it as how fast the odds climb as $p$ climbs.)

Step 3. Plug in a number. Suppose the true defect fraction is $p=0.2$. Then the slope is $1/(1-0.2{)}^2=1/(0.8{)}^2=1/0.64=1.5625$.

Step 4. Carry the wobble across. Suppose your estimate of $p$ has a standard wobble (a standard error) of $0.01$. Then the odds estimate has standard wobble about $0.01\times 1.5625=0.0156$.

Step 5. What this tells us. The odds estimate at $p=0.2$ wobbles about $1.56$ times as much as the raw proportion estimate, because the odds function is climbing steeply there. Had we worked at $p=0.5$, the slope would have been $1/(0.5{)}^2=4$, so the odds estimate would have wobbled four times as much. The delta method turns a known wobble on the raw scale into a wobble on any smooth transformed scale, just by reading off one slope.

Check your understanding Beginner

Formal definition Intermediate+

Throughout, $T_n$ is an estimator sequence in ${\mathbb{R}}^k$, $\theta \in {\mathbb{R}}^k$ a fixed parameter, $\Sigma$ a fixed covariance matrix, and $\stackrel{d}{\to }$, $\stackrel{\mathbb{P}}{\to }$ denote convergence in distribution and in probability as set up in 45.04.01. The driving hypothesis is asymptotic normality at rate $\sqrt{n}$: there is a sequence $r_n\to \infty$ (taken as $r_n=\sqrt{n}$ unless stated otherwise) with

$$r_n(T_n-\theta )\stackrel{d}{\to }N(0,\Sigma ).$$

The central limit theorem 37.03.02 is the canonical source of such a statement, with $T_n={\bar{X}}_n$, $\theta =\mathbb{E}{X}_1$, and $\Sigma =\mathrm{Cov}(X_1)$.

Definition (differentiability used by the delta method). A map $g:{\mathbb{R}}^k\to {\mathbb{R}}^m$ is differentiable at $\theta$ if there is a linear map $g_{\theta }^{\prime }:{\mathbb{R}}^k\to {\mathbb{R}}^m$, represented by the Jacobian matrix $\nabla g(\theta )\in {\mathbb{R}}^{m\times k}$ with entries $\partial g_i/\partial {\theta }_j$, such that

$$g(\theta +h)-g(\theta )=\nabla g(\theta )h+o(|h|)(h\to 0).$$

In the scalar case $k=m=1$ the Jacobian is the ordinary derivative $g^{\prime }(\theta )$. The little-$o$ remainder is deterministic here; the delta method converts it into a stochastic $o_{\mathbb{P}}$ remainder via the boundedness in probability of $r_n(T_n-\theta )$.

Definition (first-order delta method, the statement). Under $r_n(T_n-\theta )\stackrel{d}{\to }N(0,\Sigma )$ and differentiability of $g$ at $\theta$,

$$r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }N(0,\nabla g(\theta )\Sigma \nabla g(\theta {)}^{\mathsf{T}}).$$

The limiting covariance $\nabla g(\theta )\Sigma \nabla g(\theta {)}^{\mathsf{T}}\in {\mathbb{R}}^{m\times m}$ is the sandwich (or delta-method) covariance. In the scalar case it reduces to $g^{\prime }(\theta {)}^2{\sigma }^2$.

Definition (second-order delta method, the statement). Suppose $k=m=1$, $g$ is twice differentiable at $\theta$, and the first derivative vanishes, $g^{\prime }(\theta )=0$ but $g^{\prime \prime }(\theta )\ne 0$. If $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2)$, then the first-order centering is degenerate and the correct rate is $n$:

$$n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{g}^{\prime \prime }(\theta ){\sigma }^2{\chi }_1^2,$$

where ${\chi }_1^2$ is a chi-squared variable with one degree of freedom (the square of a standard normal). The limit is a scaled chi-squared, generally not normal; its sign tracks the sign of $g^{\prime \prime }(\theta )$.

Definition (variance-stabilizing transformation). Given a family with asymptotic variance depending on the mean, $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2(\theta ))$, a variance-stabilizing transformation is a smooth $g$ chosen so that the delta-method variance $g^{\prime }(\theta {)}^2{\sigma }^2(\theta )$ is constant in $\theta$. The defining condition is the ordinary differential equation

$$g^{\prime }(\theta )=\frac{c}{\sigma (\theta )},\text{equivalently}g(\theta )=c\int \frac{d\theta }{\sigma (\theta )},$$

for a constant $c$; then $\sqrt{n}(g(T_n)-g(\theta ))\stackrel{d}{\to }N(0,c^2)$, free of $\theta$.

Counterexamples to common slips Intermediate+

• Differentiability is needed, not just continuity. The map $g(x)=|x|$ is continuous everywhere but not differentiable at $0$. If $\sqrt{n}{T}_n\stackrel{d}{\to }N(0,1)$ (so $\theta =0$), then $\sqrt{n}g(T_n)=|\sqrt{n}{T}_n|\stackrel{d}{\to }|Z|$, a folded normal — not a centered normal. The delta method does not apply at a kink.

• The derivative is evaluated at $\theta$, not at $T_n$. Writing the limiting variance as $g^{\prime }(T_n{)}^2{\sigma }^2$ is an estimator of it, not the limit. The theorem's variance is the deterministic $g^{\prime }(\theta {)}^2{\sigma }^2$; plugging in $T_n$ is legitimate only afterward, justified by a separate Slutsky/continuous-mapping step (which is how standard errors are actually computed).

• A vanishing gradient is not an edge case to ignore. When $\nabla g(\theta )=0$ the sandwich covariance is the zero matrix, and the first-order statement degenerates to $r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }0$. This is correct but uninformative: the true fluctuation is at the slower scale $r_n^2$, captured only by the second-order method.

• The chi-squared limit is one-sided when $g$ has a strict extremum. If $\theta$ is a local maximum of $g$ then $g^{\prime \prime }(\theta )<0$ and $n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{g}^{\prime \prime }(\theta ){\sigma }^2{\chi }_1^2\le 0$ almost surely in the limit. Expecting a symmetric limit here is the standard error.

Key theorem with proof Intermediate+

Theorem (the delta method, first and second order). Let $T_n$ be random vectors in ${\mathbb{R}}^k$ and $\theta \in {\mathbb{R}}^k$ fixed.

(i) First order. If $r_n(T_n-\theta )\stackrel{d}{\to }S$ for some random vector $S$ and some $r_n\to \infty$, and $g:{\mathbb{R}}^k\to {\mathbb{R}}^m$ is differentiable at $\theta$ with Jacobian $\nabla g(\theta )$, then

$$r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }\nabla g(\theta )S.$$

In particular, if $S\sim N(0,\Sigma )$ then the limit is $N(0,\nabla g(\theta )\Sigma \nabla g(\theta {)}^{\mathsf{T}})$.

(ii) Second order. If $k=m=1$, $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }S$, and $g$ is twice differentiable at $\theta$ with $g^{\prime }(\theta )=0$, then

$$n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{g}^{\prime \prime }(\theta )S^2.$$

For $S\sim N(0,{\sigma }^2)$ this is $\frac{1}{2}{g}^{\prime \prime }(\theta ){\sigma }^2{\chi }_1^2$.

Proof. (i). Convergence in distribution at a diverging rate forces $T_n-\theta \stackrel{\mathbb{P}}{\to }0$: for the sequence $r_n(T_n-\theta )$ to converge in distribution it is bounded in probability, $r_n(T_n-\theta )=O_{\mathbb{P}}(1)$, so $T_n-\theta =O_{\mathbb{P}}(1/r_n)=o_{\mathbb{P}}(1)$ 45.04.01. Differentiability at $\theta$ gives the exact identity, valid for every $h$,

$$g(\theta +h)-g(\theta )=\nabla g(\theta )h+R(h),\frac{|R(h)|}{|h|}\to 0(h\to 0),$$

where $R(0)=0$ and we extend $R$ continuously. Set $h=T_n-\theta$ and multiply by $r_n$:

$$r_n(g(T_n)-g(\theta ))=\nabla g(\theta )[r_n(T_n-\theta )]+r_{n}R(T_n-\theta ).$$

The first term converges in distribution to $\nabla g(\theta )S$ by the continuous-mapping theorem applied to the linear (hence continuous) map $x\mapsto \nabla g(\theta )x$. For the remainder, define $\epsilon (h)=R(h)/|h|$ for $h\ne 0$ and $\epsilon (0)=0$, so $\epsilon$ is continuous at $0$ with $\epsilon (0)=0$. Then

$$r_{n}R(T_n-\theta )=\epsilon (T_n-\theta )\cdot |r_n(T_n-\theta )|.$$

Here $\epsilon (T_n-\theta )\stackrel{\mathbb{P}}{\to }0$ because $T_n-\theta \stackrel{\mathbb{P}}{\to }0$ and $\epsilon$ is continuous at $0$ (continuous-mapping in probability, 45.04.01), while $|r_n(T_n-\theta )|=O_{\mathbb{P}}(1)$. By the order rule $o_{\mathbb{P}}(1)\cdot O_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$, the remainder is $o_{\mathbb{P}}(1)$. Slutsky's lemma — a convergent-in-distribution leading term plus an $o_{\mathbb{P}}(1)$ remainder — gives $r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }\nabla g(\theta )S$.

(ii). With $g^{\prime }(\theta )=0$, the second-order Taylor expansion at $\theta$ is

$$g(\theta +h)-g(\theta )=\frac{1}{2}{g}^{\prime \prime }(\theta )h^2+R_2(h),\frac{R_2(h)}{h^2}\to 0(h\to 0).$$

Set $h=T_n-\theta$ and multiply by $n$:

$$n(g(T_n)-g(\theta ))=\frac{1}{2}{g}^{\prime \prime }(\theta )[\sqrt{n}(T_n-\theta ){]}^2+n{R}_2(T_n-\theta ).$$

The leading bracket squared converges in distribution to $S^2$ by the continuous-mapping theorem ($x\mapsto \frac{1}{2}{g}^{\prime \prime }(\theta )x^2$ is continuous), giving the stated limit. For the remainder, write $n{R}_2(T_n-\theta )=\delta (T_n-\theta )\cdot [\sqrt{n}(T_n-\theta ){]}^2$ with $\delta (h)=R_2(h)/h^2\to 0$ as $h\to 0$ and $\delta (0)=0$. Then $\delta (T_n-\theta )\stackrel{\mathbb{P}}{\to }0$ and $[\sqrt{n}(T_n-\theta ){]}^2=O_{\mathbb{P}}(1)$, so the product is $o_{\mathbb{P}}(1)$, and Slutsky's lemma delivers $n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{g}^{\prime \prime }(\theta )S^2$. $\square$

Bridge. This theorem builds toward the asymptotic distribution theory of every plug-in and maximum-likelihood estimator in the chapter, and the same Taylor-plus-Slutsky mechanism appears again in the proof of asymptotic normality of M-estimators, where the score function is expanded about the truth and the remainder is certified as $o_{\mathbb{P}}$. The foundational reason the method works is that a smooth map is locally linear and the linearisation error, deterministic and $o(|h|)$, is upgraded to a stochastic $o_{\mathbb{P}}$ remainder exactly because $r_n(T_n-\theta )$ is bounded in probability — this is exactly the $O_{\mathbb{P}}/o_{\mathbb{P}}$ calculus of 45.04.01 doing its job. Putting these together with the continuous-mapping theorem, the delta method generalises the central limit theorem 37.03.02 from sample averages to arbitrary smooth functions of them, and the second-order form is the bridge to the chi-squared limits that govern likelihood-ratio and score tests when a parameter sits at a stationary point of the functional. The bridge is the passage from a limit law for an estimator to a limit law for any smooth quantity reported from it.

Exercises Intermediate+

Advanced results Master

Theorem 1 (the multivariate delta method and the sandwich covariance). Let $r_n(T_n-\theta )\stackrel{d}{\to }S$ in ${\mathbb{R}}^k$ and let $g:{\mathbb{R}}^k\to {\mathbb{R}}^m$ be differentiable at $\theta$ with Jacobian $D=\nabla g(\theta )\in {\mathbb{R}}^{m\times k}$. Then $r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }DS$, so for $S\sim N(0,\Sigma )$ the limit is $N(0,D\Sigma D^{\mathsf{T}})$. The covariance $D\Sigma D^{\mathsf{T}}$ is degenerate (rank-deficient) exactly when $D$ has a nontrivial kernel intersecting the range of $\Sigma$, which is the multivariate signature of a vanishing directional derivative; in those directions the fluctuation lives at a slower rate and a second-order analysis is required. The sandwich form is the universal template for standard errors of plug-in functionals, including the asymptotic covariance $J^{-1}\Sigma J^{-1}$ of an estimating-equation solution, where $J$ is the derivative of the estimating function ^{[van der Vaart 1998]}.

Theorem 2 (second-order delta method, multivariate quadratic form). Let $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }S$ in ${\mathbb{R}}^k$ with $g:{\mathbb{R}}^k\to \mathbb{R}$ twice differentiable at $\theta$ and $\nabla g(\theta )=0$. With Hessian $H={\nabla }^{2}g(\theta )$,

$$n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{S}^{\mathsf{T}}HS.$$

For $S\sim N(0,\Sigma )$ the limit $\frac{1}{2}{S}^{\mathsf{T}}HS$ is a quadratic form in a Gaussian, distributed as $\frac{1}{2}{\sum }_j{\lambda }_j{W}_j$ with $W_j$ independent ${\chi }_1^2$ and ${\lambda }_j$ the eigenvalues of $H^{1/2}\Sigma H^{1/2}$ (when $H⪰0$). When $H=c{\Sigma }^{-1}$ the limit collapses to $\frac{c}{2}{\chi }_k^2$ — the structural origin of the chi-squared null distributions of Wald, score, and likelihood-ratio statistics under regularity, with $k$ the number of constrained parameters ^{[Lehmann 1999]}.

Theorem 3 (variance stabilization as an ODE, and the canonical solutions). If $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2(\theta ))$ with $\sigma$ continuous and positive, the transformation solving $g^{\prime }(\theta )=1/\sigma (\theta )$ yields $\sqrt{n}(g(T_n)-g(\theta ))\stackrel{d}{\to }N(0,1)$. Three classical instances follow by integrating $1/\sigma$. For the binomial proportion, ${\sigma }^2(p)=p(1-p)$ gives $g(p)=2\arcsin \sqrt{p}$ (the arcsine transform). For Poisson counts, ${\sigma }^2(\lambda )=\lambda$ gives $g(\lambda )=2\sqrt{\lambda }$ (the square-root transform). For the bivariate-normal correlation, ${\sigma }^2(\rho )=(1-{\rho }^2{)}^2$ gives $g(\rho )=\mathrm{arctanh}\rho$ (Fisher's $z$). For the exponential/scale mean with ${\sigma }^2(\mu )={\mu }^2$, the solution is $g(\mu )=\log \mu$, explaining the ubiquity of the log scale for positive quantities with constant coefficient of variation ^{[Fisher 1921]}.

Theorem 4 (functional delta method via Hadamard differentiability). When the estimator is a map $\varphi ({\hat{F}}_n)$ of an empirical distribution ${\hat{F}}_n$ in a normed space — quantiles, trimmed means, the Wilcoxon functional, copula functionals — the Euclidean Jacobian is replaced by a Hadamard derivative ${\varphi }_F^{\prime }$, a continuous linear map between the relevant Banach spaces. If $r_n({\hat{F}}_n-F)\stackrel{d}{\to }G$ in distribution in the space (e.g. $G$ a Brownian bridge for the empirical process) and $\varphi$ is Hadamard-differentiable at $F$ tangentially to the support of $G$, then $r_n(\varphi ({\hat{F}}_n)-\varphi (F))\stackrel{d}{\to }{\varphi }_F^{\prime }(G)$. The finite-dimensional delta method is the special case where the space is ${\mathbb{R}}^k$ and Hadamard differentiability coincides with ordinary differentiability; the extension is what makes the bootstrap consistent for smooth statistical functionals ^{[van der Vaart 1998]}.

Synthesis. The foundational reason the delta method is the workhorse of large-sample inference is that it converts a single limit law — the central limit theorem 37.03.02 — into a limit law for every smooth quantity reported from an estimator, by linearising the map and discarding the linearisation error as $o_{\mathbb{P}}$ of the leading term. The central insight is that the rate of the input fluctuation and the order of vanishing of the map together determine the output: a nonzero gradient gives a Gaussian limit at the input rate with the sandwich covariance $\nabla g\Sigma \nabla g^{\mathsf{T}}$; a vanishing gradient pushes the fluctuation to the squared rate, where the Hessian produces a quadratic-form-in-a-Gaussian, and this is exactly the mechanism that generalises the scalar chi-squared limit into the ${\chi }_k^2$ null laws of the classical tests. Putting these together, variance stabilization is the dual move — instead of accepting the $\theta$-dependent sandwich variance one solves the ordinary differential equation $g^{\prime }=1/\sigma$ to make it constant, and the arcsine, square-root, logarithm, and Fisher $z$ transforms are its canonical solutions. The bridge from the finite-dimensional method to the functional delta method is Hadamard differentiability, which is dual to ordinary differentiability in that it is precisely the regularity making the chain rule and the bootstrap valid for maps of the empirical process; the entire apparatus — first order, second order, stabilization, functional — is one principle, local linearity transported through a limit, applied at successively finer scales and in successively larger spaces.

Full proof set Master

Proposition 1 (first-order delta method, with the remainder bookkeeping explicit). If $r_n(T_n-\theta )\stackrel{d}{\to }S$ in ${\mathbb{R}}^k$ with $r_n\to \infty$ and $g$ is differentiable at $\theta$, then $r_n(g(T_n)-g(\theta ))\stackrel{d}{\to }\nabla g(\theta )S$.

Proof. Boundedness in probability of a convergent-in-distribution sequence gives $r_n(T_n-\theta )=O_{\mathbb{P}}(1)$, hence $T_n-\theta =o_{\mathbb{P}}(1)$ 45.04.01. Define $\epsilon (h)=(g(\theta +h)-g(\theta )-\nabla g(\theta )h)/|h|$ for $h\ne 0$ and $\epsilon (0)=0$; differentiability is the statement $\epsilon (h)\to 0$ as $h\to 0$, so $\epsilon$ is continuous at $0$. Then

$$r_n(g(T_n)-g(\theta ))=\nabla g(\theta )r_n(T_n-\theta )+\epsilon (T_n-\theta )|r_n(T_n-\theta )|.$$

By continuous mapping in probability $\epsilon (T_n-\theta )\stackrel{\mathbb{P}}{\to }0$, while $|r_n(T_n-\theta )|=O_{\mathbb{P}}(1)$, so the second term is $o_{\mathbb{P}}(1)O_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$. The first term tends in distribution to $\nabla g(\theta )S$ by continuous mapping under the linear map. Slutsky's lemma combines them. $\square$

Proposition 2 (second-order delta method, scalar). If $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }S$ in $\mathbb{R}$, $g$ is twice differentiable at $\theta$, and $g^{\prime }(\theta )=0$, then $n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{g}^{\prime \prime }(\theta )S^2$.

Proof. Twice differentiability at $\theta$ with $g^{\prime }(\theta )=0$ gives $g(\theta +h)-g(\theta )=\frac{1}{2}{g}^{\prime \prime }(\theta )h^2+\delta (h)h^2$ with $\delta (h)\to 0$ as $h\to 0$, $\delta (0)=0$. Setting $h=T_n-\theta$ and multiplying by $n=(\sqrt{n}{)}^2$,

$$n(g(T_n)-g(\theta ))=\frac{1}{2}{g}^{\prime \prime }(\theta )[\sqrt{n}(T_n-\theta ){]}^2+\delta (T_n-\theta )[\sqrt{n}(T_n-\theta ){]}^2.$$

The bracket squared converges in distribution to $S^2$ by continuous mapping; $\delta (T_n-\theta )\stackrel{\mathbb{P}}{\to }0$ by continuity of $\delta$ at $0$ and $T_n\stackrel{\mathbb{P}}{\to }\theta$; and $[\sqrt{n}(T_n-\theta ){]}^2=O_{\mathbb{P}}(1)$. The remainder is $o_{\mathbb{P}}(1)$ and Slutsky gives the limit. The distributional identity $S^2={\sigma }^2{\chi }_1^2$ for $S\sim N(0,{\sigma }^2)$ follows because $(S/\sigma {)}^2$ is the square of a standard normal. $\square$

Proposition 3 (multivariate second order: the quadratic-form limit). Let $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }S$ in ${\mathbb{R}}^k$, $g:{\mathbb{R}}^k\to \mathbb{R}$ twice differentiable at $\theta$ with $\nabla g(\theta )=0$ and Hessian $H$. Then $n(g(T_n)-g(\theta ))\stackrel{d}{\to }\frac{1}{2}{S}^{\mathsf{T}}HS$.

Proof. The second-order Taylor expansion with vanishing gradient is $g(\theta +h)-g(\theta )=\frac{1}{2}{h}^{\mathsf{T}}Hh+\delta (h)|h{|}^2$ with $\delta (h)\to 0$ as $h\to 0$. Multiply by $n$ after substituting $h=T_n-\theta$:

$$n(g(T_n)-g(\theta ))=\frac{1}{2}[\sqrt{n}(T_n-\theta ){]}^{\mathsf{T}}H[\sqrt{n}(T_n-\theta )]+\delta (T_n-\theta )|\sqrt{n}(T_n-\theta ){|}^2.$$

The map $x\mapsto \frac{1}{2}{x}^{\mathsf{T}}Hx$ is continuous, so the leading term converges in distribution to $\frac{1}{2}{S}^{\mathsf{T}}HS$; the remainder is $o_{\mathbb{P}}(1)O_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$. Slutsky finishes. For $S\sim N(0,\Sigma )$ with $H⪰0$, diagonalise $H^{1/2}\Sigma H^{1/2}=O\Lambda O^{\mathsf{T}}$: then $S^{\mathsf{T}}HS=\Vert H^{1/2}S{\Vert }^2$ has the law ${\sum }_j{\lambda }_j{W}_j$ with $W_j$ i.i.d. ${\chi }_1^2$, and reduces to $c{\chi }_k^2$ exactly when $H^{1/2}\Sigma H^{1/2}=cI$, i.e. $H=c{\Sigma }^{-1}$. $\square$

Proposition 4 (variance-stabilizing ODE has a strictly monotone solution). If $\sigma :I\to (0,\infty )$ is continuous on an interval $I$, then $g(\theta )={\int }_{{\theta }_0}^{\theta }\sigma (u{)}^{-1}du$ is strictly increasing, continuously differentiable, satisfies $g^{\prime }(\theta )=1/\sigma (\theta )$, and yields $\sqrt{n}(g(T_n)-g(\theta ))\stackrel{d}{\to }N(0,1)$.

Proof. The integrand $1/\sigma$ is continuous and strictly positive, so by the fundamental theorem of calculus $g$ is $C^1$ with $g^{\prime }(\theta )=1/\sigma (\theta )>0$, hence strictly increasing and invertible on $I$. Applying the first-order delta method (Proposition 1) to $g$ at $\theta$ under $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2(\theta ))$ gives asymptotic variance $g^{\prime }(\theta {)}^2{\sigma }^2(\theta )=\sigma (\theta {)}^{-2}{\sigma }^2(\theta )=1$, so the limit is $N(0,1)$ regardless of $\theta$. Strict monotonicity guarantees the back-transformation $g^{-1}$ used to form confidence intervals on the original scale is well defined. $\square$

Proposition 5 (the kink obstruction: necessity of differentiability). There exist $T_n$ with $\sqrt{n}{T}_n\stackrel{d}{\to }N(0,1)$ and a continuous $g$ for which $\sqrt{n}g(T_n)$ does not converge to any normal law.

Proof. Take $\theta =0$, $g(x)=|x|$, and $T_n$ with $\sqrt{n}{T}_n\stackrel{d}{\to }Z\sim N(0,1)$. Then $\sqrt{n}g(T_n)=|\sqrt{n}{T}_n|\stackrel{d}{\to }|Z|$ by the continuous-mapping theorem, since $x\mapsto |x|$ is continuous. The limit $|Z|$ is a folded (half-)normal supported on $[0,\infty )$, not a centered normal. The obstruction is the non-differentiability of $g$ at $0$: no single linear map approximates $|\cdot |$ near $0$, so the delta-method linearisation has no candidate slope, and the conclusion legitimately fails. This delimits the hypothesis of the theorem. $\square$

Connections Master

Consistency of estimators and the modes of stochastic convergence 45.04.01 is the immediate substrate: the proof of the delta method runs entirely on the $O_{\mathbb{P}}/o_{\mathbb{P}}$ calculus, the continuous-mapping theorem, and Slutsky's lemma developed there, and the step that the $\sqrt{n}$ rate forces $T_n\stackrel{\mathbb{P}}{\to }\theta$ — turning a deterministic Taylor remainder into a stochastic $o_{\mathbb{P}}(1)$ one — is exactly the order-calculus bridge that unit certifies.

The Lindeberg-Feller central limit theorem 37.03.02 supplies the input hypothesis $\sqrt{n}(T_n-\theta )\stackrel{d}{\to }N(0,\Sigma )$ for the leading case $T_n={\bar{X}}_n$, and the multivariate central limit theorem supplies the joint Gaussian limit that the multivariate delta method transports through the Jacobian; without that input limit law there is nothing for the method to transform.

Consistency and asymptotic normality of M-estimators and maximum likelihood 45.04.04 consumes the delta method twice over: the asymptotic normality of $\hat{\theta }$ is itself a delta-method-style linearisation of the score equation, and the asymptotic distribution of any smooth function $\psi (\hat{\theta })$ of the estimate is obtained by composing with the delta method, giving the sandwich covariance $\nabla \psi I(\theta {)}^{-1}\nabla {\psi }^{\mathsf{T}}$ in terms of the Fisher information.

The chi-squared limits of Wald, score, and likelihood-ratio tests 45.06.01 are the second-order delta method in action: under the null a test statistic is a quadratic functional of an asymptotically normal vector evaluated where the first-order term vanishes, and Theorem 2 / Proposition 3 deliver the ${\chi }_k^2$ null distribution with $k$ the number of restrictions.

Empirical processes and Glivenko-Cantelli classes 45.05.02 are where the functional delta method (Theorem 4) lives: the weak convergence of the empirical process to a Brownian bridge is the infinite-dimensional analogue of the central limit input, and Hadamard differentiability is the regularity that lets the finite-dimensional delta method extend to quantiles, L-statistics, and the bootstrap of smooth functionals.

Historical & philosophical context Master

The delta method has roots in the nineteenth-century propagation-of-errors calculus of Gauss and Laplace, where the variance of a function of measured quantities was approximated by squaring the relevant partial derivatives. Its modern statistical form — as a theorem about the asymptotic distribution of a smooth function of an asymptotically normal estimator — was systematised by Harald Cramér in his 1946 Mathematical Methods of Statistics ^{[Cramér 1946]}, which treated the asymptotic normality of functions of sample moments with the Taylor-expansion-plus-remainder argument that remains the standard proof. The name "delta method" became common in the mid-twentieth-century statistical literature, the $\delta$ recalling the small increment in the Taylor expansion.

The variance-stabilizing transformations predate the general theorem and motivated it. Ronald Fisher's 1921 Metron paper ^{[Fisher 1921]} introduced the $z=\mathrm{arctanh}r$ transformation of the sample correlation precisely to obtain an approximately normal statistic with variance nearly independent of the unknown $\rho$, an explicit solution of the stabilizing differential equation before that equation was stated abstractly. Maurice Bartlett's 1936 analysis ^{[Bartlett 1936]} developed the square-root transform for count data and the general principle that one integrates the reciprocal of the standard-deviation function to stabilize variance. The second-order form, needed when the first derivative vanishes, and the functional generalisation through Hadamard differentiability were consolidated in the modern asymptotic-statistics literature, where the delta method is presented as a single principle with finite-dimensional, second-order, and infinite-dimensional faces.

Bibliography Master

@book{Cramer1946,
  author    = {Cram\'er, Harald},
  title     = {Mathematical Methods of Statistics},
  publisher = {Princeton University Press},
  address   = {Princeton},
  year      = {1946}
}

@article{Fisher1921,
  author  = {Fisher, Ronald A.},
  title   = {On the `probable error' of a coefficient of correlation deduced from a small sample},
  journal = {Metron},
  volume  = {1},
  year    = {1921},
  pages   = {3--32}
}

@article{Bartlett1936,
  author  = {Bartlett, Maurice S.},
  title   = {The square root transformation in analysis of variance},
  journal = {Supplement to the Journal of the Royal Statistical Society},
  volume  = {3},
  number  = {1},
  year    = {1936},
  pages   = {68--78}
}

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{Ferguson1996,
  author    = {Ferguson, Thomas S.},
  title     = {A Course in Large Sample Theory},
  publisher = {Chapman \& Hall},
  address   = {London},
  year      = {1996}
}

@book{Lehmann1999,
  author    = {Lehmann, Erich L.},
  title     = {Elements of Large-Sample Theory},
  publisher = {Springer-Verlag},
  address   = {New York},
  year      = {1999}
}

@book{BickelDoksum2015,
  author    = {Bickel, Peter J. and Doksum, Kjell A.},
  title     = {Mathematical Statistics: Basic Ideas and Selected Topics, Volume I},
  edition   = {2nd},
  publisher = {CRC Press},
  address   = {Boca Raton},
  year      = {2015}
}