M-Estimators and Z-Estimators: The Argmax Theorem and the Master Theorem

Intuition Beginner

Almost every estimator you meet is the answer to one of two questions. The first is: which setting of the unknown makes my data score best, where "score" is some quality you choose to average over the sample? The second is: which setting makes a chosen balancing condition come out to exactly zero on average? The first kind is found by maximising; the second by solving an equation. The surprise is that these are two views of one machine, and one body of theory tells you how good the answer is for both.

The maximum-likelihood recipe is the headline case of the first kind: turn the dial until the data look least surprising. But the sample mean, the sample median, the slope in a regression, and a long list of robust alternatives all fit the same template — pick a per-observation score, average it, and optimise. Picking the score is where the design freedom lives. A score that grows without bound for outliers gives you the fragile mean; a score that flattens out for far-away points gives you something that shrugs off a few bad measurements.

The questions this unit answers are the same two we always ask. Does the recipe home in on the truth as data accumulate? And once it does, how wide is the leftover wobble? The answer to the second is a tidy formula — a "sandwich" — built from two ingredients: how steeply the balancing condition responds to moving the dial, and how much the condition itself jitters from sample to sample.

The one-sentence takeaway: estimators that maximise an averaged score or solve an averaged equation form one family, they home in on the truth when the population version has a clean best setting, and their wobble is a sandwich of slope and jitter that collapses to the maximum-likelihood answer when the score is the likelihood's.

Visual Beginner

Picture two ways of finding the same point. On the left, a curved hill whose height is the averaged score; you walk to the top and read off the setting — that is maximising. On the right, a line crossing zero; you find where the averaged balancing condition equals zero — that is solving. The top of the hill sits directly above the zero-crossing, because the slope of the hill is the balancing condition. Two pictures, one answer.

recipe	what you do	the wobble depends on
maximise an averaged score	climb to the top of the hill	how sharply the top is curved
solve an averaged equation	find the zero-crossing	the steepness at the crossing and the jitter of the condition
both, when the score is the likelihood	the two coincide	one number, the Fisher information

The takeaway: maximising a score and solving its balancing equation land on the same estimate, and the spread of that estimate is set by how steeply the condition crosses zero against how much the condition jitters.

Worked example Beginner

We find the spread of a simple location estimator that solves a balancing equation, using concrete numbers. The recipe: estimate the centre of a symmetric measurement by the setting at which the average of the clipped deviations is zero, where clipping caps each deviation at plus or minus $2$. This is the Huber idea — far-away points count, but only up to a cap.

Step 1. The balancing condition. For a guess $\theta$, each observation $x$ contributes a clipped deviation: $x-\theta$ if that is between $-2$ and $2$, and otherwise $+2$ or $-2$. The estimate is the $\theta$ that makes the average contribution zero.

Step 2. The steepness. Suppose the true centre is $0$ and the measurements are spread so that $80\%$ of them land within the clip range. Inside the range, nudging $\theta$ up by a small amount lowers each contribution one-for-one; outside, it does nothing. So the average contribution responds with steepness $0.80$ — only the unclipped fraction pushes back. Call this $A=0.80$.

Step 3. The jitter. The clipped deviations have a spread from sample to sample; suppose one observation's clipped deviation has variance $B=0.90$.

Step 4. The sandwich. The large-sample variance of the estimate, per observation, is jitter divided by steepness squared: $B/A^2=0.90/0.64=1.41$. For $n=100$ observations the variance is $1.41/100=0.0141$, and the typical error is the square root, about $0.119$.

Step 5. What this tells us. The cap costs a little efficiency — the variance $1.41$ is above the $1.00$ you would get from the plain average on clean normal data — but in exchange a handful of wild outliers can move the estimate by no more than the cap allows. The sandwich formula, jitter over steepness squared, is the price tag.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_1,\dots ,X_n$ be i.i.d. from a law $P$ on a sample space $\mathcal{X}$, and write $P_n=n^{-1}{\sum }_{i=1}^n{\delta }_{X_i}$ for the empirical measure, so that for a function $f$ the empirical average is $P_{n}f=n^{-1}{\sum }_{i=1}^{n}f(X_i)$ and the population average is $Pf={\mathbb{E}}_{P}f(X)$. The parameter $\theta$ ranges over an open $\Theta \subseteq {\mathbb{R}}^k$. The modes of stochastic convergence $\stackrel{d}{\to },\stackrel{\mathbb{P}}{\to }$ and the $o_{\mathbb{P}}/O_{\mathbb{P}}$ calculus are those of 45.04.01.

Definition (M-estimator). Given a criterion function $m_{\theta }:\mathcal{X}\to \mathbb{R}$, the associated M-estimator is any (measurable) maximiser of the empirical criterion,

$${\hat{\theta }}_n\in \arg \underset{\theta \in \Theta }{\max }{M}_n(\theta ),M_n(\theta )=P_n{m}_{\theta }=\frac{1}{n}\sum _{i=1}^n{m}_{\theta }(X_i),$$

with population criterion $M(\theta )=P{m}_{\theta }$. (Minimisers of a loss $-m_{\theta }$ are subsumed by sign change; the letter $M$ stands for maximum-likelihood-type.)

Definition (Z-estimator). Given an estimating function (also score function or psi-function) ${\psi }_{\theta }:\mathcal{X}\to {\mathbb{R}}^k$, the associated Z-estimator is any (measurable) solution of the empirical estimating equation

$${\Psi }_n(\theta ):=P_n{\psi }_{\theta }=\frac{1}{n}\sum _{i=1}^n{\psi }_{\theta }(X_i)=0,$$

with population estimating function $\Psi (\theta )=P{\psi }_{\theta }$. (The letter $Z$ stands for zero.) When $m_{\theta }$ is differentiable in $\theta$ the M-estimator with ${\psi }_{\theta }={\nabla }_{\theta }{m}_{\theta }$ is a Z-estimator with the same solution set at interior maxima; the Z-formulation does not require an underlying criterion and so is the more general object.

Definition (Fisher consistency). The estimating function ${\psi }_{\theta }$ is Fisher consistent for ${\theta }_0$ at $P$ if $\Psi ({\theta }_0)=P{\psi }_{{\theta }_0}=0$: the population estimating equation is solved at the true parameter. For an M-estimator this is the first-order condition ${\nabla }_{\theta }M({\theta }_0)=0$ at the population maximiser.

Definition (the sandwich matrices). Assume $\theta \mapsto {\psi }_{\theta }$ is differentiable and the interchange of differentiation and integration is licensed. The sensitivity matrix and the variability matrix at ${\theta }_0$ are

$$A=A({\theta }_0)=-P{\nabla }_{\theta }{\psi }_{{\theta }_0}=-{\mathbb{E}}_P[{\nabla }_{\theta }{\psi }_{{\theta }_0}(X)]\in {\mathbb{R}}^{k\times k},B=B({\theta }_0)=P\left({\psi }_{{\theta }_0}{\psi }_{{\theta }_0}^{\mathsf{T}}\right)={\mathrm{Cov}}_P({\psi }_{{\theta }_0}(X)),$$

the second equality for $B$ using Fisher consistency $P{\psi }_{{\theta }_0}=0$. The sandwich (asymptotic) covariance is $V=A^{-1}B{A}^{-\mathsf{T}}$, assumed well defined ($A$ nonsingular, $B$ finite).

Definition (influence function). Regard the Z-estimator as a statistical functional $T(P)$ defined by $P{\psi }_{T(P)}=0$. The influence function of $T$ at $P$ is the Gateaux derivative in the direction of a point mass: with $P_{\epsilon ,x}=(1-\epsilon )P+\epsilon {\delta }_x$,

$$\mathrm{IF}(x;T,P)={\frac{\mathrm{d}}{\mathrm{d}\epsilon }|}_{\epsilon =0}T(P_{\epsilon ,x})=A^{-1}{\psi }_{{\theta }_0}(x).$$

The estimator is B-robust if its influence function is bounded, ${\sup }_x|\mathrm{IF}(x;T,P)|<\infty$; the supremum is the gross-error sensitivity. Because $\mathrm{IF}\propto {\psi }_{{\theta }_0}$, B-robustness is exactly boundedness of the estimating function.

Counterexamples to common slips Intermediate+

• The sandwich is $A^{-1}B{A}^{-\mathsf{T}}$, not $A^{-1}$, in general. Only when the second Bartlett identity $A=B$ holds — which it does for the likelihood score by the information equality of 45.04.03 — does the sandwich collapse to $A^{-1}=B^{-1}=I({\theta }_0{)}^{-1}$. Reporting $A^{-1}$ for a misspecified or non-likelihood $\psi$ understates the variance.

• Fisher consistency is a condition on $\psi$ relative to $P$, not an automatic property. If $P$ is not in the model, $P{\psi }_{{\theta }_0}\ne 0$ at the model's ${\theta }_0$; the estimator then targets the pseudo-true root ${\theta }_{\ast }$ of $P{\psi }_{\theta }=0$, which need not equal any model parameter.

• Consistency needs the well-separated maximum, not merely $M({\theta }_0)=\max M$. A population criterion with a flat ridge or a far-away near-maximiser breaks the argmax theorem even when ${\theta }_0$ is a global maximiser; the separation ${\sup }_{|\theta -{\theta }_0|\ge \epsilon }M(\theta )<M({\theta }_0)$ is the substantive hypothesis (as in 45.04.01).

• A bounded $\psi$ buys robustness but generally costs efficiency. The Huber estimator's sandwich variance exceeds the Fisher floor at the exact normal model; B-robustness and full efficiency are in tension, and the Huber tuning constant is the dial that trades them off.

Key theorem with proof Intermediate+

The signature result is the Z-estimator master theorem: under a one-term Taylor expansion of the estimating equation controlled by consistency and a derivative law of large numbers, a Z-estimator is asymptotically normal with the sandwich covariance, and the maximum-likelihood case recovers the inverse Fisher information of 45.04.03. The proof is the same Taylor-plus-Slutsky mechanism as the MLE theorem, with the likelihood score $s_{\theta }$ replaced by a general estimating function ${\psi }_{\theta }$ and the information $I({\theta }_0)$ split into its two halves $A$ and $B$.

Theorem (Z-estimator master theorem). Let $X_1,\dots ,X_n$ be i.i.d. from $P$ and let ${\hat{\theta }}_n$ solve $P_n{\psi }_{{\hat{\theta }}_n}=0$, with true root ${\theta }_0$ interior to $\Theta$, $P{\psi }_{{\theta }_0}=0$. Assume

(Z1) consistency: ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$ (for instance via the argmax/argmin theorem of 45.04.01, applied to a criterion whose gradient is $\psi$, or via the Z-analogue Proposition 1 below);

(Z2) the map $\theta \mapsto {\psi }_{\theta }(x)$ is continuously differentiable in a neighbourhood $U$ of ${\theta }_0$ for $P$-a.e. $x$, the derivative is dominated, $\Vert {\nabla }_{\theta }{\psi }_{\theta }(x)\Vert \le \dot{m}(x)$ on $U$ with $P\dot{m}<\infty$, and $A=-P{\nabla }_{\theta }{\psi }_{{\theta }_0}$ is nonsingular;

(Z3) $B=P({\psi }_{{\theta }_0}{\psi }_{{\theta }_0}^{\mathsf{T}})$ is finite.

Then

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,A^{-1}B{A}^{-\mathsf{T}}).$$

Proof. By (Z1) the event $\{{\hat{\theta }}_n\in U\}$ has probability tending to $1$, and on it the estimating equation ${\Psi }_n({\hat{\theta }}_n)=P_n{\psi }_{{\hat{\theta }}_n}=0$ holds with ${\hat{\theta }}_n$ interior. A first-order Taylor expansion of $\theta \mapsto {\Psi }_n(\theta )$ about ${\theta }_0$, with the mean-value remainder at intermediate points ${\bar{\theta }}_n$ on the segment from ${\theta }_0$ to ${\hat{\theta }}_n$, gives

$$0={\Psi }_n({\hat{\theta }}_n)={\Psi }_n({\theta }_0)+[{\nabla }_{\theta }{\Psi }_n({\bar{\theta }}_n)]({\hat{\theta }}_n-{\theta }_0),$$

where ${\nabla }_{\theta }{\Psi }_n(\theta )=P_n{\nabla }_{\theta }{\psi }_{\theta }$. Writing $A_n(\theta )=-{\nabla }_{\theta }{\Psi }_n(\theta )=-P_n{\nabla }_{\theta }{\psi }_{\theta }$ and inserting the rate,

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=A_n({\bar{\theta }}_n{)}^{-1}\sqrt{n}{\Psi }_n({\theta }_0),$$

valid once $A_n({\bar{\theta }}_n)$ is invertible, which holds with probability tending to $1$ by the steps below.

The estimating-function term obeys a CLT. The summands ${\psi }_{{\theta }_0}(X_i)$ are i.i.d. with mean $P{\psi }_{{\theta }_0}=0$ (Fisher consistency) and covariance $B$. The multivariate central limit theorem 37.03.02 gives

$$\sqrt{n}{\Psi }_n({\theta }_0)=\frac{1}{\sqrt{n}}\sum _{i=1}^n{\psi }_{{\theta }_0}(X_i)\stackrel{d}{\to }N(0,B).$$

The sensitivity term obeys an LLN. At the truth, the law of large numbers gives $A_n({\theta }_0)=-P_n{\nabla }_{\theta }{\psi }_{{\theta }_0}\stackrel{\mathbb{P}}{\to }-P{\nabla }_{\theta }{\psi }_{{\theta }_0}=A$. Evaluated at the random ${\bar{\theta }}_n$ (which lies between ${\theta }_0$ and ${\hat{\theta }}_n$, hence ${\bar{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$ by (Z1)), the domination (Z2) controls the difference: on $U$,

$$\Vert A_n({\bar{\theta }}_n)-A_n({\theta }_0)\Vert \le |{\bar{\theta }}_n-{\theta }_0|P_n\dot{m},$$

and $P_n\dot{m}\stackrel{\mathbb{P}}{\to }P\dot{m}<\infty$ while $|{\bar{\theta }}_n-{\theta }_0|\stackrel{\mathbb{P}}{\to }0$, so $A_n({\bar{\theta }}_n)-A_n({\theta }_0)\stackrel{\mathbb{P}}{\to }0$ and hence $A_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }A$.

Assemble by Slutsky. By (Z2) $A$ is nonsingular, so the continuous-mapping theorem 45.04.01 gives $A_n({\bar{\theta }}_n{)}^{-1}\stackrel{\mathbb{P}}{\to }{A}^{-1}$ (inversion is continuous on the invertible matrices). Slutsky's lemma multiplies the convergent-in-probability matrix factor by the convergent-in-distribution estimating-function factor:

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=A_n({\bar{\theta }}_n{)}^{-1}\sqrt{n}{\Psi }_n({\theta }_0)\stackrel{d}{\to }{A}^{-1}N(0,B)=N(0,A^{-1}B(A^{-1}{)}^{\mathsf{T}})=N(0,A^{-1}B{A}^{-\mathsf{T}}),$$

the covariance computed from $\mathrm{Cov}(CZ)=C\mathrm{Cov}(Z)C^{\mathsf{T}}$ with $C=A^{-1}$, $\mathrm{Cov}(Z)=B$. $\square$

Bridge. This theorem builds toward the asymptotic theory of every estimator defined by optimisation or estimating equations — robust location, quantile and least-absolute-deviation regression, generalized method of moments, generalized estimating equations — and it appears again in the test theory of 45.06.01, where the same sandwich furnishes the null covariance of Wald-type statistics under possible misspecification. The foundational reason the sandwich has its two-matrix shape is that an estimating equation converts the Gaussian fluctuation of $\sqrt{n}{\Psi }_n({\theta }_0)$, whose covariance is the variability $B$, into a fluctuation of ${\hat{\theta }}_n$ by inverting the sensitivity $A$ at which the equation crosses zero; the variability is sandwiched between two inverse sensitivities. This is exactly the delta-method principle of 45.04.02 applied to the implicitly defined root of ${\Psi }_n=0$: the implicit-function linearisation ${\hat{\theta }}_n-{\theta }_0\approx A^{-1}{\Psi }_n({\theta }_0)$ is upgraded to an $o_{\mathbb{P}}$ statement because $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$ is bounded in probability. Putting these together, the maximum-likelihood theorem of 45.04.03 is the special case $\psi =s$, where the two Bartlett identities force $A=B=I({\theta }_0)$ and the sandwich collapses to $I({\theta }_0{)}^{-1}$; the bridge is the recognition that what made the MLE efficient — the coincidence of sensitivity and variability — is precisely the structure a general $\psi$ need not have, and the gap between $A$ and $B$ is the price of departing from the likelihood.

Exercises Intermediate+

Advanced results Master

The master theorem extends to non-smooth estimating functions through stochastic equicontinuity, the argmax theorem supplies the consistency hypothesis it assumes, the influence function makes the sandwich a derivative of the estimator functional and turns robustness into a boundedness statement, and misspecification replaces the model parameter by the Kullback-Leibler projection while leaving the sandwich form intact. The likelihood case is the rigid centre from which all of these are deformations.

Theorem 1 (argmax/argmin consistency theorem). Let $M_n(\theta )=P_n{m}_{\theta }$ and $M(\theta )=P{m}_{\theta }$, and suppose ${\theta }_0$ is a well-separated maximiser of $M$: for every $\epsilon >0$, ${\sup }_{|\theta -{\theta }_0|\ge \epsilon }M(\theta )<M({\theta }_0)$. If a uniform law of large numbers holds, ${\sup }_{\theta \in \Theta }|M_n(\theta )-M(\theta )|\stackrel{\mathbb{P}}{\to }0$, and ${\hat{\theta }}_n$ nearly maximises, $M_n({\hat{\theta }}_n)\ge M_n({\theta }_0)-o_{\mathbb{P}}(1)$, then ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$. The uniform law is the substantive input; it is delivered by Glivenko-Cantelli classes 45.05.02 under finite bracketing numbers, and the separation converts closeness of objective values into closeness of argmaxima. This is hypothesis (Z1) of the master theorem and the consistency half of the whole subject ^{[van der Vaart — Asymptotic Statistics]}.

Theorem 2 (the influence function and B-robustness). The Z-estimator functional $T(P)$, $P{\psi }_{T(P)}=0$, has influence function $\mathrm{IF}(x;T,P)=A^{-1}{\psi }_{{\theta }_0}(x)$, a Gateaux derivative in the direction ${\delta }_x-P$; its second moment $P(\mathrm{IF}{\mathrm{IF}}^{\mathsf{T}})=A^{-1}B{A}^{-\mathsf{T}}$ is the sandwich, so the asymptotic variance is the squared norm of the influence function and $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)={\mathbb{G}}_n\mathrm{IF}+o_{\mathbb{P}}(1)$ is the influence-function linearisation. The estimator is B-robust — bounded gross-error sensitivity ${\gamma }^{\ast }={\sup }_x|\mathrm{IF}(x)|<\infty$ — exactly when $\psi$ is bounded. The mean ($\psi =x-\theta$) is not B-robust; the median ($\psi =\mathrm{sign}$) and the Huber estimator (bounded ${\psi }_c$) are ^{[Hampel — The influence curve and its role in robust estimation]}.

Theorem 3 (Huber's minimax-variance estimator). Over the $\epsilon$-contamination neighbourhood ${\mathcal{P}}_{\epsilon }=\{(1-\epsilon )\Phi +\epsilon H:H\text{ arbitrary}\}$ of the standard normal $\Phi$, the M-estimator of location minimising the maximum asymptotic variance is the Huber estimator with ${\psi }_c(u)=\max (-c,\min (c,u))$, the clipping constant $c$ determined by $\epsilon$ through $2\varphi (c)/c-2\Phi (-c)=\epsilon /(1-\epsilon )$. Its least-favourable distribution is normal in the centre with exponential (Laplace) tails, for which the Huber $\psi$ is the likelihood score; the estimator is thus the maximum-likelihood estimator for the worst case in the neighbourhood, the saddle point of the variance-versus-contamination game. As $\epsilon \to 0$, $c\to \infty$ and Huber reduces to the mean; as $\epsilon$ grows, $c$ shrinks toward the median ^{[Huber — Robust estimation of a location parameter]}.

Theorem 4 (the sandwich under misspecification; White). If $P$ lies outside the model, the M-estimator with $m_{\theta }=\log f_{\theta }$ converges to the pseudo-true parameter ${\theta }_{\ast }$ minimising the Kullback-Leibler divergence $\mathrm{KL}(P\Vert f_{\theta })$ — the unique root of $P{s}_{\theta }=0$ when it is well-separated — and $\sqrt{n}({\hat{\theta }}_n-{\theta }_{\ast })\stackrel{d}{\to }N(0,A^{-1}B{A}^{-1})$ with $A=-P{\nabla }_{\theta }{s}_{{\theta }_{\ast }}$ and $B={\mathrm{Cov}}_P(s_{{\theta }_{\ast }})$. Under correct specification the information equality forces $A=B=I({\theta }_0)$ and the sandwich collapses to $I({\theta }_0{)}^{-1}$; the discrepancy between the observed ${\hat{A}}_n$ and ${\hat{B}}_n$ is therefore the basis of the information-matrix test for misspecification ^{[White — Maximum likelihood estimation of misspecified models]}.

Theorem 5 (efficiency among Z-estimators; the score is optimal). Among all Fisher-consistent estimating functions ${\psi }_{\theta }$ for a correctly specified regular model, the asymptotic variance $A^{-1}B{A}^{-\mathsf{T}}$ is minimised in the Loewner order by the likelihood score $\psi =s$, attaining $I({\theta }_0{)}^{-1}$. The proof is a Cauchy-Schwarz / projection argument: for any $\psi$ with $P{\psi }_{{\theta }_0}=0$ and $A=-P{\nabla }_{\theta }{\psi }_{{\theta }_0}=P({\psi }_{{\theta }_0}{s}_{{\theta }_0}^{\mathsf{T}})$ (an integration-by-parts identity), the matrix inequality $A^{-1}B{A}^{-\mathsf{T}}⪰I({\theta }_0{)}^{-1}$ follows from $\mathrm{Cov}({\psi }_{{\theta }_0}-A{s}_{{\theta }_0}^{\mathsf{T}}{I}^{-1}\cdot )⪰0$, with equality iff ${\psi }_{{\theta }_0}$ is a fixed linear transform of the score. Departing from the score to gain robustness (a bounded $\psi$) therefore necessarily raises the variance — the efficiency-robustness tradeoff made quantitative ^{[Huber, P. J. & Ronchetti, E. M. — Robust Statistics (2nd ed.)]}.

Synthesis. The foundational reason a single theory governs maximisers and equation-solvers is that the first-order condition identifies them: an M-estimator's criterion gradient is a Z-estimator's estimating function, so the central limit theorem 37.03.02 acting on ${\psi }_{{\theta }_0}$ produces a $N(0,B)$ fluctuation, and inverting the sensitivity $A$ — which the law of large numbers pins down — converts it into the estimate's $N(0,A^{-1}B{A}^{-\mathsf{T}})$ law. The central insight is that the variance is the squared norm of the influence function $A^{-1}{\psi }_{{\theta }_0}$, so the asymptotic behaviour of the estimator is read off a single function of one observation; this is exactly the linearisation $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)={\mathbb{G}}_n\mathrm{IF}+o_{\mathbb{P}}(1)$, the delta method of 45.04.02 applied to the implicitly defined root. The maximum-likelihood theory of 45.04.03 is the rigid special case where the two Bartlett identities force $A=B=I({\theta }_0)$ and the sandwich collapses to the inverse Fisher information; this is dual to the Cramér-Rao computation, the finite-sample optimality of the score reappearing as the Loewner-minimality of its sandwich among all Fisher-consistent $\psi$.

Putting these together, robustness and efficiency are the two ends of one dial: boundedness of $\psi$ caps the gross-error sensitivity but, by the Cauchy-Schwarz projection of Theorem 5, necessarily separates $A$ from $B$ and inflates the variance above $I^{-1}$, and the Huber estimator is the saddle point where the worst-case variance over a contamination neighbourhood is smallest. The bridge from the smooth case to the rough one — quantile regression, the median, GMM with kinks — is stochastic equicontinuity over a Donsker class 45.05.03, which moves the differentiability from each rough integrand to the smooth population map $\theta \mapsto P{\psi }_{\theta }$ and keeps the sandwich intact.

Full proof set Master

The asymptotic-normality master theorem is proved in the Key theorem; the influence-function identity, the GMM variance, the Huber sandwich, and the equicontinuity repair are Exercises 5, 7, 4, and 8. The remaining Master claims are recorded here.

Proposition 1 (Z-estimator consistency via a well-separated root). Suppose $\Psi (\theta )=P{\psi }_{\theta }$ is continuous with $\Psi ({\theta }_0)=0$, the root is well-separated — for every $\epsilon >0$, ${\inf }_{|\theta -{\theta }_0|\ge \epsilon }|\Psi (\theta )|>0$ — a uniform law of large numbers holds, ${\sup }_{\theta }\Vert {\Psi }_n(\theta )-\Psi (\theta )\Vert \stackrel{\mathbb{P}}{\to }0$, and ${\hat{\theta }}_n$ nearly solves, ${\Psi }_n({\hat{\theta }}_n)=o_{\mathbb{P}}(1)$. Then ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$.

Proof. By the triangle inequality and uniform convergence, $\Vert \Psi ({\hat{\theta }}_n)\Vert \le \Vert \Psi ({\hat{\theta }}_n)-{\Psi }_n({\hat{\theta }}_n)\Vert +\Vert {\Psi }_n({\hat{\theta }}_n)\Vert \le {\sup }_{\theta }\Vert {\Psi }_n-\Psi \Vert +o_{\mathbb{P}}(1)=o_{\mathbb{P}}(1)$. Fix $\epsilon >0$ and set $\eta ={\inf }_{|\theta -{\theta }_0|\ge \epsilon }\Vert \Psi (\theta )\Vert >0$ by separation. On the event $\{|{\hat{\theta }}_n-{\theta }_0|\ge \epsilon \}$ we have $\Vert \Psi ({\hat{\theta }}_n)\Vert \ge \eta$, so $\mathrm{Pr}(|{\hat{\theta }}_n-{\theta }_0|\ge \epsilon )\le \mathrm{Pr}(\Vert \Psi ({\hat{\theta }}_n)\Vert \ge \eta )\to 0$. Hence ${\hat{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$. The argmax theorem (Theorem 1) is the criterion-function counterpart, with $M({\theta }_0)-M(\theta )$ playing the role of $\Vert \Psi (\theta )\Vert$. $\square$

Proposition 2 (the estimating-function CLT and the sensitivity LLN). Under (Z2)-(Z3), $\sqrt{n}{\Psi }_n({\theta }_0)\stackrel{d}{\to }N(0,B)$ and, for any ${\theta }_n^{\ast }\stackrel{\mathbb{P}}{\to }{\theta }_0$ with ${\theta }_n^{\ast }\in U$ eventually, $-P_n{\nabla }_{\theta }{\psi }_{{\theta }_n^{\ast }}\stackrel{\mathbb{P}}{\to }A$.

Proof. The summands ${\psi }_{{\theta }_0}(X_i)$ are i.i.d. with mean $P{\psi }_{{\theta }_0}=0$ and finite covariance $B$, so the multivariate Lindeberg-Lévy central limit theorem 37.03.02 gives $n^{-1/2}{\sum }_i{\psi }_{{\theta }_0}(X_i)\stackrel{d}{\to }N(0,B)$. For the second claim, the LLN gives $-P_n{\nabla }_{\theta }{\psi }_{{\theta }_0}\stackrel{\mathbb{P}}{\to }-P{\nabla }_{\theta }{\psi }_{{\theta }_0}=A$; the domination (Z2) yields the Lipschitz bound $\Vert P_n{\nabla }_{\theta }{\psi }_{{\theta }_n^{\ast }}-P_n{\nabla }_{\theta }{\psi }_{{\theta }_0}\Vert \le |{\theta }_n^{\ast }-{\theta }_0|P_n\dot{m}$, and $P_n\dot{m}\stackrel{\mathbb{P}}{\to }P\dot{m}<\infty$ while $|{\theta }_n^{\ast }-{\theta }_0|\stackrel{\mathbb{P}}{\to }0$, so the difference is $o_{\mathbb{P}}(1)$ and $-P_n{\nabla }_{\theta }{\psi }_{{\theta }_n^{\ast }}\stackrel{\mathbb{P}}{\to }A$. $\square$

Proposition 3 (the asymptotic-normality assembly). Under (Z1)-(Z3), $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }N(0,A^{-1}B{A}^{-\mathsf{T}})$.

Proof. On the probability-tending-to-one event that ${\hat{\theta }}_n$ is an interior root, the mean-value expansion gives $0={\Psi }_n({\theta }_0)+{\nabla }_{\theta }{\Psi }_n({\bar{\theta }}_n)({\hat{\theta }}_n-{\theta }_0)$ with ${\bar{\theta }}_n$ between ${\theta }_0$ and ${\hat{\theta }}_n$, hence ${\bar{\theta }}_n\stackrel{\mathbb{P}}{\to }{\theta }_0$. Thus $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=[-{\nabla }_{\theta }{\Psi }_n({\bar{\theta }}_n){]}^{-1}\sqrt{n}{\Psi }_n({\theta }_0)$. By Proposition 2 the matrix factor $-{\nabla }_{\theta }{\Psi }_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }A$, nonsingular, so its inverse converges to $A^{-1}$ by continuous mapping; the vector factor converges in distribution to $N(0,B)$. Slutsky combines them: $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }{A}^{-1}N(0,B)=N(0,A^{-1}B{A}^{-\mathsf{T}})$. $\square$

Proposition 4 (the influence-function representation and the variance). Under (Z1)-(Z3), $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)={\mathbb{G}}_n\mathrm{IF}+o_{\mathbb{P}}(1)$ with $\mathrm{IF}(x)=A^{-1}{\psi }_{{\theta }_0}(x)$ and ${\mathbb{G}}_{n}h=\sqrt{n}(P_n-P)h$, and the asymptotic variance is $P(\mathrm{IF}{\mathrm{IF}}^{\mathsf{T}})=A^{-1}B{A}^{-\mathsf{T}}$.

Proof. From the expansion of Proposition 3, $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=A_n({\bar{\theta }}_n{)}^{-1}\sqrt{n}{\Psi }_n({\theta }_0)$ with $A_n({\bar{\theta }}_n)\stackrel{\mathbb{P}}{\to }A$, so $A_n({\bar{\theta }}_n{)}^{-1}=A^{-1}+o_{\mathbb{P}}(1)$. Since $\sqrt{n}{\Psi }_n({\theta }_0)={\mathbb{G}}_n{\psi }_{{\theta }_0}$ (as $P{\psi }_{{\theta }_0}=0$) is $O_{\mathbb{P}}(1)$, the order calculus of 45.04.01 gives $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)=A^{-1}{\mathbb{G}}_n{\psi }_{{\theta }_0}+o_{\mathbb{P}}(1)O_{\mathbb{P}}(1)={\mathbb{G}}_n(A^{-1}{\psi }_{{\theta }_0})+o_{\mathbb{P}}(1)={\mathbb{G}}_n\mathrm{IF}+o_{\mathbb{P}}(1)$. The variance is $\mathrm{Cov}(A^{-1}{\psi }_{{\theta }_0})=A^{-1}B{A}^{-\mathsf{T}}=P(\mathrm{IF}{\mathrm{IF}}^{\mathsf{T}})$, the influence function's second moment, since $P\mathrm{IF}=A^{-1}P{\psi }_{{\theta }_0}=0$. $\square$

Proposition 5 (efficiency of the score among Fisher-consistent $\psi$). For a correctly specified regular model and any Fisher-consistent estimating function ${\psi }_{\theta }$ with finite variability and nonsingular sensitivity, the sandwich satisfies $A^{-1}B{A}^{-\mathsf{T}}⪰I({\theta }_0{)}^{-1}$, with equality iff ${\psi }_{{\theta }_0}=C{s}_{{\theta }_0}$ $P$-a.s. for a fixed nonsingular $C$.

Proof. Differentiating $P{\psi }_{\theta }=0$ at ${\theta }_0$ and integrating by parts (the regularity of 45.04.03) gives the cross-identity $A=-P{\nabla }_{\theta }{\psi }_{{\theta }_0}=P({\psi }_{{\theta }_0}{s}_{{\theta }_0}^{\mathsf{T}})$. Consider the centred random vector ${\psi }_{{\theta }_0}-A{I}^{-1}{s}_{{\theta }_0}$, where $I=I({\theta }_0)=P(s_{{\theta }_0}{s}_{{\theta }_0}^{\mathsf{T}})$. Its covariance is positive semidefinite:

$$0⪯\mathrm{Cov}({\psi }_{{\theta }_0}-A{I}^{-1}{s}_{{\theta }_0})=B-A{I}^{-1}{A}^{\mathsf{T}}-A{I}^{-1}{A}^{\mathsf{T}}+A{I}^{-1}I{I}^{-1}{A}^{\mathsf{T}}=B-A{I}^{-1}{A}^{\mathsf{T}},$$

using $P({\psi }_{{\theta }_0}{s}_{{\theta }_0}^{\mathsf{T}})=A$ and $P(s_{{\theta }_0}{s}_{{\theta }_0}^{\mathsf{T}})=I$. Hence $B⪰A{I}^{-1}{A}^{\mathsf{T}}$, and conjugating by $A^{-1}$ (which preserves the Loewner order) gives $A^{-1}B{A}^{-\mathsf{T}}⪰I^{-1}$. Equality holds iff the covariance above is zero, i.e. ${\psi }_{{\theta }_0}=A{I}^{-1}{s}_{{\theta }_0}$ $P$-a.s., a fixed nonsingular linear transform of the score. The score itself ($\psi =s$, $A=B=I$) attains $I^{-1}$, so it is the efficient choice. $\square$

Connections Master

Asymptotic normality and efficiency of the maximum-likelihood estimator 45.04.03 is the rigid special case of the master theorem: with $\psi =s$ the likelihood score, the two Bartlett identities force the sensitivity and variability to coincide, $A=B=I({\theta }_0)$, and the sandwich $A^{-1}B{A}^{-\mathsf{T}}$ collapses to the inverse Fisher information; the efficiency theorem there is the $\psi =s$ corner of Proposition 5 here, and the one-step estimator there is a Newton iterate on exactly the estimating equation studied here.

Consistency of estimators and the modes of stochastic convergence 45.04.01 supplies the proof machinery: the argmax/argmin theorem stated and proved there (Exercise 8 of that unit) is hypothesis (Z1) of the master theorem, the $o_{\mathbb{P}}/O_{\mathbb{P}}$ calculus upgrades the deterministic Taylor remainder to a stochastic one, the continuous-mapping theorem inverts the sensitivity matrix, and Slutsky's lemma fuses the convergent-in-probability sensitivity factor with the convergent-in-distribution estimating-function factor.

The delta method 45.04.02 is the conceptual core of the master theorem: the Z-estimator is the implicitly defined root of ${\Psi }_n=0$, and the linearisation ${\hat{\theta }}_n-{\theta }_0\approx A^{-1}{\Psi }_n({\theta }_0)$ is the implicit-function form of the delta method, with the influence function $A^{-1}{\psi }_{{\theta }_0}$ the gradient through which the empirical fluctuation passes; the asymptotic distribution of any smooth $g({\hat{\theta }}_n)$ is then the first-order delta method applied to the sandwich limit.

The empirical-distribution function and Glivenko-Cantelli classes 45.05.02 deliver the uniform law of large numbers that the argmax theorem (Theorem 1) and the Z-consistency proposition assume: finite bracketing numbers make $\{m_{\theta }\}$ or $\{{\psi }_{\theta }\}$ a Glivenko-Cantelli class, giving ${\sup }_{\theta }\Vert {\Psi }_n(\theta )-\Psi (\theta )\Vert \stackrel{\mathbb{P}}{\to }0$, the substantive input that converts closeness of objective values into closeness of roots.

Donsker classes and stochastic equicontinuity 45.05.03 repair the master theorem when $\psi$ is non-smooth — the median, quantile regression, kinked GMM — by replacing the pointwise Taylor expansion with the linearisation ${\mathbb{G}}_n{\psi }_{{\hat{\theta }}_n}={\mathbb{G}}_n{\psi }_{{\theta }_0}+o_{\mathbb{P}}(1)$ valid over a Donsker class, shifting differentiability from each rough integrand to the smooth population map $\theta \mapsto P{\psi }_{\theta }$ while leaving the sandwich covariance unchanged (Exercise 8).

The chi-squared limits of the Wald, score, and likelihood-ratio tests 45.06.01 are built on this theorem: the Wald statistic is a quadratic form in the asymptotically normal $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)$ with the sandwich as its metric, the robust "sandwich-corrected" tests use ${\hat{A}}_n^{-1}{\hat{B}}_n{\hat{A}}_n^{-\mathsf{T}}$ in place of $I^{-1}$ to stay valid under misspecification, and the information-matrix test compares ${\hat{A}}_n$ against ${\hat{B}}_n$ as a specification diagnostic.

Historical & philosophical context Master

The general M-estimator was introduced by Peter J. Huber in his 1964 Annals of Mathematical Statistics paper on robust location ^{[Huber 1964]}, which defined estimators of location by minimising ${\sum }_i\rho (x_i-\theta )$ for a general convex $\rho$ — the letter $M$ marking the maximum-likelihood-type construction — and solved the minimax problem over an $\epsilon$-contaminated normal neighbourhood, producing the clipped-score estimator that now bears his name. Huber's insight was that the choice of $\rho$ (equivalently of the estimating function $\psi ={\rho }^{\prime }$) is a design variable, and that bounding $\psi$ buys insensitivity to outliers at a controlled cost in efficiency.

The influence function as the organising object of robustness is due to Frank Hampel, whose 1974 Journal of the American Statistical Association paper ^{[Hampel 1974]} defined it as the Gateaux derivative of an estimator functional in the direction of a point mass and characterised B-robustness as its boundedness, with the gross-error sensitivity as the relevant scalar summary; the identity $\mathrm{IF}\propto A^{-1}\psi$ ties Hampel's functional-analytic picture to Huber's estimating-equation one. The Z-estimator viewpoint and the uniform-asymptotics treatment via empirical processes were consolidated by Aad van der Vaart, whose 1998 Asymptotic Statistics states the master theorem and the argmax theorem in the generality used here and supplies the Donsker-class route for non-smooth $\psi$ ^{[van der Vaart — Asymptotic Statistics]}.

The estimating-equation form crossed into econometrics through Lars Peter Hansen, whose 1982 Econometrica paper ^{[Hansen 1982]} introduced the generalized method of moments as an overidentified Z-estimator with an optimal weighting matrix, and Halbert White, whose 1982 Econometrica paper ^{[White 1982]} derived the sandwich covariance under model misspecification and proposed the information-matrix equality $A=B$ as a specification test, establishing the robust standard error as the default in applied work.

Bibliography Master

@book{vanderVaart1998,
  author    = {van der Vaart, Aad W.},
  title     = {Asymptotic Statistics},
  publisher = {Cambridge University Press},
  series    = {Cambridge Series in Statistical and Probabilistic Mathematics},
  year      = {1998}
}

@book{HuberRonchetti2009,
  author    = {Huber, Peter J. and Ronchetti, Elvezio M.},
  title     = {Robust Statistics},
  edition   = {2},
  series    = {Wiley Series in Probability and Statistics},
  publisher = {Wiley},
  year      = {2009}
}

@article{Huber1964,
  author  = {Huber, Peter J.},
  title   = {Robust estimation of a location parameter},
  journal = {Annals of Mathematical Statistics},
  volume  = {35},
  number  = {1},
  year    = {1964},
  pages   = {73--101}
}

@article{Hampel1974,
  author  = {Hampel, Frank R.},
  title   = {The influence curve and its role in robust estimation},
  journal = {Journal of the American Statistical Association},
  volume  = {69},
  number  = {346},
  year    = {1974},
  pages   = {383--393}
}

@article{Hansen1982,
  author  = {Hansen, Lars Peter},
  title   = {Large sample properties of generalized method of moments estimators},
  journal = {Econometrica},
  volume  = {50},
  number  = {4},
  year    = {1982},
  pages   = {1029--1054}
}

@article{White1982,
  author  = {White, Halbert},
  title   = {Maximum likelihood estimation of misspecified models},
  journal = {Econometrica},
  volume  = {50},
  number  = {1},
  year    = {1982},
  pages   = {1--25}
}

@article{Hampel1971,
  author  = {Hampel, Frank R.},
  title   = {A general qualitative definition of robustness},
  journal = {Annals of Mathematical Statistics},
  volume  = {42},
  number  = {6},
  year    = {1971},
  pages   = {1887--1896}
}