Point Estimation: The Method of Moments and Maximum Likelihood

Intuition Beginner

You have a coin that lands heads with some unknown chance, and a stack of data: a hundred flips, sixty heads. What number should you report as your best guess for that chance? Point estimation is the task of turning a pile of data into a single best number for an unknown quantity. The two oldest recipes for doing this are the method of moments and maximum likelihood, and both are surprisingly simple ideas dressed in formal clothing.

The method of moments says: match the data to the model on summary numbers you can both compute. The average of your sixty-out-of-a-hundred flips is $0.6$; the model says the long-run average should equal the unknown chance; so set them equal and read off the guess $0.6$. You match the sample average to the model average, the sample spread to the model spread, and so on, using as many of these summaries as you have unknowns to pin down.

Maximum likelihood asks a different question: of all the possible values for the unknown, which one makes the data you actually saw the least surprising? For each candidate chance, you can compute how probable your exact sequence of sixty heads was. The candidate that scores highest — the one under which your data looks most expected — is your estimate. For the coin this again gives $0.6$, but the recipe is general and tends to be the sharper of the two.

Visual Beginner

Picture a dial you can turn from $0$ to $1$, setting a candidate value for the coin's chance of heads. For each setting, a meter reads off how probable your observed data (sixty heads in a hundred flips) would be at that setting. As you sweep the dial up from zero the meter climbs, peaks, and falls again. Maximum likelihood is the instruction: stop the dial where the meter peaks.

recipe	what you match or maximise	coin estimate
method of moments	sample average set equal to model average	$0.6$
maximum likelihood	the setting making the data most probable	$0.6$

The takeaway: both recipes convert data into a single number. The method of moments matches summaries; maximum likelihood turns the dial to where the data is least surprising. They often agree, and where they differ, the likelihood peak is usually the better aim.

Worked example Beginner

A factory line produces bolts whose lengths vary. You measure five bolts, in millimetres: $50,52,48,51,49$. You believe the lengths follow a bell-shaped pattern with an unknown centre and an unknown spread, and you want best guesses for both.

Step 1. The sample average. Add the five numbers: $50+52+48+51+49=250$, and divide by $5$ to get $250/5=50$. Both recipes agree the best guess for the centre is this average, $50$ millimetres.

Step 2. The sample spread. Take each measurement's distance from the average, square it, and average those: the distances are $0,2,-2,1,-1$, their squares are $0,4,4,1,1$, summing to $10$, and dividing by $5$ gives $10/5=2$. So the best guess for the spread (the variance) is $2$ square millimetres.

Step 3. Read the guesses. The maximum likelihood guess for the centre is $50$ and for the variance is $2$; the method of moments returns the same two numbers here, because matching the sample average and sample spread to the model's average and spread is exactly what maximum likelihood does for the bell-shaped pattern.

Step 4. A note of caution on the spread. Dividing the squared distances by $5$ gives a guess that runs a little small on average, because the measurements were compared to their own average rather than the true centre. A common fix divides by $4$ instead of $5$, giving $10/4=2.5$. The maximum likelihood recipe uses the $5$; the corrected version trades a touch of this small-on-average tendency for a different balance.

What this tells us: feed in five numbers, get out two best guesses. The centre comes straight from the average; the spread comes from the averaged squared distances. The recipe is mechanical, and the only subtlety is that the natural spread estimate is slightly biased small at finite sample size.

Check your understanding Beginner

Formal definition Intermediate+

Let $X_1,\dots ,X_n$ be independent and identically distributed observations from a family of densities (or mass functions) $\{f(x\mid \theta ):\theta \in \Theta \}$ with respect to a fixed dominating measure, where $\Theta \subseteq {\mathbb{R}}^k$ is the parameter space 26.03.01. A point estimator of $\theta$ is a measurable map $\hat{\theta }=T(X_1,\dots ,X_n)$ from the sample into $\Theta$; an estimate is its realised value on observed data. Write $x=(x_1,\dots ,x_n)$ for a realisation.

Definition (method of moments). For $j=1,\dots ,k$ let ${\mu }_j(\theta )={\mathbb{E}}_{\theta }[X^j]$ be the $j$-th population moment and $m_j=\frac{1}{n}{\sum }_{i=1}^n{X}_i^j$ the $j$-th sample moment. The method-of-moments estimator $\tilde{\theta }$ solves the system

$${\mu }_j(\tilde{\theta })=m_j,j=1,\dots ,k,$$

when a solution exists in $\Theta$. The recipe matches the first $k$ theoretical moments to their empirical counterparts; central moments or other identifiable functionals may be substituted when more convenient.

Definition (likelihood and log-likelihood). The likelihood function is the joint density of the sample read as a function of the parameter with the data held fixed,

$$L(\theta \mid x)=\prod _{i=1}^{n}f(x_i\mid \theta ),$$

and the log-likelihood is $\ell (\theta )=\log L(\theta \mid x)={\sum }_{i=1}^n\log f(x_i\mid \theta )$. When $f(\cdot \mid \theta )$ is differentiable in $\theta$, the score is the gradient $s(\theta )={\nabla }_{\theta }\ell (\theta )$, with components $\partial \ell /\partial {\theta }_j$, and the likelihood equations (or score equations) are $s(\theta )=0$.

Definition (maximum likelihood estimator). A maximum likelihood estimator (MLE) is any

$$\hat{\theta }\in \arg \underset{\theta \in \Theta }{\max }L(\theta \mid x)=\arg \underset{\theta \in \Theta }{\max }\ell (\theta ).$$

Because $\log$ is strictly increasing, maximising $L$ and maximising $\ell$ give the same maximiser; the log form turns the product into a sum and is the working object. When $\Theta$ is open and $\ell$ is differentiable, an interior MLE satisfies the score equations, but the MLE is defined as the global maximiser, not as a stationary point — a distinction that matters when the maximum sits on the boundary of $\Theta$ or where the support of $f$ depends on $\theta$.

Definition (invariance / equivariance). If $\hat{\theta }$ is the MLE of $\theta$ and $\tau :\Theta \to {\mathbb{R}}^p$ is any function, the induced likelihood of $\eta =\tau (\theta )$ is $L^{\ast }(\eta \mid x)={\sup }_{\{\theta :\tau (\theta )=\eta \}}L(\theta \mid x)$, and the MLE of $\eta$ is $\hat{\eta }=\tau (\hat{\theta })$. This invariance property lets one estimate any reparametrisation by transforming the original MLE directly.

The new symbols are introduced here: $L(\theta \mid x)$ (likelihood), $\ell (\theta )$ (log-likelihood), $s(\theta )$ (score), ${\mu }_j(\theta )$ (population moment), $m_j$ (sample moment), $\hat{\theta }$ (MLE), $\tilde{\theta }$ (method-of-moments estimator). The mean squared error of an estimator $\hat{\theta }$ of a scalar $\theta$ is ${\mathrm{MSE}}_{\theta }(\hat{\theta })={\mathbb{E}}_{\theta }[(\hat{\theta }-\theta {)}^2]$, and the bias is ${\mathrm{bias}}_{\theta }(\hat{\theta })={\mathbb{E}}_{\theta }[\hat{\theta }]-\theta$.

Counterexamples to common slips Intermediate+

• "The MLE is whatever solves the score equations." Only when the maximum is interior and the family is regular. For $X_1,\dots ,X_n$ uniform on $(0,\theta )$ the likelihood is ${\theta }^{-n}1\{\theta \ge {\max }_i{x}_i\}$, which has no stationary point: it is strictly decreasing wherever it is positive, so the MLE is the left endpoint $\hat{\theta }={\max }_i{x}_i$, found by inspecting the support constraint, not by differentiating.

• "The MLE is unbiased." The normal-variance MLE ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(X_i-\bar{X}{)}^2$ has $\mathbb{E}[{\hat{\sigma }}^2]=\frac{n-1}{n}{\sigma }^2<{\sigma }^2$: it is biased low. Maximum likelihood optimises the likelihood, not unbiasedness; the two criteria genuinely differ at finite $n$.

• "The method of moments and the MLE always agree." They coincide for the Bernoulli and (in mean and variance) the normal, but diverge in general. For the uniform on $(0,\theta )$ the method of moments gives $\tilde{\theta }=2\bar{X}$ while the MLE gives $\hat{\theta }={\max }_i{X}_i$; the two can even order differently on a given sample.

• "A method-of-moments estimate always lands in the parameter space." Matching moments solves an equation that need not have a solution inside $\Theta$. A moment estimate of a variance can come out negative, or a probability can exceed one; the recipe offers no guarantee of feasibility, whereas the MLE is by construction an element of $\Theta$.

Key theorem with proof Intermediate+

The signature result is that for a full-rank exponential family the maximum likelihood estimator is exactly the parameter value whose model moments match the observed sufficient statistic. This single identity unifies the two recipes, supplies the score equations in closed form, and — through the strict concavity of the log-likelihood in the natural parameter — settles existence and uniqueness.

Theorem (exponential-family likelihood equation: moment matching). Let $X_1,\dots ,X_n$ be i.i.d. from a full-rank exponential family with density

$$f(x\mid \eta )=h(x)\exp ({\eta }^{\top }T(x)-A(\eta )),\eta \in \mathcal{N}\subseteq {\mathbb{R}}^k,$$

in natural parameter $\eta$, with $T(x)=(T_1(x),\dots ,T_k(x))$ the natural sufficient statistic, $A(\eta )=\log \int h(x)\exp ({\eta }^{\top }T(x))dx$ the log-partition function, and $\mathcal{N}$ the open natural parameter space. Then the log-likelihood is strictly concave in $\eta$, and the maximum likelihood estimator $\hat{\eta }$, when it exists in $\mathcal{N}$, is the unique solution of the likelihood equation

$$\nabla A(\hat{\eta })=\frac{1}{n}\sum _{i=1}^{n}T(x_i),\text{equivalently}{\mathbb{E}}_{\hat{\eta }}[T(X)]=\frac{1}{n}\sum _{i=1}^{n}T(x_i).$$

Proof. Write $\bar{T}=\frac{1}{n}{\sum }_{i=1}^{n}T(x_i)$. The log-likelihood is

$$\ell (\eta )=\sum _{i=1}^n(\log h(x_i)+{\eta }^{\top }T(x_i)-A(\eta ))=\text{const}+n({\eta }^{\top }\bar{T}-A(\eta )).$$

Standard properties of the log-partition function, obtained by differentiating $A(\eta )=\log \int h(x)e^{{\eta }^{\top }T(x)}dx$ under the integral sign (valid on the open set $\mathcal{N}$, where the integral is finite and differentiation under the integral is justified by dominated convergence), give the cumulant identities

$$\nabla A(\eta )={\mathbb{E}}_{\eta }[T(X)],{\nabla }^{2}A(\eta )={\mathrm{Cov}}_{\eta }(T(X)).$$

For the first, ${\partial }_{j}A=(\int h{e}^{{\eta }^{\top }T}{)}^{-1}\int T_{j}h{e}^{{\eta }^{\top }T}={\mathbb{E}}_{\eta }[T_j]$. For the second, differentiating once more produces ${\mathbb{E}}_{\eta }[T_j{T}_l]-{\mathbb{E}}_{\eta }[T_j]{\mathbb{E}}_{\eta }[T_l]={\mathrm{Cov}}_{\eta }(T_j,T_l)$. The Hessian of $\ell$ is therefore

$${\nabla }^2\ell (\eta )=-n{\nabla }^{2}A(\eta )=-n{\mathrm{Cov}}_{\eta }(T(X)),$$

which is negative definite because full rank means $T_1,\dots ,T_k$ are affinely independent, so the covariance matrix of $T(X)$ is positive definite (no nonzero linear combination $a^{\top }T$ is almost-surely constant). Hence $\ell$ is strictly concave on the convex set $\mathcal{N}$. A strictly concave differentiable function on an open convex set has at most one stationary point, and any such point is its unique global maximiser. Setting the gradient to zero,

$$\nabla \ell (\eta )=n(\bar{T}-\nabla A(\eta ))=0⟺\nabla A(\eta )=\bar{T}⟺{\mathbb{E}}_{\eta }[T(X)]=\bar{T},$$

which is the stated likelihood equation, and its solution $\hat{\eta }$, when it lies in $\mathcal{N}$, is the unique MLE. $\square$

Bridge. This moment-matching identity is the foundational reason the method of moments and maximum likelihood coincide for exponential families: the MLE equates the model expectation of the sufficient statistic to its sample average, which is precisely a moment equation, so the likelihood recipe is a moment recipe on the canonical statistic $T$. It builds toward the computational theory, because the same gradient $\nabla A(\eta )-\bar{T}$ and Hessian ${\mathrm{Cov}}_{\eta }(T)$ are exactly what a Newton or Fisher-scoring iteration needs, and it appears again in the sufficiency and information geometry of the co-produced sufficiency unit, where $T$ carries all the data's information about $\eta$. This is exactly the convexity that the optimization theory exploits: the central insight is that strict concavity of $\ell$ in the natural parameter turns "find the MLE" into a convex program with a unique solution, so existence-and-uniqueness questions reduce to whether $\bar{T}$ lies in the interior of the range of $\nabla A$. Putting these together, the boundary cases (uniform endpoint) are exactly the non-exponential, support-dependent families where this convex picture and its score equations break down, which is why they must be handled by direct maximisation rather than differentiation, and the bias and finite-sample behaviour studied next are read off from the same $A$ and $T$.

Exercises Intermediate+

Advanced results Master

The results below organise finite-sample point estimation around the exponential-family moment identity of the Key theorem, the equivariance structure that makes the MLE coordinate-free, the bias-variance accounting that compares the two recipes at fixed $n$, and the computational machinery that takes over when no closed form exists. Asymptotic normality and efficiency belong to the large-sample theory and are taken up in 45.04.03.

Theorem 1 (full equivariance of the MLE). The MLE is equivariant in two senses. Reparametrisation: for any $\tau :\Theta \to {\mathbb{R}}^p$, the MLE of $\eta =\tau (\theta )$ is $\tau (\hat{\theta })$, via the induced likelihood $L^{\ast }(\eta )={\sup }_{\tau (\theta )=\eta }L(\theta )$. Data transformation: if $Y=g(X)$ for a smooth invertible $g$, the MLE of $\theta$ computed from $Y$ equals that computed from $X$, because the Jacobian factor in the transformed density does not depend on $\theta$ and so cancels in the argmax. Equivariance is the structural reason the MLE behaves coherently under changes of units and of parametrisation, a property the method of moments lacks: matching moments in $\theta$ and in $\tau (\theta )$ generally yields incompatible estimates ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Theorem 2 (mean squared error decomposition and recipe comparison). For an estimator $\hat{\theta }$ of a scalar $\theta$, ${\mathrm{MSE}}_{\theta }(\hat{\theta })={\mathrm{bias}}_{\theta }(\hat{\theta }{)}^2+{\mathrm{Var}}_{\theta }(\hat{\theta })$. This decomposition is the finite-sample yardstick: an estimator with small bias but large variance can be worse than a biased one with small variance. For the uniform-$(0,\theta )$ family the MLE $X_{(n)}$ has $\mathrm{MSE}=2{\theta }^2/[(n+1)(n+2)]=O(n^{-2})$, while the moment estimator $2\bar{X}$ has $\mathrm{MSE}={\theta }^2/(3n)=O(n^{-1})$; the MLE is the sharper estimator by an order in $n$, and its bias-corrected rescaling $\frac{n+1}{n}{X}_{(n)}$ improves the constant further. The normal-variance MLE ${\hat{\sigma }}^2$ trades a small negative bias for a variance slightly below that of the unbiased $S^2$, and in MSE the MLE-style divisor $n+1$ beats both $n$ and $n-1$ for the normal variance ^{[Casella, G. & Berger, R. L. — Statistical Inference (2nd ed.)]}.

Theorem 3 (existence and uniqueness via concavity). For a full-rank exponential family in the natural parameter, the log-likelihood $\ell (\eta )=n({\eta }^{\top }\bar{T}-A(\eta ))$ is strictly concave on the open natural space $\mathcal{N}$, with ${\nabla }^2\ell =-n{\mathrm{Cov}}_{\eta }(T)\prec 0$. The MLE exists and is unique precisely when the observed mean statistic $\bar{T}$ lies in the interior of the convex hull of the support of the distribution of $T$ — equivalently in the range of the moment map $\nabla A$. When $\bar{T}$ falls on the boundary (for instance all-successes data in a binomial, giving $\hat{p}=1$ on the parameter boundary), the supremum of $\ell$ is not attained in the open set and the MLE escapes to the boundary of $\Theta$. The concave-program picture is exactly what makes the computational methods below globally convergent ^{[Lehmann, E. L. & Casella, G. — Theory of Point Estimation (2nd ed.)]}.

Theorem 4 (computation: Newton and Fisher scoring). When the likelihood equation $\nabla \ell (\theta )=0$ has no closed-form solution, the MLE is computed by iterating a Newton step on $\ell$. Newton's method for optimization 44.03.03 uses ${\theta }_{t+1}={\theta }_t-[{\nabla }^2\ell ({\theta }_t){]}^{-1}\nabla \ell ({\theta }_t)$, with the observed-information matrix $-{\nabla }^2\ell$ in the curvature role; Fisher scoring replaces $-{\nabla }^2\ell ({\theta }_t)$ by its expectation, the Fisher information $I({\theta }_t)={\mathbb{E}}_{\theta }[-{\nabla }^2\ell ]$, giving ${\theta }_{t+1}={\theta }_t+I({\theta }_t{)}^{-1}\nabla \ell ({\theta }_t)$. For a full-rank exponential family the Hessian $-n{\mathrm{Cov}}_{\eta }(T)$ is deterministic, so observed and expected information coincide and Newton equals Fisher scoring; strict concavity guarantees convergence to the unique maximiser from any start once the iterate enters the basin where the local quadratic model is accurate ^{[Casella, G. & Berger, R. L. — Statistical Inference (2nd ed.)]}.

Theorem 5 (computation with latent structure: the EM principle). When the data are incomplete — mixtures, censoring, latent-variable models — the likelihood $\ell (\theta )=\log \int p(x,z\mid \theta )dz$ marginalises over hidden $z$ and is typically multimodal with no score-equation closed form. The expectation-maximisation algorithm 45.08.07 ascends $\ell$ by alternately computing the expected complete-data log-likelihood $Q(\theta \mid {\theta }_t)={\mathbb{E}}_{z\mid x,{\theta }_t}[\log p(x,z\mid \theta )]$ (E-step) and maximising it (M-step); each iteration increases the observed-data likelihood, and for exponential-family complete data the M-step is again the moment-matching equation ${\mathbb{E}}_{\theta }[T(X,Z)]={\mathbb{E}}_{z\mid x,{\theta }_t}[\bar{T}(X,Z)]$, the latent generalisation of the Key theorem's identity ^{[Casella, G. & Berger, R. L. — Statistical Inference (2nd ed.)]}.

Synthesis. Finite-sample point estimation organises around one identity and its failure modes. The foundational reason maximum likelihood and the method of moments agree for exponential families is the Key theorem's equation ${\mathbb{E}}_{\hat{\eta }}[T(X)]=\bar{T}$: the MLE is a moment estimator on the canonical statistic, and this is exactly the moment-matching the EM M-step reproduces with latent data. The central insight is that strict concavity of the log-likelihood in the natural parameter converts estimation into a convex program with a unique solution, which is what makes Newton, Fisher scoring, and EM globally well behaved — the computation theory of 44.03.03 and 45.08.07 is the same concave maximisation seen through an algorithm. Equivariance generalises this coherence: the MLE of $\tau (\theta )$ is $\tau (\hat{\theta })$ because the maximiser, not the moment list, is the coordinate-free object, and this is dual to the method of moments, which fixes a coordinate system and pays for it when the parametrisation changes. Putting these together, the bias-variance accounting is the finite-sample price tag — the MLE need not be unbiased, the normal variance and the uniform endpoint both run low, and the mean squared error ranks the recipes. The bridge forward is that everything quantitative about how fast these estimators concentrate — consistency made rigorous, asymptotic normality, and the efficiency the Fisher information measures — is the large-sample theory of 45.04.03, for which the score, the information $I(\theta )$, and the concavity assembled here are the exact inputs.

Full proof set Master

Proposition 1 (invariance of the MLE under reparametrisation). If $\hat{\theta }$ maximises $L(\theta \mid x)$ over $\Theta$ and $\tau :\Theta \to {\mathbb{R}}^p$, then $\tau (\hat{\theta })$ maximises the induced likelihood $L^{\ast }(\eta \mid x)={\sup }_{\{\theta :\tau (\theta )=\eta \}}L(\theta \mid x)$.

Proof. The function $\tau$ partitions $\Theta$ into level sets ${\Theta }_{\eta }=\{\theta :\tau (\theta )=\eta \}$ indexed by $\eta \in \tau (\Theta )$. By definition $L^{\ast }(\eta )={\sup }_{\theta \in {\Theta }_{\eta }}L(\theta )$, and

$$\underset{\eta }{\sup }{L}^{\ast }(\eta )=\underset{\eta }{\sup }\underset{\theta \in {\Theta }_{\eta }}{\sup }L(\theta )=\underset{\theta \in \Theta }{\sup }L(\theta )=L(\hat{\theta }),$$

since the level sets cover $\Theta$. Let $\hat{\eta }=\tau (\hat{\theta })$. Then $\hat{\theta }\in {\Theta }_{\hat{\eta }}$, so $L^{\ast }(\hat{\eta })={\sup }_{\theta \in {\Theta }_{\hat{\eta }}}L(\theta )\ge L(\hat{\theta })={\sup }_{\eta }{L}^{\ast }(\eta )$, and the reverse inequality holds because $L^{\ast }(\hat{\eta })$ is one of the values being supremised. Therefore $L^{\ast }(\hat{\eta })={\sup }_{\eta }{L}^{\ast }(\eta )$, i.e. $\hat{\eta }=\tau (\hat{\theta })$ maximises $L^{\ast }$. $\square$

Proposition 2 (exponential-family likelihood equation; moment matching). Under the hypotheses of the Key theorem, $\hat{\eta }$ satisfies ${\mathbb{E}}_{\hat{\eta }}[T(X)]=\bar{T}$ and is the unique maximiser.

Proof. As in the Key theorem, $\ell (\eta )=\text{const}+n({\eta }^{\top }\bar{T}-A(\eta ))$ with $\nabla A(\eta )={\mathbb{E}}_{\eta }[T]$ and ${\nabla }^{2}A(\eta )={\mathrm{Cov}}_{\eta }(T)\succ 0$ by full rank. Thus $\nabla \ell (\eta )=n(\bar{T}-{\mathbb{E}}_{\eta }[T])$ and ${\nabla }^2\ell (\eta )=-n{\mathrm{Cov}}_{\eta }(T)\prec 0$, so $\ell$ is strictly concave. A stationary point solves ${\mathbb{E}}_{\eta }[T]=\bar{T}$; strict concavity on the open convex $\mathcal{N}$ makes any stationary point the unique global maximiser. $\square$

Proposition 3 (normal-variance MLE is biased; the Bessel correction). For $X_1,\dots ,X_n$ i.i.d. $N(\mu ,{\sigma }^2)$, the MLE ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(X_i-\bar{X}{)}^2$ has $\mathbb{E}[{\hat{\sigma }}^2]=\frac{n-1}{n}{\sigma }^2$, and $S^2=\frac{1}{n-1}{\sum }_i(X_i-\bar{X}{)}^2$ is unbiased.

Proof. Write $W={\sum }_i(X_i-\bar{X}{)}^2$. Expanding, $W={\sum }_i(X_i-\mu {)}^2-n(\bar{X}-\mu {)}^2$. Taking expectations, $\mathbb{E}[{\sum }_i(X_i-\mu {)}^2]=n{\sigma }^2$ and $\mathbb{E}[(\bar{X}-\mu {)}^2]=\mathrm{Var}(\bar{X})={\sigma }^2/n$, so $\mathbb{E}[W]=n{\sigma }^2-n\cdot {\sigma }^2/n=(n-1){\sigma }^2$. Hence $\mathbb{E}[{\hat{\sigma }}^2]=\mathbb{E}[W]/n=\frac{n-1}{n}{\sigma }^2$, biased low by $-{\sigma }^2/n$, and $\mathbb{E}[S^2]=\mathbb{E}[W]/(n-1)={\sigma }^2$. This identity uses only $\mathbb{E}[X_i]=\mu$, $\mathrm{Var}(X_i)={\sigma }^2$, and independence, so it holds for any such family, not the normal alone. $\square$

Proposition 4 (uniform endpoint MLE: distribution, bias, and MSE). For $X_1,\dots ,X_n$ i.i.d. uniform on $(0,\theta )$, $\hat{\theta }=X_{(n)}$ has density $n{t}^{n-1}/{\theta }^n$ on $[0,\theta ]$, ${\mathbb{E}}_{\theta }[\hat{\theta }]=\frac{n}{n+1}\theta$, and $\mathrm{MSE}=\frac{2{\theta }^2}{(n+1)(n+2)}$.

Proof. Independence gives $\mathbb{P}(X_{(n)}\le t)={\prod }_i\mathbb{P}(X_i\le t)=(t/\theta {)}^n$ for $0\le t\le \theta$, with density $f(t)=n{t}^{n-1}/{\theta }^n$. Then ${\mathbb{E}}_{\theta }[X_{(n)}]={\int }_0^{\theta }tn{t}^{n-1}{\theta }^{-n}dt=\frac{n}{n+1}\theta$ and ${\mathbb{E}}_{\theta }[X_{(n)}^2]={\int }_0^{\theta }{t}^{2}n{t}^{n-1}{\theta }^{-n}dt=\frac{n}{n+2}{\theta }^2$. The variance is $\frac{n}{n+2}{\theta }^2-(\frac{n}{n+1}{)}^2{\theta }^2=\frac{n{\theta }^2}{(n+1{)}^2(n+2)}$, the squared bias is $(\frac{n}{n+1}\theta -\theta {)}^2=\frac{{\theta }^2}{(n+1{)}^2}$, and summing,

$$\mathrm{MSE}=\frac{n{\theta }^2}{(n+1{)}^2(n+2)}+\frac{{\theta }^2}{(n+1{)}^2}=\frac{{\theta }^2}{(n+1{)}^2}\cdot \frac{n+(n+2)}{n+2}=\frac{2{\theta }^2}{(n+1)(n+2)}.$$

This is $O(n^{-2})$, against $O(n^{-1})$ for the moment estimator $2\bar{X}$. $\square$

Proposition 5 (Fisher scoring equals Newton for exponential families). For a full-rank exponential family in the natural parameter, the observed information $-{\nabla }^2\ell (\eta )$ equals the Fisher information $I(\eta )={\mathbb{E}}_{\eta }[-{\nabla }^2\ell (\eta )]$, so the Newton and Fisher-scoring iterations coincide.

Proof. From Proposition 2, ${\nabla }^2\ell (\eta )=-n{\mathrm{Cov}}_{\eta }(T)$, which depends on $\eta$ but not on the data $x$. A deterministic matrix equals its own expectation, so $-{\nabla }^2\ell (\eta )={\mathbb{E}}_{\eta }[-{\nabla }^2\ell (\eta )]=I(\eta )=n{\mathrm{Cov}}_{\eta }(T)$. Both iterations therefore use the same update ${\eta }_{t+1}={\eta }_t+[n{\mathrm{Cov}}_{{\eta }_t}(T){]}^{-1}\nabla \ell ({\eta }_t)$. Strict concavity (${\mathrm{Cov}}_{\eta }(T)\succ 0$) makes this a Newton step on a strictly concave objective, so the local quadratic-convergence theory of 44.03.03 applies once the iterate is near $\hat{\eta }$. $\square$

Connections Master

• Sufficiency and the exponential-family structure (the co-produced unit 45.01.02) is where the moment-matching Key theorem draws its force: the natural sufficient statistic $T$ carries all the sample's information about $\eta$, and the likelihood equation ${\mathbb{E}}_{\hat{\eta }}[T]=\bar{T}$ shows the MLE depends on the data only through $T$. That unit owns the factorisation theorem and the completeness of $T$ that this unit cites when reducing maximum likelihood to a moment condition; the present unit specialises it to the construction and finite-sample behaviour of the estimator itself.

• The strong law of large numbers 37.02.02 is the engine behind the heuristic consistency of both recipes: the sample moments $m_j=\frac{1}{n}{\sum }_i{X}_i^j$ converge almost surely to the population moments ${\mu }_j(\theta )$, so the method-of-moments solution converges to the true $\theta$ when the moment map is continuously invertible, and the average log-likelihood $\frac{1}{n}\ell (\theta )\to {\mathbb{E}}_{{\theta }_0}[\log f(X\mid \theta )]$ converges to a population objective maximised at the truth. The rigorous convergence rates and the asymptotic distribution that this concentration only sketches are developed in the large-sample theory 45.04.03.

• Newton's method for optimization 44.03.03 is the computational backbone when the score equations have no closed form: the MLE is the maximiser of the concave log-likelihood, the observed-information matrix is its curvature, and Fisher scoring is the variant that replaces observed by expected information. The expectation-maximisation algorithm 45.08.07 extends this to latent-variable likelihoods, where each M-step is again the moment-matching equation of this unit's Key theorem applied to the completed data, so the present unit supplies the optimality conditions that both algorithms solve.

• The estimation-and-inference vocabulary of 26.05.01 frames why point estimation matters downstream: a point estimate is the centre around which confidence intervals are built and against which hypothesis tests are calibrated, and the MLE's score and information reappear as the basis of the likelihood-ratio, Wald, and score tests. The mean squared error decomposition introduced here is the finite-sample currency that decision-theoretic comparison of estimators (risk, admissibility, the Cramér-Rao bound) trades in.

Historical & philosophical context Master

The method of moments was systematised by Karl Pearson in the 1890s as the fitting device for his system of frequency curves, equating sample moments to the moments of a candidate distribution to solve for its parameters ^{[Pearson 1894]}. It was the dominant estimation technique of early biometry precisely because it was computationally direct: a few sums and a polynomial solve, with no optimisation. Its weaknesses — non-uniqueness of which moments to use, estimates that can leave the parameter space, and a loss of efficiency relative to likelihood — became the motivation for a different principle.

That principle was supplied by Ronald A. Fisher, who introduced maximum likelihood and the surrounding concepts of likelihood, sufficiency, and information in his 1922 paper On the mathematical foundations of theoretical statistics and developed them through the 1920s ^{[Fisher 1922; Fisher 1925]}. Fisher argued that the likelihood, rather than any list of moments, is the proper summary of what the data say about a parameter, and that the value maximising it is the estimate that extracts the most information. The exponential-family moment-matching identity — that the MLE equates the sufficient statistic to its expectation — sits at the junction of Pearson's and Fisher's programmes, recovering a moment equation as the likelihood equation for the families where a low-dimensional sufficient statistic exists, a structure later codified by Koopman, Darmois, and Pitman around 1935-1936 ^{[Darmois 1935; Koopman 1936]}. The non-regular boundary case — the uniform endpoint, where the support depends on the parameter and the score equation fails — was the standard counterexample by which Fisher and his successors marked the limits of the differentiate-and-solve recipe, and it remains the textbook illustration that the MLE is a maximiser, not a stationary point.

Bibliography Master

@book{casellaberger2002,
  author    = {Casella, George and Berger, Roger L.},
  title     = {Statistical Inference},
  edition   = {2},
  publisher = {Duxbury / Thomson Learning},
  year      = {2002}
}

@book{lehmanncasella1998,
  author    = {Lehmann, Erich L. and Casella, George},
  title     = {Theory of Point Estimation},
  edition   = {2},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {1998}
}

@book{wasserman2004,
  author    = {Wasserman, Larry},
  title     = {All of Statistics: A Concise Course in Statistical Inference},
  series    = {Springer Texts in Statistics},
  publisher = {Springer},
  year      = {2004}
}

@article{fisher1922,
  author  = {Fisher, Ronald A.},
  title   = {On the mathematical foundations of theoretical statistics},
  journal = {Philosophical Transactions of the Royal Society of London, Series A},
  volume  = {222},
  year    = {1922},
  pages   = {309--368}
}

@article{fisher1925,
  author  = {Fisher, Ronald A.},
  title   = {Theory of statistical estimation},
  journal = {Proceedings of the Cambridge Philosophical Society},
  volume  = {22},
  year    = {1925},
  pages   = {700--725}
}

@article{pearson1894,
  author  = {Pearson, Karl},
  title   = {Contributions to the mathematical theory of evolution},
  journal = {Philosophical Transactions of the Royal Society of London, Series A},
  volume  = {185},
  year    = {1894},
  pages   = {71--110}
}

@article{darmois1935,
  author  = {Darmois, Georges},
  title   = {Sur les lois de probabilit\'e \`a estimation exhaustive},
  journal = {Comptes Rendus de l'Acad\'emie des Sciences},
  volume  = {200},
  year    = {1935},
  pages   = {1265--1266}
}

@article{koopman1936,
  author  = {Koopman, Bernard O.},
  title   = {On distributions admitting a sufficient statistic},
  journal = {Transactions of the American Mathematical Society},
  volume  = {39},
  year    = {1936},
  pages   = {399--409}
}