Linear and Quadratic Discriminant Analysis and Logistic Regression

Intuition Beginner

You have measurements on some objects and a label attached to each one — emails marked spam or not, tumours marked benign or malignant, irises sorted into species. A classifier is a rule that takes a fresh, unlabelled object and guesses its label from its measurements. This unit is about the simplest honest rules: ones that carve the measurement space into regions with a straight (or gently curved) boundary, one region per label.

There are two opposite ways to build such a rule. The first studies each label on its own: it asks, for each species of iris, what petals typically look like, building a little model of "what an object of this label tends to measure." Then a new object is handed to the label whose model fits it best. This is the generative way; linear and quadratic discriminant analysis are its standard members.

The second way skips the per-label modelling and goes straight for the boundary: it asks only "given these measurements, how should the odds of the labels tip?" and tunes a dial — the weights on the measurements — until the predicted odds line up with the labels actually seen. This is the discriminative way, and logistic regression is its workhorse. It never bothers to describe what a typical object of each label looks like; it only describes where to draw the line.

A surprise sits at the centre of the unit: when the generative models are matched bell-shaped clouds of the same shape, the two ways produce a boundary of the same straight form. They disagree only on how to place it — by fitting the clouds, or by fitting the line directly.

Visual Beginner

Approach	What it models	Boundary it produces	Member methods
Generative, shared shape	a bell-shaped cloud per label, same spread	a straight line / flat plane	linear discriminant analysis
Generative, free shape	a bell-shaped cloud per label, each its own spread	a curved boundary	quadratic discriminant analysis
Discriminative	only the odds of the labels given the data	a straight line / flat plane	logistic regression

The picture to keep: two clouds of labelled points. If the clouds have the same shape, the fair dividing line between them is straight; if one cloud is fatter than the other, the fair divider bends. Linear discriminant analysis and logistic regression both draw the straight line, by different means; quadratic discriminant analysis draws the curve.

Worked example Beginner

Two kinds of fish are sorted by a single measurement, body length in centimetres. Kind A lengths average $10$, kind B lengths average $20$, and both kinds spread the same amount around their average. The two kinds are equally common in the catch. A fish $30$ cm long arrives. Which kind is it?

The fair rule, when the two clouds have the same shape and are equally common, is the simplest one: assign the fish to whichever average it is closer to. The midpoint between the averages is $(10+20)/2=15$. A length above $15$ leans toward kind B; a length below $15$ leans toward kind A.

Our fish is $30$ cm. Its distance to the kind-A average is $|30-10|=20$. Its distance to the kind-B average is $|30-20|=10$. The fish is closer to kind B, so the rule calls it kind B.

Now change the catch. Suppose kind A is four times as common as kind B. Being common shifts the boundary toward the rarer kind: you need stronger length-evidence to overrule the fact that most fish are kind A. The boundary moves up from $15$ to about $17.8$ for this spread, but our $30$ cm fish sits well above even the shifted line, so it is still called kind B.

What this tells us: with equal, same-shaped clouds the boundary is a single cut-point halfway between the averages, and how common each kind is slides that cut-point toward the rarer kind. Everything later in the unit is this one cut-point, dressed up for many measurements at once.

Check your understanding Beginner

Formal definition Intermediate+

Let $(X,G)$ be a feature-label pair with $X\in {\mathbb{R}}^p$ and $G\in \{1,\dots ,K\}$. Write the prior class probabilities ${\pi }_k=\mathrm{Pr}(G=k)$ and the class-conditional densities $f_k(x)=f(x\mid G=k)$. By Bayes' rule the posterior is $\mathrm{Pr}(G=k\mid X=x)={\pi }_k{f}_k(x)/{\sum }_{\ell }{\pi }_{\ell }{f}_{\ell }(x)$.

Definition (Bayes classifier). Under $0$-$1$ loss the Bayes classifier assigns

$$\hat{G}(x)=\arg \underset{k}{\max }\mathrm{Pr}(G=k\mid X=x)=\arg \underset{k}{\max }{\pi }_k{f}_k(x).$$

It minimises the misclassification probability among all rules; its risk is the Bayes rate. The set $\{x:{\pi }_k{f}_k(x)={\pi }_{\ell }{f}_{\ell }(x)\}$ is the decision boundary between classes $k$ and $\ell$.

The generative model: discriminant analysis. Take Gaussian class-conditionals $f_k(x)=(2\pi {)}^{-p/2}|{\Sigma }_k{|}^{-1/2}\exp (-\frac{1}{2}(x-{\mu }_k{)}^{\top }{\Sigma }_k^{-1}(x-{\mu }_k))$ 26.03.01. Comparing $\log {\pi }_k{f}_k(x)$ across $k$ and discarding the $k$-free term gives the quadratic discriminant function

$${\delta }_k(x)=-\frac{1}{2}\log |{\Sigma }_k|-\frac{1}{2}(x-{\mu }_k{)}^{\top }{\Sigma }_k^{-1}(x-{\mu }_k)+\log {\pi }_k,$$

and the rule $\hat{G}(x)=\arg {\max }_k{\delta }_k(x)$ is quadratic discriminant analysis (QDA): its boundaries are quadric surfaces. If all classes share a covariance ${\Sigma }_k\equiv \Sigma$, the quadratic term $x^{\top }{\Sigma }^{-1}x$ is common to every class and cancels, leaving the linear discriminant function

$${\delta }_k(x)=x^{\top }{\Sigma }^{-1}{\mu }_k-\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}{\mu }_k+\log {\pi }_k,$$

linear in $x$. The rule $\hat{G}(x)=\arg {\max }_k{\delta }_k(x)$ is linear discriminant analysis (LDA), with hyperplane boundaries. Parameters are estimated by their maximum-likelihood plug-ins 45.01.04: ${\hat{\pi }}_k=n_k/n$, ${\hat{\mu }}_k=n_k^{-1}{\sum }_{g_i=k}{x}_i$, and for LDA the pooled $\hat{\Sigma }=(n-K{)}^{-1}{\sum }_k{\sum }_{g_i=k}(x_i-{\hat{\mu }}_k)(x_i-{\hat{\mu }}_k{)}^{\top }$.

The discriminative model: logistic regression. For $K=2$ classes coded $\{0,1\}$, posit the logit-linear (log-odds) model

$$\log \frac{\mathrm{Pr}(G=1\mid x)}{\mathrm{Pr}(G=0\mid x)}={\beta }_0+{\beta }^{\top }x=:\eta (x),p(x):=\mathrm{Pr}(G=1\mid x)=\sigma (\eta (x))=\frac{1}{1+e^{-\eta (x)}},$$

with $\sigma$ the logistic (sigmoid) function. Absorbing the intercept into $\beta$ and the constant $1$ into $x$, the log-likelihood of an i.i.d. sample is

$$\ell (\beta )=\sum _{i=1}^n[y_i\log p_i+(1-y_i)\log (1-p_i)]=\sum _{i=1}^n[y_i{x}_i^{\top }\beta -\log (1+e^{x_i^{\top }\beta })],$$

and the MLE $\hat{\beta }=\arg {\max }_{\beta }\ell (\beta )$ is found by maximising it. The multinomial (softmax) extension for $K>2$ classes fixes a reference class $K$ and models $\log [\mathrm{Pr}(G=k\mid x)/\mathrm{Pr}(G=K\mid x)]={\beta }_{k0}+{\beta }_k^{\top }x$, giving $\mathrm{Pr}(G=k\mid x)=e^{{\beta }_k^{\top }x}/{\sum }_{\ell }{e}^{{\beta }_{\ell }^{\top }x}$.

Counterexamples to common slips

• LDA is not nearest-centroid. The rule "assign to the nearest class mean in Euclidean distance" matches LDA only when $\Sigma =I$ and the priors are equal. In general the relevant distance is the Mahalanobis distance $(x-{\mu }_k{)}^{\top }{\Sigma }^{-1}(x-{\mu }_k)$, and unequal priors shift the boundary by $\log {\pi }_k$.
• The logit is the log-odds, not the log-probability. Modelling $\log p(x)$ as linear violates the constraint $p\le 1$; the logit $\log [p/(1-p)]$ maps $(0,1)$ onto all of $\mathbb{R}$, which is what makes a linear model on it coherent.
• More classes does not always favour QDA. QDA estimates a separate ${\Sigma }_k$ per class — $Kp(p+1)/2$ covariance parameters — and on small samples this variance can swamp the bias LDA pays for pooling; the pooled estimate is often the better classifier despite its wrong shared-covariance assumption.
• Perfect separation is not a perfect fit. If a hyperplane separates the two classes with no overlap, the logistic likelihood has no maximiser: $\Vert \hat{\beta }\Vert \to \infty$ drives the fitted probabilities to $\{0,1\}$ and the likelihood up without bound. The MLE fails to exist exactly when the data are linearly separable.

Key theorem with proof Intermediate+

The signature computational result is that the maximum-likelihood fit of logistic regression is a sequence of weighted least-squares fits: Newton-Raphson on the log-likelihood is iteratively reweighted least squares (IRLS), reusing the normal-equations machinery of the linear model 45.06.01 with a diagonal weight matrix that the current fitted probabilities supply. The result follows Hastie, Tibshirani & Friedman ^{[Hastie 2009]} §4.4 and McCullagh & Nelder ^{[McCullagh 1989]} Ch. 4.

Theorem (logistic regression by IRLS). Let $X\in {\mathbb{R}}^{n\times (p+1)}$ have rows $x_i^{\top }$ (intercept absorbed) and full column rank, $y\in \{0,1{\}}^n$. With $p_i=\sigma (x_i^{\top }\beta )$, $p=(p_1,\dots ,p_n{)}^{\top }$, and the diagonal weight matrix $W=\mathrm{diag}(p_i(1-p_i))$, the gradient and Hessian of the log-likelihood are

$$\nabla \ell (\beta )=X^{\top }(y-p),{\nabla }^2\ell (\beta )=-X^{\top }WX.$$

The Newton-Raphson update ${\beta }^{\mathrm{new}}=\beta -[{\nabla }^2\ell (\beta ){]}^{-1}\nabla \ell (\beta )$ equals the weighted least-squares solution

$${\beta }^{\mathrm{new}}=(X^{\top }WX{)}^{-1}{X}^{\top }Wz,z=X\beta +W^{-1}(y-p),$$

where $z$ is the working response (the adjusted dependent variable). Each step solves a weighted least-squares problem; hence the name.

Proof. Differentiate $\ell (\beta )={\sum }_i[y_i{x}_i^{\top }\beta -\log (1+e^{x_i^{\top }\beta })]$. The $i$-th summand has gradient $y_i{x}_i-x_i{e}^{x_i^{\top }\beta }/(1+e^{x_i^{\top }\beta })=(y_i-p_i)x_i$, since $\sigma (\eta )=e^{\eta }/(1+e^{\eta })$. Summing, $\nabla \ell (\beta )={\sum }_i(y_i-p_i)x_i=X^{\top }(y-p)$.

For the Hessian, use ${\sigma }^{\prime }(\eta )=\sigma (\eta )(1-\sigma (\eta ))$, so $\partial p_i/\partial \beta =p_i(1-p_i)x_i$. Then

$${\nabla }^2\ell (\beta )=\frac{\partial }{\partial \beta }\sum _i(y_i-p_i)x_i=-\sum _i{p}_i(1-p_i)x_i{x}_i^{\top }=-X^{\top }WX,$$

with $W=\mathrm{diag}(p_i(1-p_i))$. The weights are positive whenever $0<p_i<1$, so $W\succ 0$, and full column rank of $X$ makes $X^{\top }WX\succ 0$, hence invertible.

Substitute into the Newton step:

$${\beta }^{\mathrm{new}}=\beta +(X^{\top }WX{)}^{-1}{X}^{\top }(y-p).$$

Factor $X^{\top }(y-p)=X^{\top }W{W}^{-1}(y-p)$ and write $\beta =(X^{\top }WX{)}^{-1}{X}^{\top }WX\beta$, so

$${\beta }^{\mathrm{new}}=(X^{\top }WX{)}^{-1}[X^{\top }WX\beta +X^{\top }W{W}^{-1}(y-p)]=(X^{\top }WX{)}^{-1}{X}^{\top }W(X\beta +W^{-1}(y-p)),$$

which is $(X^{\top }WX{)}^{-1}{X}^{\top }Wz$ with $z=X\beta +W^{-1}(y-p)$. This is the normal equation for the weighted least-squares regression of $z$ on $X$ with weights $W$. $\square$

Bridge. The proof's engine is that the Hessian $-X^{\top }WX$ is the weighted Gram matrix of the linear model 45.06.01 with the Bernoulli variance $p_i(1-p_i)$ supplying the weights, so each Newton step is the generalized-least-squares solve of that unit applied to the working response $z$. This is exactly the Fisher-scoring identity of 45.01.04 specialised to the canonical-link binomial: because the logistic link is canonical, the observed and expected information coincide and Newton equals Fisher scoring, which is the foundational reason the update reduces to clean weighted least squares rather than something messier. The negative log-likelihood is convex — $X^{\top }WX⪰0$ makes $-\ell$ have a positive-semidefinite Hessian everywhere — so the iteration converges to the unique global maximiser whenever one exists. The result builds toward the regularized fits below, where adding a penalty restores existence and uniqueness on separated or high-dimensional data, and it appears again in the generative/discriminative comparison: putting these together, LDA and logistic regression share the linear log-odds form ${\beta }_0+{\beta }^{\top }x$, and the central insight of the unit is that they differ only in how that single linear function is fitted — generatively from the Gaussian clouds, or discriminatively by this IRLS maximisation of the conditional likelihood.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not host the classification layer, so no module is wired in (lean_status: none). The intended statements — the convexity of the logistic deviance and the IRLS identity — would read roughly as follows.

-- Intended shape; not part of the current Babel Bible Lean build.
-- Requires: the logistic function, a design matrix X, Bernoulli responses y,
-- and the convex-analysis API (`ConvexOn`, `StrictConvexOn`).
variable {n p : ℕ} (X : Matrix (Fin n) (Fin p) ℝ) (y : Fin n → ℝ)

noncomputable def sigmoid (t : ℝ) : ℝ := 1 / (1 + Real.exp (-t))

-- negative log-likelihood (deviance) of logistic regression
noncomputable def negLogLik (β : Fin p → ℝ) : ℝ :=
  ∑ i, (Real.log (1 + Real.exp ((X i) ⬝ᵥ β)) - y i * ((X i) ⬝ᵥ β))

-- convexity of the deviance: Hessian Xᵀ W X is PSD with W = diag(pᵢ(1-pᵢ))
theorem negLogLik_convex : ConvexOn ℝ Set.univ (negLogLik X y) := by
  sorry

-- full column rank upgrades convex to strictly convex (unique MLE when it exists)
theorem negLogLik_strictConvex (hX : Function.Injective X.mulVec) :
    StrictConvexOn ℝ Set.univ (negLogLik X y) := by
  sorry

the Mathlib gap analysis records what is missing: the Bayes-rule optimality for $0$-$1$ loss, the LDA/QDA discriminant derivation from the Gaussian density, Fisher's between/within generalised eigenproblem, the IRLS update ${\beta }^{\mathrm{new}}=(X^{\top }WX{)}^{-1}{X}^{\top }Wz$, and the separation/non-existence dichotomy of the MLE.

Advanced results Master

The generative and discriminative routes both produce a posterior of the same functional form, and identifying where they part is the organising fact of the unit.

Theorem 1 (LDA posterior is logistic). Under shared-covariance Gaussian class-conditionals with two classes, the LDA posterior log-odds is exactly linear:

$$\log \frac{\mathrm{Pr}(G=1\mid x)}{\mathrm{Pr}(G=0\mid x)}=\log \frac{{\pi }_1}{{\pi }_0}-\frac{1}{2}({\mu }_1+{\mu }_0{)}^{\top }{\Sigma }^{-1}({\mu }_1-{\mu }_0)+x^{\top }{\Sigma }^{-1}({\mu }_1-{\mu }_0)=:{\alpha }_0+{\alpha }^{\top }x.$$

This is the logistic model. LDA and logistic regression therefore posit the same linear log-odds; they differ only in how ${\alpha }_0,\alpha$ are estimated. LDA fits the full joint likelihood ${\prod }_i{f}_{g_i}(x_i){\pi }_{g_i}$ — the generative likelihood, which the Gaussian-plus-prior parameters maximise in closed form — whereas logistic regression fits the conditional likelihood ${\prod }_i\mathrm{Pr}(g_i\mid x_i)$, leaving the marginal of $X$ unmodelled. The generative fit uses the marginal information in the $x_i$ (the shape and separation of the clouds) that the conditional fit discards.

Theorem 2 (efficiency-robustness trade). When the Gaussian model holds, LDA is asymptotically more efficient: it uses more of the data's information, and Efron ^{[Efron 1975]} showed logistic regression is roughly one-half to two-thirds as efficient in the regimes he computed. When the Gaussian model fails — skewed features, outliers, qualitative predictors — the LDA covariance and mean estimates are biased and its boundary degrades, while logistic regression, committing to nothing about the marginal of $X$, remains consistent for the log-odds whenever the log-odds is genuinely linear. The discriminative method buys robustness with the efficiency it forgoes.

Theorem 3 (Fisher's reduced-rank discriminant). With $K$ classes in ${\mathbb{R}}^p$, the class means ${\hat{\mu }}_1,\dots ,{\hat{\mu }}_K$ span an affine subspace of dimension at most $K-1$. Fisher's discriminant seeks the directions $a$ maximising the Rayleigh quotient $a^{\top }Ba/a^{\top }Wa$, where $B={\sum }_k{n}_k({\hat{\mu }}_k-\bar{\mu })({\hat{\mu }}_k-\bar{\mu }{)}^{\top }$ is the between-class scatter and $W$ the pooled within-class scatter. The successive maximisers are the leading generalised eigenvectors of the pencil $(B,W)$ 01.01.19; equivalently, whiten by $W^{-1/2}$ so the problem becomes the ordinary eigenproblem of the transformed $B$. Projecting onto the leading $L\le K-1$ such directions and applying LDA in the reduced space is reduced-rank LDA, a supervised dimension reduction that loses no LDA information when $L=K-1$.

Theorem 4 (regularized logistic regression). Adding a penalty to the deviance,

$${\hat{\beta }}_{\lambda }=\arg \underset{\beta }{\min }-\ell (\beta )+\lambda \Omega (\beta ),\Omega \in \{\frac{1}{2}\Vert \beta {\Vert }_2^2,\Vert \beta {\Vert }_1\},$$

serves two ends. The ${\ell }_2$ (ridge) penalty makes the objective strictly convex and coercive, so the minimiser exists and is unique even under linear separation or when $p>n$, shrinking coefficients toward zero. The ${\ell }_1$ (lasso) penalty additionally produces exact zeros, performing simultaneous fitting and variable selection 45.06.06; the optimisation is convex but non-smooth, solved by coordinate descent or proximal methods rather than plain Newton. Both penalties trade a controlled bias for variance reduction, the same bias-variance bargain that shrinkage strikes for the linear model 45.06.01.

Synthesis. The foundational reason the whole subject coheres is the shared linear log-odds ${\alpha }_0+{\alpha }^{\top }x$: LDA derives it from matched Gaussian clouds and logistic regression posits it directly, so the two are dual estimators of one boundary — generative versus discriminative fits of the same model. This is exactly the efficiency-robustness duality that Efron quantified: the generative fit harvests the marginal information in $X$ and wins on efficiency when its Gaussian assumption is true, while the discriminative fit, indifferent to that marginal, wins on robustness when it is false, and putting these together explains why neither dominates in practice. The central insight extends to dimension reduction: Fisher's quotient $a^{\top }Ba/a^{\top }Wa$ is the generalised eigenproblem of 01.01.19 wearing statistical clothing, the between/within scatters playing the roles of the two quadratic forms, so supervised projection and classification are one optimisation seen twice. The IRLS engine generalises beyond the logit — every canonical-link generalized linear model fits by the same weighted least squares, with the Bernoulli variance $p(1-p)$ replaced by the model's variance function — and the bridge to the high-dimensional regime is regularization, which restores the existence the separated-data MLE loses and connects classification to the sparsity theory of 45.06.06. The quadratic case sits at the boundary of this picture: dropping the shared-covariance assumption breaks the linearity, turns the discriminant quadratic, and forfeits the logistic identification, which is the precise sense in which QDA is the generative method without a matching discriminative twin.

Full proof set Master

Proposition 1 (Bayes-rule optimality for $0$-$1$ loss). Among all classifiers $g:{\mathbb{R}}^p\to \{1,\dots ,K\}$, the Bayes classifier $g^{\ast }(x)=\arg {\max }_k\mathrm{Pr}(G=k\mid X=x)$ minimises the misclassification probability $\mathrm{Pr}(g(X)\ne G)$.

Proof. Condition on $X=x$. For any rule $g$, $\mathrm{Pr}(g(X)\ne G\mid X=x)=1-\mathrm{Pr}(G=g(x)\mid X=x)$. This conditional error is minimised, pointwise in $x$, by choosing $g(x)$ to maximise $\mathrm{Pr}(G=g(x)\mid X=x)$, which is the Bayes rule $g^{\ast }(x)$. Taking expectation over $X$, $\mathrm{Pr}(g(X)\ne G)={\mathbb{E}}_X[1-\mathrm{Pr}(G=g(X)\mid X)]\ge {\mathbb{E}}_X[1-\mathrm{Pr}(G=g^{\ast }(X)\mid X)]=\mathrm{Pr}(g^{\ast }(X)\ne G)$, since the integrand is minimised pointwise. $\square$

Proposition 2 (the QDA discriminant and its cancellation to LDA). For Gaussian class-conditionals $N({\mu }_k,{\Sigma }_k)$, the Bayes rule is $\arg {\max }_k{\delta }_k(x)$ with the quadratic discriminant ${\delta }_k$; if ${\Sigma }_k\equiv \Sigma$, ${\delta }_k$ reduces to a linear function of $x$.

Proof. The Bayes rule maximises $\log {\pi }_k{f}_k(x)$. Expanding the Gaussian log-density,

$$\log {\pi }_k{f}_k(x)=\log {\pi }_k-\frac{p}{2}\log (2\pi )-\frac{1}{2}\log |{\Sigma }_k|-\frac{1}{2}(x-{\mu }_k{)}^{\top }{\Sigma }_k^{-1}(x-{\mu }_k).$$

The term $-\frac{p}{2}\log (2\pi )$ is $k$-free and drops from the $\arg \max$, leaving ${\delta }_k(x)$ as stated. When ${\Sigma }_k\equiv \Sigma$, expand $-\frac{1}{2}(x-{\mu }_k{)}^{\top }{\Sigma }^{-1}(x-{\mu }_k)=-\frac{1}{2}{x}^{\top }{\Sigma }^{-1}x+x^{\top }{\Sigma }^{-1}{\mu }_k-\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}{\mu }_k$. The quadratic $-\frac{1}{2}{x}^{\top }{\Sigma }^{-1}x$ and the constant $-\frac{1}{2}\log |\Sigma |$ are $k$-free and drop, leaving ${\delta }_k(x)=x^{\top }{\Sigma }^{-1}{\mu }_k-\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}{\mu }_k+\log {\pi }_k$, linear in $x$. $\square$

Proposition 3 (gradient, Hessian, and convexity of the logistic deviance). With $p_i=\sigma (x_i^{\top }\beta )$ and $W=\mathrm{diag}(p_i(1-p_i))$, $\nabla \ell =X^{\top }(y-p)$ and ${\nabla }^2\ell =-X^{\top }WX⪯0$; under full column rank of $X$ the inequality is strict, so $-\ell$ is strictly convex.

Proof. The per-observation log-likelihood is ${\ell }_i(\beta )=y_i{x}_i^{\top }\beta -\log (1+e^{x_i^{\top }\beta })$. Its gradient is $(y_i-p_i)x_i$ using ${\partial }_{\beta }\log (1+e^{x_i^{\top }\beta })=p_i{x}_i$, and its second derivative is $-p_i(1-p_i)x_i{x}_i^{\top }$ using ${\sigma }^{\prime }=\sigma (1-\sigma )$. Summing gives $\nabla \ell =X^{\top }(y-p)$ and ${\nabla }^2\ell =-{\sum }_i{p}_i(1-p_i)x_i{x}_i^{\top }=-X^{\top }WX$. For any $v$, $v^{\top }{X}^{\top }WXv={\sum }_i{p}_i(1-p_i)(x_i^{\top }v{)}^2\ge 0$ since each weight is positive, so $-\ell$ is convex; equality needs $x_i^{\top }v=0$ for every $i$, i.e. $Xv=0$, which under full column rank forces $v=0$. Hence $X^{\top }WX\succ 0$ and $-\ell$ is strictly convex, with at most one minimiser. $\square$

Proposition 4 (IRLS = Newton-Raphson). The Newton step on $\ell$ equals the weighted least-squares fit ${\beta }^{\mathrm{new}}=(X^{\top }WX{)}^{-1}{X}^{\top }Wz$ with working response $z=X\beta +W^{-1}(y-p)$.

Proof. As established in the Key theorem: ${\beta }^{\mathrm{new}}=\beta -({\nabla }^2\ell {)}^{-1}\nabla \ell =\beta +(X^{\top }WX{)}^{-1}{X}^{\top }(y-p)$. Writing $X^{\top }(y-p)=X^{\top }W{W}^{-1}(y-p)$ and $X^{\top }WX\beta =X^{\top }W(X\beta )$, factor out $(X^{\top }WX{)}^{-1}{X}^{\top }W$ to obtain $(X^{\top }WX{)}^{-1}{X}^{\top }W(X\beta +W^{-1}(y-p))$, the weighted normal equation for regressing $z$ on $X$ with weights $W$. $\square$

Proposition 5 (non-existence of the MLE under separation; ridge restores it). If the data are linearly separable then ${\sup }_{\beta }\ell (\beta )=0$ is not attained. For any $\lambda >0$ the penalised objective $\ell (\beta )-\frac{\lambda }{2}\Vert \beta {\Vert }^2$ attains a unique maximum.

Proof. Suppose ${\beta }^{\ast }$ separates: $x_i^{\top }{\beta }^{\ast }>0$ when $y_i=1$ and $<0$ when $y_i=0$ (intercept absorbed). For $t>0$ and $\beta =t{\beta }^{\ast }$, each term $y_i\log p_i+(1-y_i)\log (1-p_i)\to 0$ as $t\to \infty$ because $p_i\to 1$ on the $y_i=1$ points and $p_i\to 0$ on the $y_i=0$ points; each term is negative for finite $t$, so $\ell (t{\beta }^{\ast })\uparrow 0$ without reaching it, and no finite $\beta$ gives $\ell =0$. Thus the supremum is unattained. For the penalised objective, $-\ell$ is convex (Proposition 3) and $\frac{\lambda }{2}\Vert \beta {\Vert }^2$ is strictly convex and coercive, so $-\ell (\beta )+\frac{\lambda }{2}\Vert \beta {\Vert }^2\to +\infty$ as $\Vert \beta \Vert \to \infty$ and is strictly convex; a strictly convex coercive function on ${\mathbb{R}}^{p+1}$ attains a unique minimum. $\square$

Connections Master

• The linear model and its hat-matrix least-squares machinery 45.06.01 is reused verbatim inside IRLS: each Newton step is the weighted least-squares solve $(X^{\top }WX{)}^{-1}{X}^{\top }Wz$, the generalized-least-squares normal equation of that unit with the Bernoulli variances $p_i(1-p_i)$ as weights and the working response $z$ as the response, so logistic fitting is a loop over linear-model fits.

• The maximum-likelihood and Fisher-scoring theory 45.01.04 supplies the existence-and-uniqueness logic here: the logistic log-likelihood is concave with Hessian $-X^{\top }WX$, and because the logit is the canonical link the observed and expected information coincide, so the Newton iteration of that unit is exactly the IRLS of this one, and the moment-matching score equation $X^{\top }(y-p)=0$ is its specialisation.

• Newton's method for optimization 44.03.03 is the algorithmic backbone of the discriminative fit: IRLS is Newton-Raphson on the concave deviance, inheriting that unit's local quadratic convergence, and the convexity established here is what upgrades local convergence to global convergence toward the unique maximiser.

• Regularized regression and the lasso 45.06.06 extend directly to classification: the ${\ell }_2$ penalty restores the existence the separated-data logistic MLE loses, while the ${\ell }_1$ penalty performs variable selection on the coefficients, the corner geometry and subgradient conditions of that unit carrying over to the logistic deviance in place of the squared-error loss.

• The multivariate normal and covariance machinery 26.03.01 is the generative engine of LDA and QDA: the class-conditional Gaussian densities, their Mahalanobis exponents, and the pooled covariance estimate are the objects whose log-ratio produces the discriminant functions, and the same Gaussian transformation rules govern the sampling behaviour of the plug-in estimates.

• The simultaneous diagonalisation of two quadratic forms and the generalised eigenvalue problem 01.01.19 is precisely Fisher's reduced-rank discriminant: maximising the between/within Rayleigh quotient $a^{\top }Ba/a^{\top }Wa$ is the generalised eigenproblem $Ba=\lambda Wa$, and whitening by $W^{-1/2}$ is the simultaneous-diagonalisation step that turns supervised dimension reduction into an ordinary eigenproblem.

Historical & philosophical context Master

Linear discriminant analysis originates with R. A. Fisher's 1936 study of iris measurements, The use of multiple measurements in taxonomic problems ^{[Fisher 1936]}, which posed the problem of finding the linear combination of features that best separates known groups and solved it through the between-to-within scatter ratio now bearing his name. Fisher framed it without an explicit probability model — as an optimal projection — and the equivalence to the Bayes rule under shared-covariance Gaussians was recognised afterward, by Welch and others, tying the geometric and the generative pictures together. The quadratic generalisation, allowing each class its own covariance, follows directly from the same Gaussian Bayes rule and was standard by the mid-century multivariate-analysis literature.

Logistic regression has a separate lineage. The logistic function was studied by Verhulst in the nineteenth century as a population-growth curve and revived for bioassay by Berkson in the 1940s, who advocated the logit against the then-dominant probit. Its maturation as a general regression method came through the unifying framework of generalized linear models, introduced by Nelder and Wedderburn in 1972 and consolidated in McCullagh and Nelder's Generalized Linear Models ^{[McCullagh 1989]}, which identified the logit as the canonical link for binomial responses and the Fisher-scoring fit as iteratively reweighted least squares. The generative-versus-discriminative comparison was sharpened by B. Efron's 1975 calculation ^{[Efron 1975]} of the asymptotic relative efficiency of logistic regression against normal discriminant analysis, quantifying the cost the conditional likelihood pays when the Gaussian model is true and, implicitly, the robustness it gains when that model is false.

Bibliography Master

@book{hastie2009elements,
  author    = {Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome},
  title     = {The Elements of Statistical Learning: Data Mining, Inference, and Prediction},
  edition   = {2},
  publisher = {Springer},
  year      = {2009},
  note      = {Chapter 4, linear methods for classification}
}

@book{mccullagh1989generalized,
  author    = {McCullagh, Peter and Nelder, John A.},
  title     = {Generalized Linear Models},
  edition   = {2},
  publisher = {Chapman and Hall},
  year      = {1989}
}

@article{fisher1936use,
  author  = {Fisher, Ronald A.},
  title   = {The use of multiple measurements in taxonomic problems},
  journal = {Annals of Eugenics},
  volume  = {7},
  number  = {2},
  pages   = {179--188},
  year    = {1936}
}

@article{efron1975efficiency,
  author  = {Efron, Bradley},
  title   = {The efficiency of logistic regression compared to normal discriminant analysis},
  journal = {Journal of the American Statistical Association},
  volume  = {70},
  number  = {352},
  pages   = {892--898},
  year    = {1975}
}

@article{nelder1972generalized,
  author  = {Nelder, John A. and Wedderburn, Robert W. M.},
  title   = {Generalized linear models},
  journal = {Journal of the Royal Statistical Society, Series A},
  volume  = {135},
  number  = {3},
  pages   = {370--384},
  year    = {1972}
}

@article{berkson1944application,
  author  = {Berkson, Joseph},
  title   = {Application of the logistic function to bio-assay},
  journal = {Journal of the American Statistical Association},
  volume  = {39},
  number  = {227},
  pages   = {357--365},
  year    = {1944}
}