45.06.08 · mathematical-statistics / 06-high-dimensional-regularization

Principal Components Regression and Partial Least Squares

shipped3 tiersLean: none

Anchor (Master): Hastie, Tibshirani & Friedman 2009 The Elements of Statistical Learning 2e (Springer) §3.5 and §3.6 (derived-input directions, the discussion comparing subset selection, ridge, lasso, PCR, and PLS as shrinkage operators); Jolliffe 2002 Principal Component Analysis 2e (Springer) Ch. 8 (PCR, the selection of components, biased regression); Wold, Sjöström & Eriksson 2001 Chemometrics and Intelligent Laboratory Systems 58 (PLS-regression: a basic tool of chemometrics, the NIPALS algorithm)

Intuition Beginner

Imagine you have fifty thermometers scattered around a building, all reporting almost the same temperature. To predict the heating bill you do not need fifty numbers; you need one or two summaries — the average temperature, maybe how fast it is changing. Stuffing all fifty raw readings into a regression is asking for trouble: they overlap so heavily that the fitting machine cannot tell them apart and the weights it picks are wild and unstable, the same collinearity headache from the ridge unit.

Derived-input regression takes a different route than penalizing the weights. It first builds a handful of clean, non-overlapping summary directions out of the predictors, and then regresses on those few summaries instead of the raw predictors. Because the summaries do not overlap, the fit is stable, and because there are only a few of them, the model is simple.

The two methods in this unit differ in how they choose the summaries. Principal components regression picks the directions in which the predictors vary the most, ignoring the thing you want to predict while it chooses them. Partial least squares is greedier: it picks directions that vary a lot and line up with the response, so it consults the answer key while building its summaries.

The everyday picture is packing for a trip. You cannot bring every item, so you pack a few versatile outfits. Principal components packs whatever takes up the most suitcase volume; partial least squares packs whatever you are most likely to actually wear.

Visual Beginner

Method	How the summary directions are chosen	Does it look at the response while choosing?	What it shrinks
Ordinary least squares	no summaries; uses raw predictors	not applicable	nothing — keeps everything
Ridge	no summaries; penalizes weight size	no	every direction, smoothly, weak ones more
Principal components regression	directions of largest predictor variance	no	drops the small-variance directions entirely
Partial least squares	directions of largest covariance with the response	yes	low-variance directions, but uses the response to decide

The single picture to carry forward: ridge turns a smooth dial down on every direction, while principal components regression flips a hard switch — full strength on the top directions, off on the rest. Partial least squares is like principal components but it lets the response vote on which directions count.

Worked example Beginner

Take two predictors that are nearly the same: $x_{1}$ and $x_{2}$ , measured on four items. Suppose after centering, the data are such that the two strong combined patterns are the sum direction $z_{1} = x_{1} + x_{2}$ (where the data spread out a lot) and the difference direction $z_{2} = x_{1} - x_{2}$ (where the data barely move at all, because the two predictors agree).

Say the sum direction has values $z_{1} = (4, 2, - 2, - 4)$ across the four items, and the difference direction has tiny values $z_{2} = (0.1, - 0.1, 0.1, - 0.1)$ . The response is $y = (3, 1, - 1, - 3)$ .

Principal components regression with one component keeps only $z_{1}$ (the big-variance direction) and regresses $y$ on it. The slope is the sum of $z_{1} y$ over the sum of $z_{1}^{2}$ . Sum of $z_{1} y = 12 + 2 + 2 + 12 = 28$ . Sum of $z_{1}^{2} = 16 + 4 + 4 + 16 = 40$ . So the slope on $z_{1}$ is $28/40 = 0.7$ , and the fitted values are $0.7 \times z_{1} = (2.8, 1.4, - 1.4, - 2.8)$ .

What this tells us: by keeping only the stable sum direction and throwing away the jittery difference direction, the fit tracks the response closely $(3, 1, - 1, - 3$ versus $2.8, 1.4, - 1.4, - 2.8)$ without ever having to split credit between the two near-identical raw predictors. The flimsy $z_{2}$ direction, where the data hardly vary, was discarded before it could destabilize anything. That discard-the-small-variance-directions step is the whole idea of principal components regression.

Check your understanding Beginner

Exercise (easy, multiple choice).

How does principal components regression decide which summary directions to keep?

A. By how strongly each direction predicts the response. B. By how much the predictors vary along each direction. C. By how many predictors load on each direction. D. By the size of the regression weights.

Hint

Principal components are the directions of maximal variance in the predictors, chosen before the response is consulted.

Answer

Option B. Principal components regression keeps the directions of largest predictor variance, deciding without looking at the response at all. Feedback if correct: this is exactly why a low-variance but highly predictive direction can be missed — variance is the only criterion. Feedback if wrong: A describes partial least squares, which does consult the response; C and D play no role in forming the components.

Formal definition Intermediate+

Fix a design matrix $X \in R^{n \times p}$ with columns centered to mean zero (and typically scaled to unit variance) and a centered response $y \in R^{n}$ ; the intercept is fitted separately. Write the thin SVD $X = U D V^{⊤}$ 01.01.12 with $U \in R^{n \times r}$ , $V \in R^{p \times r}$ orthonormal columns $u_{j}, v_{j}$ , singular values $d_{1} \geq \dots \geq d_{r} > 0$ , and $r = rank (X)$ . The right singular vectors $v_{j}$ are the principal component directions: $v_{j}$ is the unit vector maximizing the sample variance $Var (X v) = \frac{1}{n} ∥ X v ∥^{2}$ subject to $∥ v ∥ = 1$ and orthogonality to $v_{1}, \dots, v_{j - 1}$ , because the $v_{j}$ are the eigenvectors of $X^{⊤} X = V D^{2} V^{⊤}$ with eigenvalues $d_{j}^{2}$ . The principal components (scores) are the projected columns $z_{j} = X v_{j} = d_{j} u_{j}$ , which are mutually orthogonal: $z_{j}^{⊤} z_{k} = d_{j}^{2} δ_{j k}$ .

Principal components regression (PCR). Choose a number $M \leq r$ of components and regress $y$ on $z_{1}, \dots, z_{M}$ by ordinary least squares. Because the scores are orthogonal, the regression decouples coordinate by coordinate: the coefficient on $z_{m}$ is $\hat{θ}_{m} = z_{m}^{⊤} y / z_{m}^{⊤} z_{m} = (u_{m}^{⊤} y) / d_{m}$ , and the fitted vector is

\overset{y}{^}_{PCR}^{(M)} = m = 1 \sum M z_{m} \hat{θ}_{m} = m = 1 \sum M u_{m} (u_{m}^{⊤} y) .

Back-transforming through $z_{m} = X v_{m}$ gives the coefficient vector in the original predictors,

\hat{β}_{PCR}^{(M)} = m = 1 \sum M v_{m} \hat{θ}_{m} = m = 1 \sum M v_{m} \frac{u _{m}^{⊤} y}{d _{m}} .

Comparing with the SVD form of the ridge estimator 45.06.03, PCR applies the shrinkage factors $f_{m} = 1 {m \leq M}$ — keep the top $M$ directions at full strength, discard the rest entirely — where ridge applies the smooth factors $d_{m}^{2} / (d_{m}^{2} + λ) \in (0, 1)$ . PCR is a hard truncation; ridge is a smooth down-weighting. At $M = r$ both reduce to ordinary least squares.

Partial least squares (PLS). Instead of variance, PLS forms each direction to maximize covariance with the response. Set $X^{(0)} = X$ , $y^{(0)} = y$ . At step $m$ the weight vector is $w_{m} \propto (X^{(m - 1)})^{⊤} y^{(m - 1)}$ , the direction of maximal sample covariance $Cov (X^{(m - 1)} w, y^{(m - 1)})$ over unit $w$ ; the score is $t_{m} = X^{(m - 1)} w_{m}$ ; the response is regressed on $t_{m}$ and both $X^{(m - 1)}$ and $y^{(m - 1)}$ are deflated by subtracting their projections onto $t_{m}$ . This is the NIPALS construction of Wold ^{[Wold 2001]}. The first PLS direction is $w_{1} \propto X^{⊤} y$ , and the span of the first $M$ PLS scores equals the Krylov subspace $K_{M} = span {X^{⊤} y, (X^{⊤} X) X^{⊤} y, \dots, (X^{⊤} X)^{M - 1} X^{⊤} y}$ .

The notation $z_{j}, t_{m}$ for derived scores, $v_{j}, w_{m}$ for direction vectors, $f_{m}$ for shrinkage factors, and $K_{M}$ for the Krylov subspace is recorded in _meta/NOTATION.md.

Counterexamples to common slips

PCR is not scale-invariant and not response-aware. The components are eigenvectors of $X^{⊤} X$ alone, so rescaling a predictor changes them, and a low-variance direction $v_{j}$ with small $d_{j}$ is dropped at modest $M$ even if $v_{j}^{⊤} β$ is large — PCR can discard the very direction that predicts $y$ ^{[Jolliffe 2002]}.
PLS components are not orthogonal in the predictor metric the way principal components are. The PLS scores $t_{m}$ are mutually orthogonal, but the weight vectors $w_{m}$ are conjugate with respect to $X^{⊤} X$ , not Euclidean-orthogonal; PLS is a non-orthogonal change of basis adapted to $y$ .
More components is not automatically better. Both methods interpolate between a high-bias one-component fit and the full OLS fit at $M = r$ ; the bias-variance trade-off has an interior optimum, so $M$ is a tuning parameter, not "as many as possible."
PLS does not always beat PCR or ridge. Using $y$ to choose directions can inflate variance: PLS can place more than unit weight on some high-variance directions, so its per-direction shrinkage factors need not lie in $[0, 1]$ .

Key theorem with proof Intermediate+

Theorem (PCR is projection onto the top principal subspace; the hard-truncation shrinkage identity). With $X = U D V^{⊤}$ and centered $y$ , the $M$ -component PCR fitted vector is the orthogonal projection of $y$ onto $U_{M} = span {u_{1}, \dots, u_{M}}$ , namely $\overset{y}{^}_{PCR}^{(M)} = \sum_{m = 1}^{M} u_{m} (u_{m}^{⊤} y)$ , and the corresponding coefficient vector is $\hat{β}_{PCR}^{(M)} = \sum_{m = 1}^{M} v_{m} (u_{m}^{⊤} y) / d_{m}$ . Consequently PCR is the SVD estimator with shrinkage factors $f_{m} = 1 {m \leq M}$ , in contrast to ridge's $f_{m} = d_{m}^{2} / (d_{m}^{2} + λ)$ 45.06.03. The presentation follows Hastie, Tibshirani & Friedman ^{[Hastie 2009]} §3.5.1.

Proof. The scores $z_{1}, \dots, z_{M}$ are orthogonal, $z_{m}^{⊤} z_{k} = d_{m}^{2} δ_{mk}$ , so the Gram matrix of the regressors is diagonal and the least-squares fit of $y$ on them decouples. The coefficient on $z_{m}$ is

\hat{θ}_{m} = \frac{z _{m}^{⊤} y}{z _{m}^{⊤} z _{m}} = \frac{( d _{m} u _{m} ) ^{⊤} y}{d _{m}^{2}} = \frac{u _{m}^{⊤} y}{d _{m}},

using $z_{m} = X v_{m} = d_{m} u_{m}$ . The fitted vector is $\overset{y}{^}_{PCR}^{(M)} = \sum_{m = 1}^{M} z_{m} \hat{θ}_{m} = \sum_{m = 1}^{M} d_{m} u_{m} \cdot (u_{m}^{⊤} y) / d_{m} = \sum_{m = 1}^{M} u_{m} (u_{m}^{⊤} y)$ . Since ${u_{1}, \dots, u_{M}}$ is orthonormal, $\sum_{m = 1}^{M} u_{m} u_{m}^{⊤}$ is the orthogonal projector onto $U_{M}$ , so $\overset{y}{^}_{PCR}^{(M)} = P_{U_{M}} y$ . Back-transforming, $\hat{β}_{PCR}^{(M)} = \sum_{m = 1}^{M} v_{m} \hat{θ}_{m} = \sum_{m = 1}^{M} v_{m} (u_{m}^{⊤} y) / d_{m}$ . The ridge fitted vector from 45.06.03 is $\sum_{j = 1}^{r} u_{j} [d_{j}^{2} / (d_{j}^{2} + λ)] (u_{j}^{⊤} y)$ ; writing PCR's fit as $\sum_{j = 1}^{r} u_{j} f_{j} (u_{j}^{⊤} y)$ forces $f_{j} = 1$ for $j \leq M$ and $f_{j} = 0$ for $j > M$ , the indicator $1 {j \leq M}$ . Both are special cases of the linear-smoother family $y \mapsto \sum_{j} u_{j} f_{j} (u_{j}^{⊤} y)$ ; PCR uses a step function of the index, ridge a continuous function of the singular value. $□$

Bridge. The proof's engine is the SVD rotation that diagonalizes the regression so the fit becomes a sum of independent one-dimensional projections weighted by factors $f_{j}$ , the same diagonalization that drives the ridge MSE computation of 45.06.03. This builds toward the unified shrinkage-operator comparison below, where subset selection, ridge, PCR, and PLS are read off a single shrinkage-factor profile, and it appears again in 45.06.05 where best-subset selection is the combinatorial analogue of PCR's projection onto a coordinate subspace. The foundational reason PCR controls variance is that projecting onto $U_{M}$ discards the small- $d_{j}$ directions where the OLS variance $σ^{2} / d_{j}^{2}$ explodes — this is exactly the per-coordinate variance of 45.06.03 with the unstable terms set to zero rather than damped. The central insight is that ridge and PCR are the same estimator viewed through different factor profiles: ridge generalises the hard $0/1$ switch into a smooth contraction, so the bridge is the common map $y \mapsto \sum_{j} u_{j} f_{j} (u_{j}^{⊤} y)$ in which the choice of ${f_{j}}$ is the entire content of a derived-input method. Putting these together, PCR is the limit of ridge in which the smooth dial is replaced by a cliff.

Exercises Intermediate+

Exercise 6 (hard, symbolic).

Show that the span of the first $M$ PLS scores is the Krylov subspace $K_{M} = span {X^{⊤} y, (X^{⊤} X) X^{⊤} y, \dots, (X^{⊤} X)^{M - 1} X^{⊤} y}$ , hence that $\hat{β}_{PLS}^{(M)} \in K_{M}$ .

Hint

The first weight is $w_{1} \propto X^{⊤} y$ . Each deflation subtracts a rank-one term built from $X^{⊤} X$ and the previous scores; argue inductively that $w_{m + 1}$ lies in $span {X^{⊤} y, \dots, (X^{⊤} X)^{m} X^{⊤} y}$ .

Answer

By Exercise 3, $w_{1} \propto X^{⊤} y \in K_{1}$ . Inductively suppose $w_{1}, \dots, w_{m}$ span $K_{m}$ . Deflating $X$ by the score $t_{m} = X w_{m}$ replaces $X^{⊤} y$ by $(X^{(m)})^{⊤} y^{(m)}$ , which differs from $X^{⊤} y$ by terms of the form $X^{⊤} X u$ with $u \in span {w_{1}, \dots, w_{m}} = K_{m}$ ; hence $w_{m + 1} \propto (X^{(m)})^{⊤} y^{(m)} \in X^{⊤} y + (X^{⊤} X) K_{m} \subseteq K_{m + 1}$ . Therefore the PLS weight vectors span $K_{M}$ after $M$ steps, and the coefficient $\hat{β}_{PLS}^{(M)}$ , a combination of these directions, lies in $K_{M}$ . This is the conjugate-gradient/Lanczos characterization: PLS minimizes the residual over the Krylov subspace generated by $X^{⊤} X$ and $X^{⊤} y$ , which is why PLS coincides with $m$ steps of conjugate gradient applied to the normal equations.

Exercise 7 (hard, short-answer).

A predictor direction $v_{p}$ has the smallest singular value $d_{p}$ but the true coefficient along it, $v_{p}^{⊤} β$ , is large, so $v_{p}$ is highly predictive of $y$ . Explain how PCR, PLS, and ridge each treat this direction, and which method is at greatest risk of discarding genuine signal.

Hint

PCR ranks by variance $d_{j}^{2}$ ; PLS ranks by covariance with $y$ ; ridge down-weights by $d_{j}^{2} / (d_{j}^{2} + λ)$ . Which of these uses the response when deciding the fate of a low-variance direction?

Answer

PCR ranks directions by predictor variance alone, so the low-variance $v_{p}$ is among the last to be retained and is dropped at any modest $M$ — PCR is at greatest risk of discarding this genuine signal, the classic Jolliffe caution. Ridge does not discard it but down-weights it heavily by the small factor $d_{p}^{2} / (d_{p}^{2} + λ)$ , so its contribution is damped but never zeroed; ridge keeps a little of it. PLS, because it forms directions to maximize covariance with $y$ , detects that $v_{p}$ correlates strongly with the response and loads on it earlier than PCR would, so PLS is the method most likely to retain the predictive low-variance direction. This is the precise sense in which using the response to build the components — the PLS strategy — protects against PCR's blindness, at the cost of the extra variance from estimating the directions with $y$ .

Lean formalization Intermediate+

Mathlib does not host the derived-input regression estimators, so no module is wired in (lean_status: none). The intended statements of the PCR projection identity and the PLS Krylov characterization, once an SVD-based least-squares API exists, would read roughly as follows.

-- Intended shape; not part of the current Babel Bible Lean build.
-- Requires: the thin SVD of a real matrix and a least-squares-on-scores API.
variable {n p : ℕ} (X : Matrix (Fin n) (Fin p) ℝ) (y : Fin n → ℝ) (M : ℕ)

-- Principal-component scores z_m = X v_m and their orthogonality z_mᵀ z_k = d_m² δ.
-- PCR fitted vector is the orthogonal projection of y onto span{u_1, …, u_M}.
theorem pcr_is_projection (hM : M ≤ X.rank) :
    pcrFit X y M = orthogonalProjection (topLeftSingularSpan X M) y := by
  sorry

-- The PCR coefficient equals the SVD estimator with factors f_m = 1{m ≤ M}.
theorem pcr_shrinkage_factors (hM : M ≤ X.rank) :
    pcrBeta X y M = svdEstimator X y (fun m => if m < M then (1 : ℝ) else 0) := by
  sorry

-- PLS coefficients live in the Krylov subspace K_M(XᵀX, Xᵀy).
theorem pls_in_krylov (M : ℕ) :
    plsBeta X y M ∈ krylovSubspace (Xᵀ * X) (Xᵀ *ᵥ y) M := by
  sorry

the Mathlib gap analysis records what is missing: the variance-maximizing principal directions with deflation, the PCR estimator as projection onto the top singular subspace, the indicator shrinkage profile, and the NIPALS/Krylov characterization of PLS together with its conjugate-gradient equivalence.

Advanced results Master

The shrinkage-operator viewpoint unifies the chapter. Every method in the bias-variance family acts on the SVD coordinates as $y \mapsto \sum_{j = 1}^{r} u_{j} f_{j} (u_{j}^{⊤} y)$ for a profile of factors $f_{j}$ : ordinary least squares takes $f_{j} = 1$ ; ridge 45.06.03 takes the smooth $f_{j} = d_{j}^{2} / (d_{j}^{2} + λ)$ ; principal components regression takes the hard indicator $f_{j} = 1 {j \leq M}$ ; best-subset selection 45.06.05 takes a data-dependent $0/1$ profile that need not respect the singular-value order. Reading these off one figure (Hastie, Tibshirani & Friedman ^{[Hastie 2009]} §3.6, Figure 3.17) shows PCR as the discontinuous cousin of ridge: where ridge tapers, PCR cliffs. Subset selection and PCR share the discontinuity; ridge and PLS are continuous in their tuning parameter.

Partial least squares does not fit cleanly into a fixed factor profile because it uses $y$ to choose its directions, so its effective shrinkage factors depend on the response and can exceed $1$ on high-variance directions. Frank and Friedman's simulation study established the empirical ordering: in the high-collinearity, large- $p$ regime, ridge and PLS are typically the strongest predictors, PCR is close behind, and best-subset selection is the most variable. The reason is variance control. PCR and ridge restrict attention to the high-variance directions where coefficients are well-determined; PLS does the same but tilts toward directions correlated with $y$ , gaining bias reduction at the cost of the extra variance from estimating those directions with the response. When the signal lives in the high-variance directions — common when predictors are collinear measurements of a few latent factors — all three sharply outperform OLS; when the signal hides in a low-variance direction, PCR can fail and PLS or ridge recover.

The conjugate-gradient identity sharpens the picture for PLS. By Exercise 6 the PLS coefficient after $M$ steps minimizes the residual sum of squares over the Krylov subspace $K_{M} (X^{⊤} X, X^{⊤} y)$ , which is precisely what $M$ iterations of conjugate gradient on the normal equations $X^{⊤} X β = X^{⊤} y$ produce. PLS is therefore an early-stopped iterative solver: each component is one conjugate-gradient step, and stopping early is the regularization. This connects derived-input regression to iterative regularization for ill-posed inverse problems, where the iteration count plays the role ridge's $λ$ plays — the Landweber and conjugate-gradient regularization of inverse problems, where stopping the iteration before convergence keeps the small-singular-value directions from blowing up.

The number of components is the bias-variance tuning parameter, selected exactly as $λ$ is for ridge: by $K$ -fold cross-validation over $M \in {0, 1, \dots, r}$ , taking the minimizer of held-out error or the most parsimonious $M$ within one standard error of it. For PCR, Jolliffe ^{[Jolliffe 2002]} argues against the naive "retain by variance" rule, recommending that components be screened by their correlation with the response or their regression $t$ -statistics, since a small-variance component can carry large predictive value — the central pathology that motivates PLS over PCR.

A brief connection to canonical correlation closes the circle. Where PCA finds directions of maximal variance within one block $X$ , and PLS finds directions of maximal covariance between $X$ and $y$ , canonical correlation analysis (CCA) finds paired directions $a, b$ maximizing the correlation $Corr (X a, Y b)$ between two blocks $X$ and $Y$ . The three are a graded family: covariance equals correlation times the two standard deviations, so PLS sits between PCA (variance only) and CCA (correlation only), interpolating by how much it normalizes for the response variance. In the single-response case CCA collapses to ordinary least squares, recovering the OLS end of the spectrum once more.

Synthesis. The foundational reason PCR, PLS, ridge, and subset selection form one theory is that the SVD diagonalizes regression into independent one-dimensional problems on the principal directions, so each method is fully specified by how it weights direction $j$ : this is exactly the shrinkage map $y \mapsto \sum_{j} u_{j} f_{j} (u_{j}^{⊤} y)$ , and the entire content of a derived-input method is its factor profile ${f_{j}}$ . PCR's hard indicator generalises into ridge's smooth taper, so the two are the discontinuous and continuous limits of a single contraction, and PLS is the response-aware version whose factors are computed from a Krylov subspace rather than fixed in advance. This is exactly the conjugate-gradient/Lanczos structure: PLS is early-stopped CG on the normal equations, so the component count is dual to ridge's penalty as a regularization dial. Putting these together with the canonical-correlation hierarchy — PCA maximizes variance, PLS covariance, CCA correlation — shows that the choice of derived inputs is a choice of how much the response participates in forming the basis, from not at all (PCR) to fully (CCA), with PLS in between; the central insight is that high-dimensional collinear regression is governed not by penalizing coefficients but by selecting which principal directions to trust, and the bridge between all these methods is the common diagonalization in which trusting a direction means assigning it a shrinkage factor near one and distrusting it means a factor near zero.

Full proof set Master

The PCR projection identity, the hard-truncation shrinkage factors, the score orthogonality, the first-PLS-direction characterization, the PCR-equals-OLS-at- $M = r$ identity, and the Krylov characterization are proved in full in the Formal definition, Key theorem, and Exercises sections. The remaining Master claims are recorded here.

Proposition 1 (PCR minimizes prediction MSE among rank- $M$ truncations along the singular basis). Among all estimators of the form $\overset{y}{^} = \sum_{j \in S} u_{j} (u_{j}^{⊤} y)$ with $∣ S ∣ = M$ chosen from the singular basis, the choice $S = {1, \dots, M}$ minimizes the expected prediction error when the signal-to-noise ratio per direction, $d_{j}^{2} α_{j}^{2} / σ^{2}$ , is nonincreasing in $j$ , where $α = V^{⊤} β$ .

Proof. Write the true response as $y = X β + ε = \sum_{j} u_{j} d_{j} α_{j} + ε$ with $Cov (ε) = σ^{2} I$ . For a retained set $S$ , the fitted vector is $\overset{y}{^} = \sum_{j \in S} u_{j} (u_{j}^{⊤} y) = \sum_{j \in S} u_{j} (d_{j} α_{j} + u_{j}^{⊤} ε)$ . The expected squared error decomposes over the orthonormal directions: each dropped direction $j \in / S$ contributes squared bias $d_{j}^{2} α_{j}^{2}$ (the lost signal), and each kept direction contributes variance $σ^{2}$ . So $E ∥ \overset{y}{^} - X β ∥^{2} = \sum_{j \in / S} d_{j}^{2} α_{j}^{2} + ∣ S ∣ σ^{2}$ . With $∣ S ∣ = M$ fixed the variance term is constant, so minimizing the error means minimizing the dropped signal $\sum_{j \in / S} d_{j}^{2} α_{j}^{2}$ , achieved by dropping the $r - M$ directions with the smallest $d_{j}^{2} α_{j}^{2}$ . When $d_{j}^{2} α_{j}^{2}$ is nonincreasing in $j$ , those are exactly $j = M + 1, \dots, r$ , so $S = {1, \dots, M}$ is optimal. $□$

Proposition 2 (PLS shrinkage factors can exceed one). There exist designs and responses for which the implied PLS shrinkage factor on some high-variance direction exceeds $1$ , so PLS is not a contraction in the sense ridge and PCR are.

Proof. Write the PLS fitted vector in the singular basis as $\overset{y}{^}_{PLS}^{(M)} = \sum_{j} u_{j} θ_{j}^{(M)} (u_{j}^{⊤} y)$ , defining the implied factors $θ_{j}^{(M)}$ . Because PLS minimizes the residual over the Krylov subspace $K_{M}$ (Exercise 6), the factors are values of the degree- $M$ polynomial $q_{M}$ with $θ_{j}^{(M)} = q_{M} (d_{j}^{2})$ that the conjugate-gradient iteration selects, normalized so that $q_{M}$ interpolates $1$ at the Ritz values. Such polynomials are not constrained to $[0, 1]$ : at a $d_{j}^{2}$ lying between two Ritz values the interpolating polynomial can overshoot, giving $q_{M} (d_{j}^{2}) > 1$ . A concrete two-component example with three well-separated singular values exhibits $θ_{1}^{(1)} > 1$ on the leading direction after one step, because the single Krylov direction $X^{⊤} y$ over-weights the direction it most aligns with. Hence the PLS factor profile is not a contraction, distinguishing it from ridge ( $f_{j} \in (0, 1)$ ) and PCR ( $f_{j} \in {0, 1}$ ). $□$

Proposition 3 (PLS equals conjugate gradient on the normal equations). The $M$ -component PLS coefficient $\hat{β}_{PLS}^{(M)}$ equals the iterate $β_{M}$ produced by $M$ steps of the conjugate-gradient method applied to $X^{⊤} X β = X^{⊤} y$ started from $β_{0} = 0$ .

Proof. Conjugate gradient started at $0$ produces, after $M$ steps, the unique minimizer of the quadratic $\frac{1}{2} β^{⊤} X^{⊤} X β - β^{⊤} X^{⊤} y$ — equivalently of $∥ y - X β ∥^{2}$ up to a constant — over the Krylov subspace $K_{M} (X^{⊤} X, X^{⊤} y) = span {X^{⊤} y, \dots, (X^{⊤} X)^{M - 1} X^{⊤} y}$ . By Exercise 6 the PLS directions span the same $K_{M}$ , and PLS by construction regresses $y$ on those directions, i.e. minimizes the residual sum of squares over $K_{M}$ . Two minimizers of the same strictly convex quadratic over the same subspace coincide, so $\hat{β}_{PLS}^{(M)} = β_{M}$ . Early stopping of conjugate gradient is therefore identical to choosing the PLS component count, and the iteration index is the regularization parameter. $□$

Proposition 4 (CCA reduces to OLS for a single response). When the second block is a single centered response $y \in R^{n}$ , the canonical-correlation direction $a$ maximizing $Corr (X a, y)$ yields the same fitted direction as ordinary least squares of $y$ on $X$ .

Proof. For scalar $y$ the canonical problem is $max_{a} Corr (X a, y) = max_{a} \frac{a ^{⊤} X ^{⊤} y}{a ^{⊤} X ^{⊤} X a ∥ y ∥}$ , since $b$ is one-dimensional and drops out up to sign. Maximizing $\frac{( a ^{⊤} c ) ^{2}}{a ^{⊤} A a}$ with $c = X^{⊤} y$ , $A = X^{⊤} X$ is a generalized Rayleigh quotient solved by $a \propto A^{- 1} c = (X^{⊤} X)^{- 1} X^{⊤} y = \hat{β}_{OLS}$ (on the range of $X^{⊤} X$ ). The maximizing direction is the OLS coefficient vector, so single-response CCA recovers least squares, placing OLS at the correlation-maximizing end of the PCA-PLS-CCA family. $□$

Connections Master

Linear model theory 45.06.01 supplies the OLS estimator and the column-space projection that PCR refines: PCR projects $y$ onto the span of the top $M$ left singular vectors rather than the full column space, so it is the Gauss-Markov fit restricted to a low-rank principal subspace, trading the unbiasedness of 45.06.01 for the variance reduction of discarding small-singular-value directions.

Ridge regression and shrinkage 45.06.03 is the continuous sibling: both ridge and PCR act on the SVD coordinates as $y \mapsto \sum_{j} u_{j} f_{j} (u_{j}^{⊤} y)$ , with ridge using the smooth factor $d_{j}^{2} / (d_{j}^{2} + λ)$ and PCR the hard indicator $1 {j \leq M}$ , so PCR is the discontinuous limit of ridge and the two share the variance-control mechanism of damping the unstable directions.

The singular value decomposition 01.01.12 is the structural backbone: the right singular vectors are the principal component directions, the left singular vectors span the subspaces PCR projects onto, and the singular values order the directions by variance — every claim about PCR, and the Krylov structure of PLS, is read off the SVD of the centered design.

Subset selection and the curse of dimensionality 45.06.05 is the combinatorial counterpart: best-subset selection is a $0/1$ shrinkage profile like PCR's, but chosen over coordinate subspaces rather than principal subspaces, so PCR is the principal-direction analogue of subset selection and both sit at the discontinuous end of the shrinkage-operator family that ridge anchors at the smooth end.

The James-Stein estimator 45.06.04 is the decision-theoretic precedent for trading bias for variance in many dimensions: PCR and PLS are the regression-specific shrinkage estimators whose admissibility-style gains over OLS in high collinearity echo the Stein phenomenon, with the principal-direction truncation playing the role of the Stein contraction toward a low-dimensional target.

Historical & philosophical context Master

Principal component analysis originated with Karl Pearson in 1901, who sought the line and plane of closest fit to a system of points, and was independently developed and named by Harold Hotelling in 1933 as the analysis of the principal components of a correlated battery of variables ^{[Jolliffe 2002]}. Their use as regressors to combat multicollinearity — principal components regression — was advanced through the 1960s by Kendall, Massy, and others; Jolliffe's monograph ^{[Jolliffe 2002]} consolidated the practice and issued the standing caution that selecting components by variance alone can discard a low-variance direction that is strongly predictive of the response, the pathology that motivates response-aware alternatives.

Partial least squares arose in a different community. Herman Wold developed the NIPALS algorithm in the 1960s for latent-variable path models in econometrics and sociology, and his son Svante Wold adapted it in the late 1970s into PLS regression, which became the workhorse of chemometrics — the analysis of spectroscopic data with thousands of collinear wavelength channels and few samples. The modern statement of the algorithm and its standing in chemometrics is given by Wold, Sjöström and Eriksson ^{[Wold 2001]}. The equivalence of PLS to the conjugate-gradient and Lanczos methods on the normal equations, recognized in the numerical-analysis literature, reframed PLS as an iterative regularization scheme rather than a purely statistical heuristic. The unified shrinkage-operator account that places PCR, PLS, ridge, and subset selection on one footing, and the empirical comparison favoring ridge and PLS under collinearity, were assembled by Frank and Friedman and presented in Hastie, Tibshirani and Friedman ^{[Hastie 2009]} chapter 3.

Bibliography Master

@article{pearson1901lines,
  author  = {Pearson, Karl},
  title   = {On Lines and Planes of Closest Fit to Systems of Points in Space},
  journal = {Philosophical Magazine},
  volume  = {2},
  number  = {11},
  pages   = {559--572},
  year    = {1901}
}

@article{hotelling1933analysis,
  author  = {Hotelling, Harold},
  title   = {Analysis of a Complex of Statistical Variables into Principal Components},
  journal = {Journal of Educational Psychology},
  volume  = {24},
  pages   = {417--441, 498--520},
  year    = {1933}
}

@book{jolliffe2002pca,
  author    = {Jolliffe, Ian T.},
  title     = {Principal Component Analysis},
  edition   = {2},
  publisher = {Springer},
  year      = {2002},
  note      = {Chapter 8, principal components in regression}
}

@article{wold2001pls,
  author  = {Wold, Svante and Sj{\"o}str{\"o}m, Michael and Eriksson, Lennart},
  title   = {{PLS}-regression: A Basic Tool of Chemometrics},
  journal = {Chemometrics and Intelligent Laboratory Systems},
  volume  = {58},
  number  = {2},
  pages   = {109--130},
  year    = {2001}
}

@incollection{wold1975soft,
  author    = {Wold, Herman},
  title     = {Soft Modelling by Latent Variables: The Non-Linear Iterative Partial Least Squares (NIPALS) Approach},
  booktitle = {Perspectives in Probability and Statistics},
  publisher = {Academic Press},
  pages     = {117--142},
  year      = {1975}
}

@article{frank1993statistical,
  author  = {Frank, Ildiko E. and Friedman, Jerome H.},
  title   = {A Statistical View of Some Chemometrics Regression Tools},
  journal = {Technometrics},
  volume  = {35},
  number  = {2},
  pages   = {109--135},
  year    = {1993}
}

@book{hastie2009elements,
  author    = {Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome},
  title     = {The Elements of Statistical Learning: Data Mining, Inference, and Prediction},
  edition   = {2},
  publisher = {Springer},
  year      = {2009},
  note      = {Chapter 3, §3.5 derived-input directions; §3.6 comparison}
}

Prerequisites

45.06.01
45.06.03
01.01.12

Tier anchors

beginner: Hastie, Tibshirani & Friedman 2009 The Elements of Statistical Learning 2e (Springer) §3.5 (methods using derived input directions — principal components regression and partial least squares, the shrinkage picture); James, Witten, Hastie & Tibshirani 2013 An Introduction to Statistical Learning (Springer) §6.3 (dimension reduction methods, PCR and PLS informally)
intermediate: Hastie, Tibshirani & Friedman 2009 The Elements of Statistical Learning 2e (Springer) §3.5.1-3.5.2 (PCR as regression on the leading principal components, the SVD truncation/shrinkage interpretation contrasted with ridge; PLS and the covariance-maximizing direction construction); Jolliffe 2002 Principal Component Analysis 2e (Springer) Ch. 8 (principal components in regression)
master: Hastie, Tibshirani & Friedman 2009 The Elements of Statistical Learning 2e (Springer) §3.5 and §3.6 (derived-input directions, the discussion comparing subset selection, ridge, lasso, PCR, and PLS as shrinkage operators); Jolliffe 2002 Principal Component Analysis 2e (Springer) Ch. 8 (PCR, the selection of components, biased regression); Wold, Sjöström & Eriksson 2001 Chemometrics and Intelligent Laboratory Systems 58 (PLS-regression: a basic tool of chemometrics, the NIPALS algorithm)

References

Hastie, T., Tibshirani, R. & Friedman, J. — The Elements of Statistical Learning (2nd ed.) · Springer, 2009. Ch. 3 (linear methods for regression): §3.5 (methods using derived input directions): §3.5.1 (principal components regression — regress y on the top M principal components z_m = X v_m where v_m are the right singular vectors / eigenvectors of X^T X, the orthogonality of the z_m, the fitted value as a sum over retained components, and the shrinkage interpretation that PCR keeps the M largest-variance directions intact and discards the rest — a hard 0/1 truncation contrasted with ridge's smooth factors d_j^2/(d_j^2 + lambda)), §3.5.2 (partial least squares — the NIPALS construction in which each direction is chosen to maximize covariance with y, the deflation step, and the comparison showing PLS shrinks low-variance directions less than ridge and can inflate high-variance ones), §3.6 (the unified comparison of subset selection, ridge, lasso, PCR, and PLS, Table 3.4 and the shrinkage-factor figure).
Jolliffe, I. T. — Principal Component Analysis (2nd ed.) · Springer, 2002. Ch. 8 (principal components in regression): §8.1 (the use of principal components as regressors to combat multicollinearity), §8.2 (strategies for choosing which components to retain — by size of variance, by correlation with the response, by t-statistics), and the cautionary discussion that selecting components by variance alone can discard a low-variance direction that is strongly predictive of y. Ch. 1-3 (the definition of principal components as the orthonormal directions of successively maximal variance, equal to the eigenvectors of the covariance matrix / right singular vectors of the centered data matrix).
Wold, S., Sjöström, M. & Eriksson, L. — PLS-regression: a basic tool of chemometrics · Chemometrics and Intelligent Laboratory Systems 58 (2), 109-130 (2001). The NIPALS / iterative construction of PLS components as the directions of maximal sample covariance with the response, the deflation of X and y after each extracted component, the interpretation of PLS as a method that uses the response to form the derived inputs (unlike PCR), and the practical regime — many collinear predictors, p large relative to n — where PLS regression is the chemometric workhorse. Originating PLS references: H. Wold (1975), the fixed-point / partial least squares algorithm for latent-variable path models.

Estimated time

beginner: 25m
intermediate: 55m
master: 85m