Ridge Regression and Shrinkage Estimation

Intuition Beginner

Least squares hands you the weights that fit your data best, full stop. That sounds ideal, but it has a failure mode: when your predictors are nearly redundant — two columns of data that say almost the same thing — the fitting machine cannot decide how to split the credit between them. It compensates by inventing huge offsetting weights, a giant positive one on the first predictor and a giant negative one on the second. These weights are wildly unstable: collect a fresh dataset and they swing somewhere completely different. The fit looks great on the data you have and predicts terribly on data you do not.

Ridge regression fixes this by adding a penalty for large weights. Instead of asking only "how small can I make the total squared miss?", it asks "how small can I make the total squared miss plus a charge on the size of the weights?" The charge is tuned by a dial. Turn the dial to zero and you are back to plain least squares. Turn it up and the weights are pulled toward zero, gently at first, harder as you keep turning.

Why does pulling the weights toward zero help? Because a small, deliberate amount of being-wrong-on-average buys a large reduction in instability. You accept a fit that is slightly off the true relationship in exchange for a fit that barely moves when the data changes. This is the bias-variance bargain from the previous unit, made into a knob you can turn.

The everyday image is a budget. Least squares spends without limit to match the data; ridge puts the weights on an allowance. Under the allowance, the fit cannot chase every quirk of one dataset, so it tracks the stable, repeatable signal instead.

Visual Beginner

Dial setting (penalty strength)	What happens to the weights	Bias	Variance	Typical outcome
Zero	exactly the least-squares weights	none	high (can be huge if predictors overlap)	overfits, unstable
Small	weights pulled slightly toward zero	small	lower	usually better on new data
Moderate	clear shrinkage, stable weights	moderate	low	often the sweet spot
Very large	weights crushed toward zero	large	tiny	underfits, predicts the average

The single picture to carry forward: every coefficient path slides toward zero as you raise the penalty, but the directions in your data that carry the most information resist the pull, while the flimsy, redundant directions collapse fast. Ridge does not switch predictors off; it quiets the unreliable ones.

Worked example Beginner

Take two data points and one predictor with no intercept, so the model is $\hat{y}=bx$. The data are $x=2$ at $y=4$, and $x=1$ at $y=1$. Plain least squares picks the slope $b$ that minimizes $(4-2b{)}^2+(1-b{)}^2$.

The least-squares slope solves the normal equation. The sum of $x^2$ is $4+1=5$, and the sum of $xy$ is $2\times 4+1\times 1=9$, so $b_{\text{OLS}}=9/5=1.8$.

Now add a ridge penalty with dial value $\lambda =5$. Ridge minimizes $(4-2b{)}^2+(1-b{)}^2+5b^2$. The recipe changes the denominator: it becomes the sum of $x^2$ plus $\lambda$, that is $5+5=10$, while the numerator stays $9$. So the ridge slope is $b_{\text{ridge}}=9/10=0.9$.

The penalty has cut the slope exactly in half, from $1.8$ to $0.9$. Try $\lambda =1$ instead: the denominator is $5+1=6$, giving $b=9/6=1.5$. And $\lambda =20$ gives denominator $25$, so $b=9/25=0.36$.

What this tells us: the ridge slope is always the least-squares slope scaled down by the factor $5/(5+\lambda )$ — the original $x^2$-sum over itself-plus-penalty. At $\lambda =0$ the factor is $1$ (no shrinkage); as $\lambda$ grows the factor shrinks toward $0$. The penalty never flips the sign and never sets the slope to exactly zero; it only damps it, and the more penalty you add, the more it damps.

Check your understanding Beginner

Formal definition Intermediate+

Fix a design matrix $X\in {\mathbb{R}}^{n\times p}$ and response $y\in {\mathbb{R}}^n$. Assume the predictors are centered and scaled to unit variance and the intercept is fitted separately (so $X$ carries no column of ones); the penalty is applied to the slope coefficients only. For a penalty parameter $\lambda \ge 0$, the ridge regression (or Tikhonov-regularized) estimator is the minimizer of the penalized residual sum of squares

$$\hat{\beta }(\lambda )=\arg \underset{\beta \in {\mathbb{R}}^p}{\min }\{\Vert y-X\beta {\Vert }^2+\lambda \Vert \beta {\Vert }^2\}.$$

The criterion is a strictly convex quadratic in $\beta$ whenever $\lambda >0$. Differentiating and setting the gradient to zero gives the regularized normal equations $(X^{\top }X+\lambda I)\beta =X^{\top }y$, and since $X^{\top }X$ is positive semidefinite, $X^{\top }X+\lambda I$ is positive definite for $\lambda >0$, hence invertible. The estimator is

$$\hat{\beta }(\lambda )=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }y.$$

The decisive structural fact, sharpening the OLS theory 45.06.01: this solution exists and is unique even when $X^{\top }X$ is singular — in particular when $p>n$ and ordinary least squares has no unique solution. Adding $\lambda I$ lifts every eigenvalue of $X^{\top }X$ by $\lambda$, curing the rank deficiency. The fitted values are $\hat{y}=X\hat{\beta }(\lambda )=H_{\lambda }y$ with the ridge hat matrix

$$H_{\lambda }=X(X^{\top }X+\lambda I{)}^{-1}{X}^{\top },$$

which is symmetric but not idempotent (it is not a projection), and the effective degrees of freedom are $\mathrm{df}(\lambda )=\mathrm{tr}(H_{\lambda })$, decreasing from $\mathrm{rank}(X)$ at $\lambda =0$ to $0$ as $\lambda \to \infty$ — the continuous complexity index of 45.06.02.

The singular value view makes the mechanism transparent. Write the thin SVD $X=UD{V}^{\top }$ 01.01.12 with $U\in {\mathbb{R}}^{n\times r}$, $V\in {\mathbb{R}}^{p\times r}$ having orthonormal columns $u_j,v_j$, and $D=\mathrm{diag}(d_1,\dots ,d_r)$ the singular values, $r=\mathrm{rank}(X)$. The ridge fitted values decompose along the principal directions as

$$\hat{y}=H_{\lambda }y=\sum _{j=1}^r{u}_j\frac{d_j^2}{d_j^2+\lambda }(u_j^{\top }y).$$

Each coordinate $u_j^{\top }y$ is multiplied by the shrinkage factor $d_j^2/(d_j^2+\lambda )\in (0,1]$. Directions with large singular values (where the data carry strong, well-determined signal) are barely touched; directions with small singular values (where the design is nearly collinear and the OLS coefficient is unstable) are shrunk hard. Ordinary least squares is the $\lambda =0$ case, where every factor equals $1$.

Counterexamples to common slips

• The ridge hat matrix is not a projection. $H_{\lambda }$ is symmetric but $H_{\lambda }^2\ne H_{\lambda }$ for $\lambda >0$; its eigenvalues $d_j^2/(d_j^2+\lambda )$ lie strictly between $0$ and $1$, so $\hat{y}$ is a contraction of $y$ toward the origin along principal directions, not an orthogonal projection onto a subspace.
• Ridge is not scale-invariant. Because the penalty $\lambda \Vert \beta {\Vert }^2$ depends on the units of each predictor, rescaling a column changes the solution. Standardizing the predictors before fitting is part of the method, not an optional cleanup; the intercept is excluded from the penalty for the same reason.
• Existence does not require full rank. The phrase "the ridge solution" is well-posed for any $\lambda >0$ regardless of $\mathrm{rank}(X)$; the regularizer guarantees a unique minimizer even when $n<p$, which is precisely the high-dimensional regime where OLS fails to define one.
• Shrinkage toward zero is a modeling choice. Ridge shrinks toward the origin only because the prior mean is taken to be zero after centering; shrinking toward any fixed ${\beta }_0$ replaces $\Vert \beta {\Vert }^2$ by $\Vert \beta -{\beta }_0{\Vert }^2$ and changes nothing structural.

Key theorem with proof Intermediate+

Theorem (existence of an improving ridge constant — Hoerl & Kennard 1970). Under the linear model $y=X\beta +\epsilon$ with $\mathbb{E}[\epsilon ]=0$, $\mathrm{Cov}(\epsilon )={\sigma }^2{I}_n$, and $X$ of full column rank, let $\mathrm{MSE}(\lambda )=\mathbb{E}[\Vert \hat{\beta }(\lambda )-\beta {\Vert }^2]$ be the total mean squared error of the ridge estimator. Then whenever ${\sigma }^2>0$ there exists $\lambda >0$ with $\mathrm{MSE}(\lambda )<\mathrm{MSE}(0)$: ridge strictly beats ordinary least squares in total MSE. The proof follows Hoerl & Kennard ^{[Hoerl 1970]} and the SVD presentation of Hastie, Tibshirani & Friedman ^{[Hastie 2009]} §3.4.1.

Proof. Rotate to the eigenbasis of $X^{\top }X$. With the SVD $X=UD{V}^{\top }$, write $\alpha =V^{\top }\beta$ for the true coefficient in singular coordinates and ${\hat{\alpha }}_j(\lambda )=V^{\top }\hat{\beta }(\lambda )$ for the estimator. A direct computation from $\hat{\beta }(\lambda )=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }y$ gives, in coordinate $j$,

$${\hat{\alpha }}_j(\lambda )=\frac{d_j}{d_j^2+\lambda }{u}_j^{\top }y,\mathbb{E}[{\hat{\alpha }}_j(\lambda )]=\frac{d_j^2}{d_j^2+\lambda }{\alpha }_j,\mathrm{Var}({\hat{\alpha }}_j(\lambda ))=\frac{d_j^2{\sigma }^2}{(d_j^2+\lambda {)}^2},$$

using $u_j^{\top }y=d_j{\alpha }_j+u_j^{\top }\epsilon$ with $\mathrm{Var}(u_j^{\top }\epsilon )={\sigma }^2$. Because $V$ is orthogonal, total MSE is the sum of coordinate MSEs, and each coordinate splits as squared bias plus variance 45.06.02:

$$\mathrm{MSE}(\lambda )=\sum _{j=1}^p\left[\underset{\text{squared bias}}{\underbrace{(\frac{\lambda }{d_j^2+\lambda }{)}^2{\alpha }_j^2}}+\underset{\text{variance}}{\underbrace{\frac{d_j^2{\sigma }^2}{(d_j^2+\lambda {)}^2}}}\right].$$

Write $g(\lambda )$ for this sum and differentiate term by term. For the variance part, $\frac{d}{d\lambda }\frac{d_j^2{\sigma }^2}{(d_j^2+\lambda {)}^2}=\frac{-2d_j^2{\sigma }^2}{(d_j^2+\lambda {)}^3}$. For the bias part, $\frac{d}{d\lambda }\frac{{\lambda }^2{\alpha }_j^2}{(d_j^2+\lambda {)}^2}=\frac{2\lambda d_j^2{\alpha }_j^2}{(d_j^2+\lambda {)}^3}$. Summing,

$$g^{\prime }(\lambda )=\sum _{j=1}^p\frac{2d_j^2(\lambda {\alpha }_j^2-{\sigma }^2)}{(d_j^2+\lambda {)}^3}.$$

Evaluate at $\lambda =0$: every numerator becomes $2d_j^2(-{\sigma }^2)<0$, so $g^{\prime }(0)=-{\sum }_{j}2{\sigma }^2/d_j^2<0$. The squared-bias terms enter through ${\lambda }^2$ and so contribute nothing to the first derivative at the origin, while the variance terms contribute a strictly negative slope. Since $g$ is continuously differentiable and $g^{\prime }(0)<0$, there is an interval $(0,{\lambda }_0)$ on which $g(\lambda )<g(0)=\mathrm{MSE}(0)$. Any $\lambda$ in that interval improves on least squares. $\square$

Bridge. The proof's engine is the SVD rotation that diagonalizes the problem so that total MSE becomes a sum of one-dimensional bias-variance trade-offs, each governed by the shrinkage factor $d_j^2/(d_j^2+\lambda )$. This builds toward the full advanced theory below, where the same factor controls effective degrees of freedom and the Bayesian posterior mean, and it appears again in 45.06.06 when the lasso replaces the smooth ${\ell }_2$ contraction with a thresholding ${\ell }_1$ map. The foundational reason ridge can win is that least squares sits at the boundary of the admissible region — it is the unbiased corner — so its MSE derivative in the bias direction is zero while the variance can only be reduced by moving inward; this is exactly the single-coordinate computation of 45.06.02 Exercise 5 promoted to all $p$ directions at once. The result generalises the Gauss-Markov benchmark of 45.06.01 rather than contradicting it: Gauss-Markov ranks OLS first among unbiased linear estimators, and ridge wins only by abandoning unbiasedness, so the central insight is that the unbiasedness constraint — not optimality itself — is what high-dimensional statistics relaxes. Putting these together, ridge is the smooth, always-feasible shrinkage that the bias-variance decomposition predicts must exist.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not yet host the statistical ridge estimator, so no module is wired in (lean_status: none). The intended statement of the closed form and the SVD shrinkage identity, once a least-squares-functional API exists, would read roughly as follows.

-- Intended shape; not part of the current Babel Bible Lean build.
-- Requires: a penalized-least-squares functional and the SVD of a real matrix.
variable {n p : ℕ} (X : Matrix (Fin n) (Fin p) ℝ) (y : Fin n → ℝ) (lam : ℝ)

-- The regularized Gram matrix is positive definite, hence invertible, for lam > 0.
theorem ridge_gram_posDef (hlam : 0 < lam) :
    (Xᵀ * X + lam • (1 : Matrix (Fin p) (Fin p) ℝ)).PosDef := by
  sorry

-- Closed form: the ridge estimator minimizes ‖y - X β‖² + lam ‖β‖².
noncomputable def ridgeHat (hlam : 0 < lam) : Fin p → ℝ :=
  (Xᵀ * X + lam • 1)⁻¹ *ᵥ (Xᵀ *ᵥ y)

theorem ridge_minimizes (hlam : 0 < lam) (β : Fin p → ℝ) :
    ‖y - X *ᵥ ridgeHat X y lam hlam‖^2 + lam * ‖ridgeHat X y lam hlam‖^2
      ≤ ‖y - X *ᵥ β‖^2 + lam * ‖β‖^2 := by
  sorry

the Mathlib gap analysis records what is missing: the penalized-least-squares functional, the proof that $X^{\top }X+\lambda I$ is positive definite from Matrix.PosSemidef plus the strictly positive shift, the SVD-coordinate shrinkage identity, and the conjugate-normal posterior-mean computation behind the Bayesian view.

Advanced results Master

The Bayesian derivation places ridge inside a coherent inferential frame. Take the conjugate Gaussian model $y\mid \beta \sim N(X\beta ,{\sigma }^2{I}_n)$ with prior $\beta \sim N(0,{\tau }^2{I}_p)$. The log-posterior in $\beta$ is, up to additive constants,

$$-\frac{1}{2{\sigma }^2}\Vert y-X\beta {\Vert }^2-\frac{1}{2{\tau }^2}\Vert \beta {\Vert }^2,$$

and maximizing it over $\beta$ is exactly the ridge criterion with $\lambda ={\sigma }^2/{\tau }^2$. The posterior is itself Gaussian, $\beta \mid y\sim N(\hat{\beta }(\lambda ),{\sigma }^2(X^{\top }X+\lambda I{)}^{-1})$, so the ridge estimator is simultaneously the maximum-a-posteriori and the posterior-mean estimate; for a Gaussian posterior the two coincide. The penalty strength is the noise-to-prior variance ratio: a tight prior (small ${\tau }^2$, strong belief that coefficients are near zero) forces a large $\lambda$ and heavy shrinkage, while a diffuse prior recovers least squares as ${\tau }^2\to \infty$. This is the forward connection promised by 45.06.02: shrinkage is prior information made quantitative, developed fully in the Bayesian treatment 45.03.02.

Effective degrees of freedom give ridge a calibrated complexity. With $\mathrm{df}(\lambda )={\sum }_j{d}_j^2/(d_j^2+\lambda )$, the optimism identity of 45.06.02 yields the in-sample risk surrogate $\mathbb{E}[\text{in-sample error}]=\mathbb{E}[\overline{\mathrm{err}}]+\frac{2{\sigma }^2}{n}\mathrm{df}(\lambda )$, and substituting an estimate of $\mathrm{df}$ into Mallows-type criteria or the generalized cross-validation score $\mathrm{GCV}(\lambda )=\frac{n^{-1}\Vert y-H_{\lambda }y{\Vert }^2}{(1-\mathrm{df}(\lambda )/n{)}^2}$ selects $\lambda$ without a held-out set. In practice $\lambda$ is chosen by $K$-fold cross-validation over a logarithmic grid: for each candidate the average held-out squared error is computed, and the minimizer (or the largest $\lambda$ within one standard error of the minimum, the one-standard-error rule) is taken. Because ridge has a closed form, the entire ridge trace $\lambda \mapsto \hat{\beta }(\lambda )$ is a smooth path computed once from the SVD, so the cross-validation sweep is cheap.

Kernel ridge regression lifts the method to nonlinear and infinite-dimensional feature maps. Replace each $x_i$ by a feature vector $\varphi (x_i)$ in a (possibly infinite-dimensional) space and penalize $\Vert \beta {\Vert }^2$ there. The closed form, after the matrix push-through identity $({\Phi }^{\top }\Phi +\lambda I{)}^{-1}{\Phi }^{\top }={\Phi }^{\top }(\Phi {\Phi }^{\top }+\lambda I{)}^{-1}$, becomes $\hat{f}(x)=k(x{)}^{\top }(K+\lambda I{)}^{-1}y$, where $K_{ij}=⟨\varphi (x_i),\varphi (x_j)⟩=\kappa (x_i,x_j)$ is the kernel Gram matrix and $k(x{)}_i=\kappa (x,x_i)$. The feature map never appears explicitly — only inner products — which is the kernel trick. The estimator lives in the reproducing-kernel Hilbert space generated by $\kappa$, the setting developed in 45.08.01, and the representer theorem guarantees the solution is a finite combination of kernels at the data points.

Ridge sits at one end of a spectrum of penalties. The general bridge penalty $\lambda {\sum }_j|{\beta }_j{|}^q$ recovers ridge at $q=2$ and the lasso at $q=1$. The qualitative break occurs at $q=1$: for $q>1$ the penalty is differentiable at the origin and the solution map is a smooth contraction (no exact zeros, as in ridge), while at $q\le 1$ the penalty has a corner at the origin and the solution map thresholds, producing exact zeros and hence variable selection. The elastic net ${\lambda }_1\Vert \beta {\Vert }_1+{\lambda }_2\Vert \beta {\Vert }_2^2$ combines both, inheriting ridge's stability under correlated predictors (it shrinks correlated coefficients together rather than arbitrarily picking one) and the lasso's sparsity. The ${\ell }_2$-versus-${\ell }_1$ contrast that organizes this chapter is the contrast between a contraction and a threshold, exactly the SVD shrinkage of ridge against the soft-thresholding of 45.06.06.

Synthesis. The foundational reason ridge unifies regularization, Bayesian shrinkage, and complexity control is that the SVD diagonalizes the estimator into independent one-dimensional shrinkages, each with factor $d_j^2/(d_j^2+\lambda )$, so the bias, the variance, the effective degrees of freedom, and the posterior covariance are all functions of that single factor. This is exactly the multivariate promotion of the one-coordinate bias-variance calculation of 45.06.02, and putting these together with the Hoerl-Kennard existence theorem shows that the optimal penalty $\lambda ={\sigma }^2/{\tau }^2$ from the Gaussian prior and the MSE-minimizing penalty from the frequentist trade-off are the same quantity viewed from two sides — the noise-to-signal ratio. The central insight is that ridge generalises ordinary least squares by relaxing the Gauss-Markov unbiasedness constraint of 45.06.01: the contraction toward zero is dual to the prior pull toward zero, and the bridge is the regularized Gram matrix $X^{\top }X+\lambda I$ whose eigenvalue lift simultaneously cures collinearity, guarantees existence when $p>n$, and quantifies the shrinkage. The same machinery, with inner products replaced by kernels, becomes kernel ridge regression and the RKHS theory of 45.08.01; with the smooth penalty replaced by a cornered one, it becomes the sparse selection of 45.06.06.

Full proof set Master

The closed form, the SVD shrinkage identity, the effective-degrees-of-freedom formula, and the Hoerl-Kennard existence theorem are proved in full in the Formal definition, Key theorem, and Exercises sections. The remaining Master claims are recorded here.

Proposition 1 (Ridge as posterior mean under a Gaussian prior). Under $y\mid \beta \sim N(X\beta ,{\sigma }^2{I}_n)$ and $\beta \sim N(0,{\tau }^2{I}_p)$, the posterior is $\beta \mid y\sim N(\hat{\beta }(\lambda ),{\sigma }^2(X^{\top }X+\lambda I{)}^{-1})$ with $\lambda ={\sigma }^2/{\tau }^2$, so $\hat{\beta }(\lambda )$ is both the posterior mean and the MAP estimate.

Proof. The log-posterior is $-\frac{1}{2{\sigma }^2}\Vert y-X\beta {\Vert }^2-\frac{1}{2{\tau }^2}\Vert \beta {\Vert }^2+\text{const}$. Expanding the quadratic in $\beta$, the coefficient of the quadratic term is $\frac{1}{2}{\beta }^{\top }(\frac{X^{\top }X}{{\sigma }^2}+\frac{I}{{\tau }^2})\beta$ and of the linear term is $-\frac{1}{{\sigma }^2}{\beta }^{\top }{X}^{\top }y$. A Gaussian density $N(m,\Sigma )$ has log-density quadratic form $-\frac{1}{2}{\beta }^{\top }{\Sigma }^{-1}\beta +{\beta }^{\top }{\Sigma }^{-1}m$; matching, ${\Sigma }^{-1}=\frac{1}{{\sigma }^2}(X^{\top }X+\frac{{\sigma }^2}{{\tau }^2}I)=\frac{1}{{\sigma }^2}(X^{\top }X+\lambda I)$ and ${\Sigma }^{-1}m=\frac{1}{{\sigma }^2}{X}^{\top }y$. Hence $\Sigma ={\sigma }^2(X^{\top }X+\lambda I{)}^{-1}$ and $m=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }y=\hat{\beta }(\lambda )$. The posterior is Gaussian, so its mean equals its mode, and both equal $\hat{\beta }(\lambda )$. $\square$

Proposition 2 (Ridge bias and the strict MSE gain at orthonormal design). For $X^{\top }X=I_p$ with ${\hat{\beta }}_j\sim N({\beta }_j,{\sigma }^2)$ independent across $j$, the ridge MSE is $\mathrm{MSE}(\lambda )=\frac{{\lambda }^2\Vert \beta {\Vert }^2+p{\sigma }^2}{(1+\lambda {)}^2}$, minimized at ${\lambda }^{\star }=p{\sigma }^2/\Vert \beta {\Vert }^2>0$.

Proof. By Exercise 4 each coordinate satisfies ${\hat{\beta }}_j(\lambda )={\hat{\beta }}_j/(1+\lambda )$, so its bias is $\frac{{\beta }_j}{1+\lambda }-{\beta }_j=\frac{-\lambda {\beta }_j}{1+\lambda }$ with squared bias $\frac{{\lambda }^2{\beta }_j^2}{(1+\lambda {)}^2}$, and its variance is $\frac{{\sigma }^2}{(1+\lambda {)}^2}$. Summing over $j$ gives $\mathrm{MSE}(\lambda )=\frac{{\lambda }^2\Vert \beta {\Vert }^2+p{\sigma }^2}{(1+\lambda {)}^2}$. Differentiating, the numerator of $\frac{d}{d\lambda }\mathrm{MSE}$ is proportional to $2\lambda \Vert \beta {\Vert }^2(1+\lambda )-2({\lambda }^2\Vert \beta {\Vert }^2+p{\sigma }^2)=2(\lambda \Vert \beta {\Vert }^2-p{\sigma }^2)$, which vanishes at ${\lambda }^{\star }=p{\sigma }^2/\Vert \beta {\Vert }^2$. Since this is positive whenever ${\sigma }^2>0$, a strictly positive penalty minimizes MSE, recovering the general Hoerl-Kennard conclusion in the orthonormal case with an explicit optimum. $\square$

Proposition 3 (Kernel ridge push-through identity). For any feature matrix $\Phi \in {\mathbb{R}}^{n\times m}$ and $\lambda >0$, $({\Phi }^{\top }\Phi +\lambda I_m{)}^{-1}{\Phi }^{\top }={\Phi }^{\top }(\Phi {\Phi }^{\top }+\lambda I_n{)}^{-1}$, so the ridge fit is $\hat{f}=\Phi \hat{\beta }(\lambda )=K(K+\lambda I_n{)}^{-1}y$ with $K=\Phi {\Phi }^{\top }$.

Proof. Both regularized matrices are invertible for $\lambda >0$. Start from the identity ${\Phi }^{\top }(\Phi {\Phi }^{\top }+\lambda I_n)=({\Phi }^{\top }\Phi +\lambda I_m){\Phi }^{\top }$, which holds because each side equals ${\Phi }^{\top }\Phi {\Phi }^{\top }+\lambda {\Phi }^{\top }$. Left-multiply by $({\Phi }^{\top }\Phi +\lambda I_m{)}^{-1}$ and right-multiply by $(\Phi {\Phi }^{\top }+\lambda I_n{)}^{-1}$ to obtain $({\Phi }^{\top }\Phi +\lambda I_m{)}^{-1}{\Phi }^{\top }={\Phi }^{\top }(\Phi {\Phi }^{\top }+\lambda I_n{)}^{-1}$. Then $\hat{\beta }(\lambda )=({\Phi }^{\top }\Phi +\lambda I{)}^{-1}{\Phi }^{\top }y={\Phi }^{\top }(K+\lambda I{)}^{-1}y$, and $\hat{f}=\Phi \hat{\beta }(\lambda )=\Phi {\Phi }^{\top }(K+\lambda I{)}^{-1}y=K(K+\lambda I{)}^{-1}y$. The estimator depends on $\Phi$ only through inner products $K_{ij}=⟨\varphi (x_i),\varphi (x_j)⟩$, so an explicit feature map is never needed. $\square$

Proposition 4 (Ridge has no exact zeros at finite $\lambda$). If $X$ has no zero singular values and $u_j^{\top }y\ne 0$ for the relevant $j$, then for every finite $\lambda \ge 0$ each principal-coordinate ridge estimate ${\hat{\alpha }}_j(\lambda )=\frac{d_j}{d_j^2+\lambda }{u}_j^{\top }y$ is nonzero; coefficients vanish only in the limit $\lambda \to \infty$.

Proof. The shrinkage multiplier $\frac{d_j}{d_j^2+\lambda }$ is a quotient of a positive number $d_j>0$ by the positive number $d_j^2+\lambda$, hence strictly positive for every finite $\lambda$. Multiplying the nonzero data coordinate $u_j^{\top }y$ by a strictly positive scalar gives a nonzero result, so ${\hat{\alpha }}_j(\lambda )\ne 0$. As $\lambda \to \infty$ the multiplier tends to $0$, so the estimate tends to $0$ but attains it only in the limit. The smooth ${\ell }_2$ penalty therefore contracts without selecting; exact zeros require the cornered ${\ell }_1$ penalty of 45.06.06. $\square$

Connections Master

Linear model theory 45.06.01 supplies the OLS estimator, the normal equations, and the hat matrix that ridge regularizes; ridge deliberately relaxes the Gauss-Markov unbiasedness constraint $CX=I$ that makes OLS the BLUE there, replacing $X^{\top }X$ by $X^{\top }X+\lambda I$ to trade unbiasedness for a variance reduction, so ridge is the unbiased benchmark of that unit moved inward off the boundary.

The bias-variance decomposition 45.06.02 is the engine of the existence theorem: the SVD turns total MSE into a sum of per-coordinate squared-bias-plus-variance terms, and the single-coordinate shrinkage computation there (Exercise 5) is exactly the $X^{\top }X=I$ special case proved here as Proposition 2; the effective degrees of freedom $\mathrm{df}(\lambda )=\mathrm{tr}(H_{\lambda })$ that index ridge complexity are the same optimism-linked quantity defined there.

The singular value decomposition 01.01.12 is the structural backbone: rotating into the singular basis diagonalizes the ridge estimator so that each principal direction shrinks independently by $d_j^2/(d_j^2+\lambda )$, which is why collinear (small-$d_j$) directions are damped and well-determined (large-$d_j$) directions are preserved.

Bayesian statistics 45.03.02 reframes ridge as the posterior mean and MAP estimate under a Gaussian prior $\beta \sim N(0,{\tau }^{2}I)$, with the penalty $\lambda ={\sigma }^2/{\tau }^2$ equal to the noise-to-prior-variance ratio; the deliberate bias of ridge is the prior pull toward zero, and the closed-form Gaussian posterior covariance ${\sigma }^2(X^{\top }X+\lambda I{)}^{-1}$ is the inferential output the frequentist treatment leaves implicit.

The lasso and sparse selection 45.06.06 is the ${\ell }_1$ counterpart: replacing the smooth ${\ell }_2$ ball by the cornered ${\ell }_1$ ball turns ridge's pure contraction into soft-thresholding with exact zeros, so the two methods are the contraction and the threshold ends of the bridge-penalty family, and the elastic net combines them.

Kernel methods and reproducing-kernel Hilbert spaces 45.08.01 are reached by the push-through identity proved here as Proposition 3: kernel ridge regression $\hat{f}(x)=k(x{)}^{\top }(K+\lambda I{)}^{-1}y$ is ridge in a feature space accessed only through inner products, and the representer theorem of that unit guarantees the solution is a finite kernel expansion at the data points.

Historical & philosophical context Master

Ridge regression was introduced by Arthur Hoerl and Robert Kennard in two companion 1970 Technometrics papers ^{[Hoerl 1970]}, motivated by the practical instability of least squares for nonorthogonal (collinear) industrial design problems. Their central theorem — that there always exists a positive ridge constant $k$ for which the biased estimator has smaller total mean squared error than least squares — turned a numerical regularization device into a statistical method with a precise optimality justification. The diagnostic they proposed, the ridge trace plotting each coefficient against $k$, made the stabilization visible and remains in use. The same regularizer had appeared independently in the inverse-problems literature as Tikhonov regularization, after Andrey Tikhonov's work on ill-posed problems in the 1940s and 1960s, where $\lambda \Vert \beta {\Vert }^2$ stabilizes the inversion of an ill-conditioned operator; the two traditions describe the same matrix $X^{\top }X+\lambda I$.

The Bayesian reading — ridge as the posterior mean under a Gaussian prior on the coefficients — connects the method to the shrinkage estimators that James and Stein had shown in 1961 to dominate the sample mean in three or more dimensions, establishing that biased estimators can be uniformly better and that shrinkage toward a point is not a heuristic but a consequence of decision theory. The synthesis of ridge, subset selection, the lasso, and their Bayesian interpretations into a single account of shrinkage in regression was consolidated by Hastie, Tibshirani and Friedman ^{[Hastie 2009]} in chapter 3 of The Elements of Statistical Learning, where the singular-value shrinkage factors $d_j^2/(d_j^2+\lambda )$ are given as the unifying description, and extended to the high-dimensional sparse regime by Hastie, Tibshirani and Wainwright ^{[Hastie 2015]}.

Bibliography Master

@article{hoerl1970ridge,
  author  = {Hoerl, Arthur E. and Kennard, Robert W.},
  title   = {Ridge Regression: Biased Estimation for Nonorthogonal Problems},
  journal = {Technometrics},
  volume  = {12},
  number  = {1},
  pages   = {55--67},
  year    = {1970}
}

@article{hoerl1970ridgeapp,
  author  = {Hoerl, Arthur E. and Kennard, Robert W.},
  title   = {Ridge Regression: Applications to Nonorthogonal Problems},
  journal = {Technometrics},
  volume  = {12},
  number  = {1},
  pages   = {69--82},
  year    = {1970}
}

@book{hastie2009elements,
  author    = {Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome},
  title     = {The Elements of Statistical Learning: Data Mining, Inference, and Prediction},
  edition   = {2},
  publisher = {Springer},
  year      = {2009},
  note      = {Chapter 3, linear methods for regression; §3.4 shrinkage methods}
}

@book{hastie2015sparsity,
  author    = {Hastie, Trevor and Tibshirani, Robert and Wainwright, Martin},
  title     = {Statistical Learning with Sparsity: The Lasso and Generalizations},
  publisher = {CRC Press},
  year      = {2015}
}

@article{james1961estimation,
  author  = {James, William and Stein, Charles},
  title   = {Estimation with Quadratic Loss},
  journal = {Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability},
  volume  = {1},
  pages   = {361--379},
  year    = {1961}
}

@book{tikhonov1977solutions,
  author    = {Tikhonov, Andrey N. and Arsenin, Vasiliy Y.},
  title     = {Solutions of Ill-Posed Problems},
  publisher = {Winston and Sons, Washington},
  year      = {1977}
}