Elastic Net and Least Angle Regression

Intuition Beginner

The lasso is a sharp tool, but it has two habits that get in the way. First, when two predictors carry nearly the same information — two columns of data that move together — the lasso will usually keep one and switch the other off entirely, and which one it keeps can flip with a tiny change in the data. That is jumpy and hard to trust. Second, when you have more predictors than data points, the lasso refuses to keep more than a handful: it caps the number it selects at the number of rows you have. In genetics or text, where predictors vastly outnumber samples, that cap is a real limit.

The elastic net fixes both by mixing two charges. It keeps the lasso's absolute-value charge, which produces the clean on-or-off selection, and adds a small dose of ridge's squared-size charge, which gently ties together predictors that move together. The squared charge has no favorites among a correlated group, so instead of picking one and dropping the rest, the elastic net keeps the whole group and splits the weight among them. That is the grouping effect, and it is exactly what you want when several measurements are really pointing at one underlying cause.

The second tool in this unit is about how you compute the whole family of solutions, not just one. As you loosen the budget, predictors switch on one at a time at definite thresholds, and between thresholds every weight changes in a perfectly straight line. Least angle regression walks along these straight pieces, finding each switch-on point exactly, so it draws the entire path of solutions for the price of about one ordinary fit.

The everyday image: the lasso is a strict editor that cuts every redundant word; the elastic net is an editor that keeps a useful synonym pair together; and least angle regression is the time-lapse that shows you every edit as the budget relaxes.

Visual Beginner

Method	Penalty charged	Correlated group of predictors	Can keep more than (rows) variables?
Ridge	squared size only	shrinks the whole group together, keeps all	yes, but never exact zeros
Lasso	absolute size only	tends to keep one, drop the rest	no, capped at the number of rows
Elastic net	both, mixed	keeps the group together, with exact zeros elsewhere	yes

The single picture to carry forward: the elastic-net region is a lasso diamond with rounded corners. The corners survive, so selection survives; the rounding ties correlated predictors together. And the path is built from straight line segments with breakpoints, which is why the whole family of solutions is cheap to draw.

Worked example Beginner

Take two predictors that are perfectly correlated: the second column is an exact copy of the first. Standardize both, and suppose the lasso would put a total weight of $6$ on this pair to fit the data. Because the two columns are identical, the lasso cannot tell them apart, and any split that adds to $6$ — all on the first, all on the second, or anything between — fits equally well and costs the same absolute-value charge. The lasso picks one of these arbitrarily, often all $6$ on one column and $0$ on the other.

Now add a small squared-size charge, the ridge part of the elastic net. The squared charge of a split $(a,b)$ with $a+b=6$ is $a^2+b^2$. Try the lopsided split $(6,0)$: its squared charge is $36+0=36$. Try the even split $(3,3)$: its squared charge is $9+9=18$. The even split costs half as much.

Check a middle case, $(4,2)$: that gives $16+4=20$, still more than $18$. The smallest squared charge for a fixed total always comes from sharing equally.

What this tells us: adding the squared charge breaks the lasso's tie in favor of splitting the weight evenly across identical predictors, instead of dumping it all on one. With real (not identical) predictors the effect is gentler, but the direction is the same — correlated predictors are kept together and share the load. That is the grouping effect in one calculation.

Check your understanding Beginner

Formal definition Intermediate+

Fix a design matrix $X\in {\mathbb{R}}^{n\times p}$ with columns $X_j$ and response $y\in {\mathbb{R}}^n$, predictors centered and standardized so that $\frac{1}{n}\Vert X_j{\Vert }_2^2=1$, the intercept fitted unpenalized. For penalty parameters ${\lambda }_1\ge 0$ and ${\lambda }_2\ge 0$, the naive elastic net estimator is the minimizer of the doubly penalized criterion

$${\hat{\beta }}^{\mathrm{naive}}({\lambda }_1,{\lambda }_2)=\arg \underset{\beta \in {\mathbb{R}}^p}{\min }\{\frac{1}{2}\Vert y-X\beta {\Vert }_2^2+{\lambda }_1\Vert \beta {\Vert }_1+{\lambda }_2\Vert \beta {\Vert }_2^2\},\Vert \beta {\Vert }_2^2=\sum _{j=1}^p{\beta }_j^2.$$

The criterion is the sum of the lasso objective 45.06.06 and the ridge penalty 45.06.03. The ${\ell }_2$ term makes the objective strongly convex for ${\lambda }_2>0$: the Hessian of the smooth part is $X^{\top }X+2{\lambda }_{2}I⪰2{\lambda }_{2}I$, positive definite regardless of the rank of $X^{\top }X$, so the minimizer is unique even when $p>n$. The optimality condition combines the ridge gradient with the lasso subgradient: $\hat{\beta }$ is the minimizer if and only if there is $s\in \partial \Vert \hat{\beta }{\Vert }_1$ with

$$X^{\top }(y-X\hat{\beta })=2{\lambda }_2\hat{\beta }+{\lambda }_{1}s,s_j=\mathrm{sign}({\hat{\beta }}_j)\text{if}{\hat{\beta }}_j\ne 0,s_j\in [-1,1]\text{if}{\hat{\beta }}_j=0.$$

The inactive-coordinate clause $|X_j^{\top }(y-X\hat{\beta })|\le {\lambda }_1$ still produces exact zeros, so selection is preserved; the active-coordinate clause now carries the extra ridge shrinkage $2{\lambda }_2{\hat{\beta }}_j$, which is what couples correlated predictors.

The augmented-design representation turns the elastic net into a pure lasso. Define

$$X_{\ast }=\frac{1}{\sqrt{1+2{\lambda }_2}}\left(\begin{array}{c}X\\ \sqrt{2{\lambda }_2}{I}_p\end{array}\right)\in {\mathbb{R}}^{(n+p)\times p},y_{\ast }=\left(\begin{array}{c}y\\ 0\end{array}\right)\in {\mathbb{R}}^{n+p}.$$

A direct computation (Proposition 1) shows the naive elastic net solves a lasso on $(X_{\ast },y_{\ast })$ with parameter $\gamma ={\lambda }_1/\sqrt{1+2{\lambda }_2}$. Because $X_{\ast }$ has full column rank $p$ for any ${\lambda }_2>0$ — its lower block $\sqrt{2{\lambda }_2}{I}_p$ alone supplies $p$ independent rows — the elastic net can select up to all $p$ predictors, removing the lasso's cap of $\min (n,p)$ active variables. Following Zou & Hastie ^{[Zou, H. & Hastie, T. — Regularization and Variable Selection via the Elastic Net]}, the corrected (proper) elastic net rescales away the double shrinkage,

$$\hat{\beta }({\lambda }_1,{\lambda }_2)=(1+2{\lambda }_2){\hat{\beta }}^{\mathrm{naive}}({\lambda }_1,{\lambda }_2),$$

since the augmented lasso shrinks once by ${\ell }_1$ and again by the $1/\sqrt{1+2{\lambda }_2}$ scaling of $X_{\ast }$; undoing the latter restores prediction quality while keeping the support and the grouping. (Zou & Hastie write the criterion as ${\lambda }_1\Vert \beta {\Vert }_1+{\lambda }_2\Vert \beta {\Vert }_2^2$ and obtain the correction factor $1+{\lambda }_2$; the factor $2$ here is the bookkeeping of writing the ridge term as ${\lambda }_2\Vert \beta {\Vert }_2^2$ rather than $\frac{{\lambda }_2}{2}\Vert \beta {\Vert }_2^2$.)

Least angle regression (LARS) is the algorithm that constructs the lasso solution path $\lambda \mapsto \hat{\beta }(\lambda )$. With residual $r=y-X\hat{\beta }$ and active set $A=\{j:{\hat{\beta }}_j\ne 0\}$, the lasso KKT condition forces every active predictor to share the maximal absolute correlation $|X_j^{\top }r|=\lambda$. LARS moves the coefficient vector in the equiangular direction

$$u_A=X_A{w}_A,w_A=c(X_A^{\top }{X}_A{)}^{-1}\mathrm{sign}(X_A^{\top }r),$$

with $c$ chosen so that $\Vert u_A\Vert$ makes every active predictor's correlation with the residual decrease at the same rate — the direction that keeps all active correlations tied. It proceeds in straight segments, adding the predictor that next ties at the threshold, until a coefficient is about to change sign. The lasso modification of Efron, Hastie, Johnstone & Tibshirani ^{[Efron, B., Hastie, T., Johnstone, I. & Tibshirani, R. — Least Angle Regression]} is a single rule: whenever an active coefficient would cross zero, remove that index from $A$ at the crossing and recompute $u_A$. With this one change, LARS traces the exact lasso path.

Counterexamples to common slips

• The naive elastic net is not the corrected estimator. The naive solution carries double shrinkage and predicts worse than ridge in correlated designs; only after the $(1+2{\lambda }_2)$ rescaling does prediction improve. Reporting the naive coefficients as "the elastic net" understates the signal.
• The augmented design must be standardized correctly. The $1/\sqrt{1+2{\lambda }_2}$ factor in $X_{\ast }$ is not cosmetic — it is what makes the augmented lasso's parameter $\gamma ={\lambda }_1/\sqrt{1+2{\lambda }_2}$ reproduce the elastic-net objective exactly. Dropping it changes the effective penalty.
• LARS is not forward selection. Greedy forward selection adds a predictor and refits it to its full least-squares value; LARS moves only partway, stopping the instant a competitor ties, so it never overcommits to a single predictor. The two coincide only when predictors are orthogonal.
• The unmodified LARS path is not the lasso path in general. Plain LARS only adds variables; the lasso can also drop a variable when its coefficient hits zero. Without the sign-crossing modification, the LARS and lasso paths diverge precisely at such drop events.

Key theorem with proof Intermediate+

The signature result is the elastic-net grouping bound: two predictors that are highly correlated receive nearly identical coefficients, with the gap controlled by the ridge parameter and the sample correlation. This is the precise statement of the grouping effect that the lasso lacks.

Theorem (grouping effect — Zou & Hastie 2005). Let $\hat{\beta }={\hat{\beta }}^{\mathrm{naive}}({\lambda }_1,{\lambda }_2)$ be the naive elastic-net estimator with ${\lambda }_2>0$ on standardized predictors ($\frac{1}{n}\Vert X_j{\Vert }_2^2=1$, equivalently $\Vert X_j{\Vert }_2^2=n$). Suppose two coordinates $i,j$ are active with the same sign, ${\hat{\beta }}_i{\hat{\beta }}_j>0$. Let ${\rho }_{ij}=\frac{1}{n}{X}_i^{\top }{X}_j$ be the sample correlation of the two predictors. Then

$$|{\hat{\beta }}_i-{\hat{\beta }}_j|\le \frac{\Vert y{\Vert }_2}{{\lambda }_2}\sqrt{2n(1-{\rho }_{ij})}.$$

In particular, as ${\rho }_{ij}\to 1$ the bound tends to zero: nearly identical predictors receive nearly identical coefficients. The argument follows Zou & Hastie ^{[Zou, H. & Hastie, T. — Regularization and Variable Selection via the Elastic Net]}.

Proof. Because ${\hat{\beta }}_i,{\hat{\beta }}_j\ne 0$ and have the same sign, the active-coordinate optimality conditions hold with $s_i=s_j=\mathrm{sign}({\hat{\beta }}_i)=\mathrm{sign}({\hat{\beta }}_j)$. Writing $r=y-X\hat{\beta }$ for the residual, the stationarity equations for the two coordinates read

$$X_i^{\top }r=2{\lambda }_2{\hat{\beta }}_i+{\lambda }_1{s}_i,X_j^{\top }r=2{\lambda }_2{\hat{\beta }}_j+{\lambda }_1{s}_j.$$

Subtract the second from the first. Since $s_i=s_j$, the ${\lambda }_1$ terms cancel, leaving

$$(X_i-X_j{)}^{\top }r=2{\lambda }_2({\hat{\beta }}_i-{\hat{\beta }}_j),\text{so}{\hat{\beta }}_i-{\hat{\beta }}_j=\frac{1}{2{\lambda }_2}(X_i-X_j{)}^{\top }r.$$

Bound the right side by Cauchy-Schwarz: $|(X_i-X_j{)}^{\top }r|\le \Vert X_i-X_j{\Vert }_2\Vert r{\Vert }_2$. The squared length of the difference is

$$\Vert X_i-X_j{\Vert }_2^2=\Vert X_i{\Vert }_2^2+\Vert X_j{\Vert }_2^2-2X_i^{\top }{X}_j=n+n-2n{\rho }_{ij}=2n(1-{\rho }_{ij}),$$

using the standardization $\Vert X_i{\Vert }_2^2=\Vert X_j{\Vert }_2^2=n$. For the residual, evaluate the elastic-net objective at $\hat{\beta }$ and at $0$: since $\hat{\beta }$ is the minimizer, $\frac{1}{2}\Vert y-X\hat{\beta }{\Vert }_2^2\le \frac{1}{2}\Vert y-X\hat{\beta }{\Vert }_2^2+{\lambda }_1\Vert \hat{\beta }{\Vert }_1+{\lambda }_2\Vert \hat{\beta }{\Vert }_2^2\le \frac{1}{2}\Vert y{\Vert }_2^2$, the value at $0$. Hence $\Vert r{\Vert }_2=\Vert y-X\hat{\beta }{\Vert }_2\le \Vert y{\Vert }_2$. Combining,

$$|{\hat{\beta }}_i-{\hat{\beta }}_j|=\frac{|(X_i-X_j{)}^{\top }r|}{2{\lambda }_2}\le \frac{\sqrt{2n(1-{\rho }_{ij})}\Vert y{\Vert }_2}{2{\lambda }_2}=\frac{\Vert y{\Vert }_2}{{\lambda }_2}\sqrt{\frac{n(1-{\rho }_{ij})}{2}}.$$

The displayed bound $\frac{\Vert y{\Vert }_2}{{\lambda }_2}\sqrt{2n(1-{\rho }_{ij})}$ is the same inequality with the constant relaxed from $\sqrt{1/2}$ to $\sqrt{2}$, which holds a fortiori. $\square$

Bridge. The proof's engine is differencing the two active stationarity equations so that the nondifferentiable ${\lambda }_1$ subgradient terms cancel for same-sign coefficients, leaving a purely ${\ell }_2$ relation ${\hat{\beta }}_i-{\hat{\beta }}_j=(2{\lambda }_2{)}^{-1}(X_i-X_j{)}^{\top }r$ whose right side is small exactly when the columns are close. This builds toward the augmented-lasso and naive-versus-corrected theory of the Master tier, where the same strong convexity that makes ${\lambda }_2$ appear in the denominator is what removes the lasso's selection cap, and it appears again in the LARS path analysis, where the active-set stationarity $|X_j^{\top }r|=\lambda$ for all $j\in A$ is the very condition that makes the path piecewise linear. The foundational reason the elastic net groups while the lasso does not is that the lasso's penalty is linear in each coordinate, so its subgradient carries no information tying two coordinates together, whereas the strictly convex ${\ell }_2$ term contributes the coupling $2{\lambda }_2({\hat{\beta }}_i-{\hat{\beta }}_j)$; this is exactly the ridge contraction of 45.06.03 acting differentially across a correlated pair. The result generalises the ridge grouping observation rather than contradicting the lasso: at ${\lambda }_2=0$ the bound degenerates ($1/{\lambda }_2\to \infty$) and the lasso's arbitrary tie-breaking returns, and the central insight is that a positive ${\ell }_2$ dose converts the flat lasso ridge of equally optimal splits into a strictly convex valley with a unique, balanced minimizer. Putting these together, the grouping bound is the differential form of the strong convexity that the augmented-design representation makes global.

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib does not yet host the elastic net or the LARS path, so no module is wired in (lean_status: none). The intended statements of strong convexity, the augmented-lasso reduction, and the grouping bound, once a penalized-empirical-risk API exists, would read roughly as follows.

-- Intended shape; not part of the current Babel Bible Lean build.
-- Requires: an ℓ₁/ℓ₂-penalized least-squares functional and the subgradient of ‖·‖₁.
variable {n p : ℕ} (X : Matrix (Fin n) (Fin p) ℝ) (y : Fin n → ℝ) (lam1 lam2 : ℝ)

-- The naive elastic-net objective (1/2)‖y - X β‖² + lam1 ‖β‖₁ + lam2 ‖β‖₂².
noncomputable def enetObj (β : Fin p → ℝ) : ℝ :=
  (1 / 2) * ‖y - X *ᵥ β‖^2 + lam1 * ∑ j, |β j| + lam2 * ‖β‖^2

-- Strong convexity for lam2 > 0 ⇒ a unique minimizer even when p > n.
theorem enet_stronglyConvex (hlam2 : 0 < lam2) :
    StrongConvexOn Set.univ (2 * lam2) (enetObj X y lam1 lam2) := by
  sorry

-- Grouping bound: same-sign active coefficients of correlated predictors are close.
theorem enet_grouping
    (hlam2 : 0 < lam2) (βhat : Fin p → ℝ)
    (hmin : IsMinOn (enetObj X y lam1 lam2) Set.univ βhat)
    (i j : Fin p) (hi : βhat i ≠ 0) (hj : βhat j ≠ 0)
    (hsign : 0 < βhat i * βhat j)
    (ρ : ℝ) (hρ : ρ = (1 / n) * (Xᵀ * X) i j) :
    |βhat i - βhat j| ≤ (‖y‖ / lam2) * Real.sqrt (2 * n * (1 - ρ)) := by
  sorry

the Mathlib gap analysis records what is missing: the elastic-net functional and its 2λ₂-strong convexity, the augmented-design reduction X* = (1+2λ₂)^{-1/2}[X; √(2λ₂) I] to a lasso, the corrected-estimator scaling, the grouping bound, the piecewise-linear-in-λ lasso solution path, the LARS equiangular-direction construction with the sign-crossing modification, and the degrees-of-freedom-equals-active-set identity along the path.

Advanced results Master

The naive elastic net carries a double shrinkage that the augmented-lasso representation makes precise. Solving the lasso on $(X_{\ast },y_{\ast })$ shrinks once through the ${\ell }_1$ threshold and a second time through the contraction implicit in the $1/\sqrt{1+2{\lambda }_2}$ scaling of $X_{\ast }$, so the naive coefficients underpredict; in the orthonormal case Exercise 4 shows the exact factor $1/(1+2{\lambda }_2)$. The corrected estimator $\hat{\beta }=(1+2{\lambda }_2){\hat{\beta }}^{\mathrm{naive}}$ removes the second shrinkage while preserving the support and the grouping, and Zou & Hastie ^{[Zou, H. & Hastie, T. — Regularization and Variable Selection via the Elastic Net]} document that on correlated designs the corrected estimator improves prediction over both the lasso (which over-selects and tie-breaks erratically) and ridge (which never selects). The mixing is usually parameterized by $\alpha ={\lambda }_2/({\lambda }_1+{\lambda }_2)\in [0,1]$, interpolating from the lasso ($\alpha =0$) to ridge ($\alpha =1$), with $\lambda$ controlling overall strength; cross-validation selects the pair, and warm-started coordinate descent solves the whole $(\alpha ,\lambda )$ grid.

Least angle regression turns the lasso from a single convex program into a homotopy. The path $\lambda \mapsto \hat{\beta }(\lambda )$ is continuous and piecewise affine (Exercise 6), with knots ${\lambda }_1>{\lambda }_2>\cdots >0$ at which the active set changes. Between knots, ${\hat{\beta }}_A(\lambda )=(X_A^{\top }{X}_A{)}^{-1}(X_A^{\top }y-n\lambda s_A)$ runs along a line; at each knot LARS solves a $|A|\times |A|$ system to update the equiangular direction (Exercise 7) and a one-dimensional search to find the next join-or-drop event. Computing $(X_A^{\top }{X}_A{)}^{-1}$ incrementally by rank-one Cholesky updates costs $O(|A{|}^2)$ per knot, and with $O(p)$ knots in the typical regime the full lasso path costs $O(p^3+n{p}^2)$ — the order of a single ordinary least-squares fit, but delivering every solution along the way. This is the computational payoff that made the lasso practical: one does not refit at each $\lambda$; one walks the path. Efron, Hastie, Johnstone & Tibshirani ^{[Efron, B., Hastie, T., Johnstone, I. & Tibshirani, R. — Least Angle Regression]} also placed LARS between the greedy forward selection and the infinitesimal forward-stagewise procedure, showing all three trace nearly the same profiles, with LARS the least committal and stagewise the most cautious.

The degrees of freedom along the path are exact and remarkably simple. For the LARS algorithm stopped after $k$ steps, the degrees of freedom equal $k$; for the lasso at a fixed $\lambda$, the degrees of freedom equal the expected size of the active set, $\mathbb{E}|\hat{S}(\lambda )|$. Both follow from Stein's identity applied to the piecewise-affine fitted-value map (Proposition 3), the same divergence computation that gave the lasso df 45.06.06 and that links back to the Stein shrinkage theory of 45.06.04. The simplicity is structural: on each linear piece the fit is the projection onto the span of the active columns, whose trace is the active-set size, and the knots form a null set, so the expected divergence is the expected active-set cardinality. This makes information criteria (Mallows-type, AIC, BIC) directly computable along the path, turning the lasso/elastic-net path into a model-selection device 45.06.09: each knot is a candidate model with a known complexity, and one selects among them by the penalized criterion or by cross-validation, reading off the chosen $\lambda$ from the path.

The modern solver is not LARS but coordinate descent 44.06.07. Because the lasso and elastic-net objectives are smooth plus a separable nonsmooth penalty, the optimality condition decouples coordinate by coordinate, and cyclically minimizing one coordinate at a time — each subproblem solved in closed form by the soft-threshold-then-ridge-scale update of Exercise 4, the ${\ell }_1$ proximal step of 44.06.02 — converges to the global minimizer. Computing the path along a decreasing grid of $\lambda$ with warm starts (initialize each fit at the previous solution) makes coordinate descent faster than LARS in practice, especially for large sparse $X$ where touching one column at a time exploits sparsity. The elastic net's ${\ell }_2$ term improves coordinate descent's conditioning, so the grouping that helps statistically also helps the solver. The two algorithms are complementary: LARS gives the exact knots and the df bookkeeping, coordinate descent gives the fast grid solve.

Synthesis. The foundational reason the elastic net and LARS belong to one unit is that a single optimality system governs both the penalty and the path: the stationarity condition $X^{\top }(y-X\hat{\beta })=2{\lambda }_2\hat{\beta }+{\lambda }_{1}s$, $s\in \partial \Vert \hat{\beta }{\Vert }_1$, produces the grouping bound by differencing two active equations, removes the lasso's selection cap through the strong convexity its ${\ell }_2$ term supplies, and — specialized to ${\lambda }_2=0$ — yields the piecewise-affine active-set solution ${\hat{\beta }}_A(\lambda )=(X_A^{\top }{X}_A{)}^{-1}(X_A^{\top }y-n\lambda s_A)$ that LARS walks. This is exactly the ${\ell }_1$-plus-${\ell }_2$ specialization of the proximal/subgradient theory of 44.06.02, whose soft-threshold prox is the coordinate update 44.06.07 that solves the path, and whose composite Fermat rule $0\in \nabla f+\partial g$ is the elastic-net KKT system.

The central insight is that the elastic net generalises ridge 45.06.03 and the lasso 45.06.06 simultaneously — it is the bridge penalty's interior, inheriting ridge's grouping of correlated predictors and the lasso's corner-geometry selection — and that the corrected rescaling $(1+2{\lambda }_2)$ is precisely the undoing of the augmented-design contraction, so naive and corrected estimators share a support but differ by a known factor. Putting these together, the degrees of freedom along the path is dual to the geometry of the knots: each knot adds or drops one active column, the fit is the projection onto the active span, and Stein's identity 45.06.04 turns that projection trace into the expected active-set size, which is exactly what makes the path a calibrated model-selection ladder 45.06.09. The bridge from a single penalized program to an entire regularization path is the piecewise linearity that the ${\ell }_1$ subgradient and the active-set linear system together force.

Full proof set Master

The grouping bound, the augmented-design reduction, the orthonormal elastic-net solution, the piecewise-linear path, and the equiangular direction are proved in full in the Formal definition, Key theorem, and Exercises sections. The remaining Master claims are recorded here.

Proposition 1 (augmented-design reduction and the $p$-variable capacity). With $X_{\ast }=(1+2{\lambda }_2{)}^{-1/2}[X;\sqrt{2{\lambda }_2}{I}_p]$ and $y_{\ast }=[y;0]$, the naive elastic net with parameters $({\lambda }_1,{\lambda }_2)$ minimizes, in the variable $b=(1+2{\lambda }_2{)}^{-1/2}\beta$, the lasso objective $\frac{1}{2}\Vert y_{\ast }-X_{\ast }{b}^{\prime }{\Vert }_2^2+\gamma \Vert b^{\prime }{\Vert }_1$ with $\gamma ={\lambda }_1/\sqrt{1+2{\lambda }_2}$, and $X_{\ast }$ has full column rank $p$ for every ${\lambda }_2>0$, so the estimator may have up to $p$ nonzero coordinates.

Proof. Expand the augmented residual sum of squares with $\beta$ as variable:

$$\frac{1}{2}\Vert y_{\ast }-X_{\ast }\beta {\Vert }_2^2=\frac{1}{2}\Vert y-(1+2{\lambda }_2{)}^{-1/2}X\beta {\Vert }_2^2+\frac{1}{2}(1+2{\lambda }_2{)}^{-1}\Vert \sqrt{2{\lambda }_2}\beta {\Vert }_2^2.$$

Substitute the change of variable $\beta =(1+2{\lambda }_2{)}^{1/2}b$ (so $(1+2{\lambda }_2{)}^{-1/2}X\beta =Xb$ and $(1+2{\lambda }_2{)}^{-1}\cdot 2{\lambda }_2\Vert \beta {\Vert }_2^2=2{\lambda }_2\Vert b{\Vert }_2^2$):

$$\frac{1}{2}\Vert y_{\ast }-X_{\ast }\beta {\Vert }_2^2=\frac{1}{2}\Vert y-Xb{\Vert }_2^2+{\lambda }_2\Vert b{\Vert }_2^2.$$

The penalty becomes $\gamma \Vert \beta {\Vert }_1=\gamma (1+2{\lambda }_2{)}^{1/2}\Vert b{\Vert }_1={\lambda }_1\Vert b{\Vert }_1$ exactly when $\gamma ={\lambda }_1/\sqrt{1+2{\lambda }_2}$. So the augmented lasso in $\beta$ equals the naive elastic net in $b$, and the minimizers correspond under $\beta =(1+2{\lambda }_2{)}^{1/2}b$. For the rank claim, the lower block $\sqrt{2{\lambda }_2}{I}_p$ has rank $p$ for ${\lambda }_2>0$; stacking rows can only increase rank, so $\mathrm{rank}(X_{\ast })=p$. A lasso on a full-column-rank design has no a priori cap on the number of active variables (the $\min (n,p)$ bound for the plain lasso comes from $X$ alone having rank $\le n$), so the elastic net can activate all $p$. $\square$

Proposition 2 (strong convexity and uniqueness for $p>n$). For ${\lambda }_2>0$ the naive elastic-net objective is $2{\lambda }_2$-strongly convex on ${\mathbb{R}}^p$, hence has a unique minimizer for every $X$ (including $p>n$, where $X^{\top }X$ is singular).

Proof. The objective is $f(\beta )+g(\beta )$ with $f(\beta )=\frac{1}{2}\Vert y-X\beta {\Vert }_2^2+{\lambda }_2\Vert \beta {\Vert }_2^2$ smooth and $g(\beta )={\lambda }_1\Vert \beta {\Vert }_1$ convex. The Hessian of $f$ is ${\nabla }^{2}f=X^{\top }X+2{\lambda }_{2}I$. Since $X^{\top }X⪰0$, ${\nabla }^{2}f⪰2{\lambda }_{2}I\succ 0$, so $f$ is $2{\lambda }_2$-strongly convex; adding the convex $g$ preserves $2{\lambda }_2$-strong convexity. A strongly convex function on ${\mathbb{R}}^p$ is coercive and has a unique global minimizer, independent of whether $X^{\top }X$ is invertible. This is the structural advantage over the lasso (${\lambda }_2=0$), whose objective is convex but not strictly convex when $p>n$, allowing a polytope of minimizers 45.06.06. $\square$

Proposition 3 (degrees of freedom along the path). For $y\sim N(\mu ,{\sigma }^2{I}_n)$: (i) the LARS fit after $k$ steps has $\mathrm{df}=k$; (ii) the lasso fit at fixed $\lambda$ has $\mathrm{df}(\lambda )=\mathbb{E}|\hat{S}(\lambda )|$, the expected active-set size.

Proof (following Efron-Hastie-Johnstone-Tibshirani via Stein). Effective degrees of freedom is $\mathrm{df}=\frac{1}{{\sigma }^2}{\sum }_i\mathrm{Cov}({\hat{y}}_i,y_i)=\mathbb{E}[\mathrm{div}\hat{y}(y)]$ by Stein's identity (the Gaussian integration-by-parts of 45.06.04), valid for the continuous, almost-everywhere-differentiable fitted-value map. For the lasso at fixed $\lambda$, on the open region where the active set $\hat{S}$ and signs are locally constant, the fit is $\hat{y}=X_{\hat{S}}(X_{\hat{S}}^{\top }{X}_{\hat{S}}{)}^{-1}(X_{\hat{S}}^{\top }y-n\lambda {\mathrm{sign}}_{\hat{S}})$, whose Jacobian in $y$ is the projection $P_{\hat{S}}=X_{\hat{S}}(X_{\hat{S}}^{\top }{X}_{\hat{S}}{)}^{-1}{X}_{\hat{S}}^{\top }$ of rank $|\hat{S}|$. Its divergence (trace) is $|\hat{S}|$; the knot set where $\hat{S}$ changes is Lebesgue-null, so $\mathrm{div}\hat{y}=|\hat{S}(\lambda )|$ almost everywhere, giving (ii). For LARS after $k$ steps the active set has size $k$ on each path segment by construction (one variable added per step, modulo drops), and the same projection-trace argument gives divergence $k$; the simple-knot covariance lemma of Efron et al. handles the boundary contributions, yielding $\mathrm{df}=k$, statement (i). $\square$

Proposition 4 (coordinate descent solves the elastic net). For ${\lambda }_2\ge 0$ and ${\lambda }_1>0$, cyclic coordinate minimization of the elastic-net objective, with each coordinate update ${\hat{\beta }}_j\leftarrow S_{{\lambda }_1}(z_j^{(-j)})/(1+2{\lambda }_2+\frac{1}{n}\Vert X_j{\Vert }_2^2-1)$ where $z_j^{(-j)}=\frac{1}{n}{X}_j^{\top }(y-X_{-j}{\hat{\beta }}_{-j})$ is the partial residual correlation, converges to the global minimizer.

Proof. The objective is $f(\beta )+{\sum }_j{g}_j({\beta }_j)$ with $f$ smooth convex and each $g_j({\beta }_j)={\lambda }_1|{\beta }_j|$ convex — a smooth-plus-separable composite, the structure for which block-coordinate descent reaches a global minimizer 44.06.07 (the optimality condition $0\in \nabla f+\partial g$ decouples across coordinates because $\partial g={\prod }_j\partial g_j$). Each one-coordinate subproblem minimizes $\frac{1}{2}{a}_j{\beta }_j^2-z_j^{(-j)}{\beta }_j+{\lambda }_2{\beta }_j^2+{\lambda }_1|{\beta }_j|$ with $a_j=\frac{1}{n}\Vert X_j{\Vert }_2^2=1$ for standardized columns; collecting quadratic terms gives coefficient $(a_j+2{\lambda }_2)$ and the closed-form soft-threshold-then-scale update ${\hat{\beta }}_j=S_{{\lambda }_1}(z_j^{(-j)})/(a_j+2{\lambda }_2)$, the ${\ell }_1$ proximal step of 44.06.02 specialized to the elastic net. Because the nonsmooth part is separable and the smooth part has a Lipschitz coordinatewise gradient, the cyclic sweeps generate a decreasing objective sequence whose limit points are coordinatewise stationary, hence (by separability) globally optimal. $\square$

Connections Master

The lasso and its oracle theory 45.06.06 is the method the elastic net refines: the elastic net keeps the lasso's ${\ell }_1$ corner geometry and exact-zero selection but adds an ${\ell }_2$ dose that repairs the lasso's two structural defects — its cap of $\min (n,p)$ selected variables and its erratic all-but-one behaviour among correlated predictors — while LARS is the homotopy that constructs the lasso solution path, with its one-line sign-crossing modification turning the additive LARS sequence into the exact lasso path that drops as well as adds variables.

Ridge regression and shrinkage 45.06.03 supplies the ${\ell }_2$ penalty whose differential action across a correlated pair is the grouping effect: the elastic-net stationarity term $2{\lambda }_2({\hat{\beta }}_i-{\hat{\beta }}_j)$ is ridge's smooth contraction made to act on coefficient differences, and the corrected estimator $(1+2{\lambda }_2){\hat{\beta }}^{\mathrm{naive}}$ undoes exactly the ridge-induced double shrinkage that the augmented-design lasso introduces.

The proximal operator and proximal-gradient method 44.06.02 gives the closed-form coordinate update: the elastic-net coordinate solution $S_{{\lambda }_1}(z)/(1+2{\lambda }_2)$ is the proximal operator of ${\lambda }_1|\cdot |+{\lambda }_2(\cdot {)}^2$, the soft-threshold-then-shrink composition, and the elastic-net KKT system is the composite Fermat rule $0\in \nabla f+\partial g$ of that unit.

Coordinate descent 44.06.07 is the modern path solver: because the elastic-net penalty is smooth-plus-separable, cyclic coordinate minimization with warm starts along a $\lambda$ grid converges to the global minimizer and outperforms LARS on large sparse designs, and the ${\ell }_2$ term improves the coordinate subproblems' conditioning, so the statistical grouping device doubles as a numerical preconditioner.

Subset selection and model selection 45.06.09 is where the path becomes a model-selection tool: the piecewise-linear lasso/elastic-net path visits a nested sequence of candidate models at its knots, each with degrees of freedom equal to its active-set size, so information criteria and cross-validation select $\lambda$ by reading complexity directly off the path rather than refitting a combinatorial family of subsets.

The James-Stein estimator and Stein's identity 45.06.04 supplies the integration-by-parts behind the degrees-of-freedom result: the divergence formula $\mathrm{df}=\mathbb{E}[\mathrm{div}\hat{y}]$ that gives James-Stein its risk accounting is the same identity that equates the lasso path's degrees of freedom with the expected active-set size and the LARS df with the step count.

Historical & philosophical context Master

Least angle regression was introduced by Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani in their 2004 Annals of Statistics paper ^{[Efron, B., Hastie, T., Johnstone, I. & Tibshirani, R. — Least Angle Regression]}, read with discussion, which unified three apparently distinct procedures — greedy forward selection, the cautious forward-stagewise method long used in boosting, and the lasso of Tibshirani (1996) — by showing that a single equiangular geometry generates all of them, and that a one-line modification of the LARS algorithm produces the entire lasso regularization path at the computational cost of one ordinary least-squares fit. The same paper proved the degrees-of-freedom identity, that the LARS fit after $k$ steps has exactly $k$ degrees of freedom, via Stein's covariance formula, which retroactively justified counting model complexity by active-set size.

The elastic net was introduced the following year by Hui Zou and Trevor Hastie in their 2005 Journal of the Royal Statistical Society, Series B paper ^{[Zou, H. & Hastie, T. — Regularization and Variable Selection via the Elastic Net]}, motivated explicitly by three failures of the lasso on the microarray and other $p\gg n$ data then driving statistical genetics: the lasso could select at most $n$ variables, it behaved erratically among groups of correlated genes by keeping one and discarding the rest, and it was outperformed in prediction by ridge whenever predictors were correlated. Their combined ${\ell }_1+{\ell }_2$ penalty repaired all three, with the augmented-design representation reducing the problem to a lasso (so LARS-EN computes its path) and the grouping-effect bound giving a precise guarantee that correlated predictors receive similar coefficients. The naive-versus-corrected distinction, and the $(1+{\lambda }_2)$ rescaling that removes the double shrinkage, were established in the same paper. The synthesis of LARS, the lasso, the elastic net and coordinate-descent path solvers into a single account of regularized regression was consolidated by Hastie, Tibshirani and Friedman ^{[Hastie, T., Tibshirani, R. & Friedman, J. — The Elements of Statistical Learning (2nd ed.)]} in chapter 3 of The Elements of Statistical Learning.

Bibliography Master

@article{efron2004lars,
  author  = {Efron, Bradley and Hastie, Trevor and Johnstone, Iain and Tibshirani, Robert},
  title   = {Least Angle Regression},
  journal = {The Annals of Statistics},
  volume  = {32},
  number  = {2},
  pages   = {407--499},
  year    = {2004}
}

@article{zou2005elasticnet,
  author  = {Zou, Hui and Hastie, Trevor},
  title   = {Regularization and Variable Selection via the Elastic Net},
  journal = {Journal of the Royal Statistical Society, Series B},
  volume  = {67},
  number  = {2},
  pages   = {301--320},
  year    = {2005}
}

@book{hastie2009elements,
  author    = {Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome},
  title     = {The Elements of Statistical Learning: Data Mining, Inference, and Prediction},
  edition   = {2},
  publisher = {Springer},
  year      = {2009},
  note      = {Chapter 3, linear methods for regression; §3.4.2 LARS, §3.4.3 elastic net, §3.8.1 coordinate descent}
}

@article{tibshirani1996lasso,
  author  = {Tibshirani, Robert},
  title   = {Regression Shrinkage and Selection via the Lasso},
  journal = {Journal of the Royal Statistical Society, Series B},
  volume  = {58},
  number  = {1},
  pages   = {267--288},
  year    = {1996}
}

@article{tibshirani2013lassounique,
  author  = {Tibshirani, Ryan J.},
  title   = {The Lasso Problem and Uniqueness},
  journal = {Electronic Journal of Statistics},
  volume  = {7},
  pages   = {1456--1490},
  year    = {2013}
}

@article{friedman2010glmnet,
  author  = {Friedman, Jerome and Hastie, Trevor and Tibshirani, Robert},
  title   = {Regularization Paths for Generalized Linear Models via Coordinate Descent},
  journal = {Journal of Statistical Software},
  volume  = {33},
  number  = {1},
  pages   = {1--22},
  year    = {2010}
}

@book{hastie2015sparsity,
  author    = {Hastie, Trevor and Tibshirani, Robert and Wainwright, Martin},
  title     = {Statistical Learning with Sparsity: The Lasso and Generalizations},
  publisher = {CRC Press},
  year      = {2015}
}