Week 04 — Regularization — Ridge, Lasso, Elastic Net

Hoerl-Kennard 1970 ridge regression: an industrial fix for unstable least squares. Tibshirani 1996 LASSO: variable selection by optimization.

Lecture

Ridge regression and the bias-variance trade-off · LASSO and the geometry of sparsity · elastic net · cross-validation for tuning the regularization parameter · the Bayesian interpretation (Gaussian and Laplace priors).

Read before the lecture

• Tibshirani, *Regression Shrinkage and Selection via the Lasso* (JRSS-B 1996)

Problem set

PS2 — Regularization theory

1. Prove that the ridge regression estimator is the MAP estimator under a Gaussian prior on coefficients.
2. Show that LASSO produces exact zeros in the solution while ridge does not.
3. Implement coordinate descent for LASSO from scratch.

Reference text for this week: chapter 04 of the bilingual notes — EN PDF · FR PDF.