Week 11 — Model Selection and Learning Theory
Why does any of this work? The statistical learning theory that bounds generalization error.
Week 11 — Model Selection and Learning Theory
Why does any of this work? The statistical learning theory that bounds generalization error.
Lecture
The PAC framework · VC dimension and uniform convergence · Rademacher complexity · the bias-variance decomposition rigorously · structural risk minimization · model selection via AIC, BIC, cross-validation.
Read before the lecture
- Hastie, Tibshirani, Friedman, chapter 7
Problem set
PS6 — Learning theory
- Prove the basic VC dimension upper bound for half-spaces in $\mathbb{R}^d$.
- Derive a generalization bound using Rademacher complexity for a linear class.
Reference text for this week: chapter 11 of the bilingual notes — EN PDF · FR PDF.