Week 02 — Linear and Polynomial Regression

Gauss 1801 predicting Ceres from forty days of observations: the original machine-learning success.

ML  ·  schedule  ·  Week 02 of 12 ·  ← 01 ·  03 →

Week 02 — Linear and Polynomial Regression

Gauss 1801 predicting Ceres from forty days of observations: the original machine-learning success.

Lecture

The least-squares estimator (matrix form, projection geometry) · the Gauss-Markov theorem · polynomial regression · regression diagnostics (residuals, leverage, Cook’s distance) · the geometry of fitting.

Read before the lecture

  • Hastie, Tibshirani, Friedman, chapter 3

Problem set

PS1 — Linear regression by hand and by code

  1. Derive the least-squares estimator from first principles and prove the Gauss-Markov theorem.
  2. Implement linear regression in pure NumPy. Compare with sklearn on three real datasets.
  3. Construct a dataset where leverage and influence diverge significantly.

Reference text for this week: chapter 02 of the bilingual notes — EN PDF · FR PDF.