Week 4’s wide regression: the training error fell at every step, the held-out error turned at step 15, and the number of steps was chosen on the held-out rows
Week 1’s seminar asked it first: “you are about to report the same held-out score you chose on”. Week 4’s asked it again
You chose the number of steps on the held-out rows. Which number do you report as the model’s error, and what would make it honest?
Today’s question: what structures can we model? Four new families, each a bet on the shape of \(f^\star\), and one protocol to compare them on
The task: 10,000 Spotify tracks
Predict a track’s popularity (Spotify’s score, 0 to 100) from eleven audio features (danceability, energy, loudness, valence, tempo, duration, …), its genre and its release year. Loss: squared error
10,000 tracks from playlists of six genres (data note). The variance of popularity is 564, the MSE of the model that predicts the mean for everyone
Three kinds of data, one rule each
Used for
How often
Training
fitting
as much as you like
Selection (books: validation)
choosing among fits: \(\lambda\), \(k\), \(J\), a depth, a method
as much as you like; the winner’s score is a selected minimum, so optimistic
Test, reserved
one number for the model you chose, with its standard error
once
ESL’s word for the third: a vault. Split once, at random, before looking (today 5,000, 2,000 and 3,000 tracks); refit the chosen model on training plus selection; open the vault once; report that number and no other
Week 1’s one rule, now for evaluation data: a row is fair game for scoring only if nothing about the model was chosen on it.
CTF competition: the baselines
What we can fit so far, each a bet on \(f^\star\): least squares, a plane (every input a slope); logistic regression, a plane for a probability (for a label, not today); \(k\) nearest neighbors, a local average, its \(k\) the knob
Eleven inputs and 5,000 training tracks: the local average (\(k = 100\)) does no better than a plane. Can a different shape beat 525? The test set stays shut until the seminar
Part 2: ridge and lasso
Regularized (i.e. penalized) regression
Motivation: week 4 stopped gradient descent early, and the number of steps became a knob that controlled how far the fit went. Can the knob be written into the objective instead?
For some penalty function \(\mathcal P_\lambda\), which depends on a tuning parameter \(\lambda\), the estimator
\[\hat \beta_\lambda = \arg \min_\beta \| \mathbf y - \mathbf X \beta \|^2_2 + \mathcal P_\lambda(\beta)\] is “regularized” or shrunk (shranken?) toward values that decrease the penalty. Often \(\mathcal P_\lambda = \lambda \| \cdot \|\) for some norm
Many ML methods are optimizing “loss + penalty”
Ridge (i.e. L2 penalized) regression
Originally motivated by problems where \(\mathbf X^T \mathbf X\) is uninvertible (or badly conditioned, i.e. almost uninvertible)
If \(p > n\) then this always happens
Least squares estimator is undefined or numerically unstable
For some constant \(\lambda > 0\), \[\text{minimize } \| \mathbf y - \mathbf X \beta \|^2_2 + \lambda \| \beta \|^2\]Shrinks coefficients \(\hat \beta\) toward 0
Larger coefficients are penalized more (squared penalty)
What \(\lambda\) controls
Set the gradient of \(\| \mathbf y - \mathbf X \beta \|^2 + \lambda \| \beta \|^2\) to zero (exercise 2): \[
(\mathbf X^T\mathbf X + \lambda I)\,\hat\beta_\lambda = \mathbf X^T\mathbf y
\]\[
\hat\beta_\lambda = (\mathbf X^T\mathbf X + \lambda I)^{-1}\mathbf X^T\mathbf y
\]
\(\mathbf X^T\mathbf X + \lambda I\) is invertible for every \(\lambda > 0\), whatever \(n\) and \(p\): one solution even when \(p > n\)
\(\lambda \to 0\): least squares (when it exists). \(\lambda \to \infty\): \(\hat\beta_\lambda \to \mathbf 0\). In between, one knob
Ridge paths: \(\hat\beta_\lambda\) against \(-\log\lambda\) (large \(\lambda\) on the left), simulated data, \(n = 100\), \(p = 20\), 8 true coefficients nonzero
Lasso vs ridge: L1 vs L2 norm penalties
A simple diff to remember lasso/ridge is via the penalty/constraint (1-norm instead of 2-norm). Lasso is
\[\text{minimize } \frac{1}{2n}\| \mathbf y - \mathbf X \beta \|_2^2 \text{ s. t. } \| \beta \|_1 \leq t\] where \(\| \beta \|_1 = \sum_{j=1}^p |\beta_j|\). Lagrangian form
\[\text{minimize } \frac{1}{2n} \| \mathbf y - \mathbf X \beta \|_2^2 + \lambda \| \beta \|_1\]
Lasso vs ridge: solution paths of \(\hat \beta\)
plot(ridge_fit, xvar ="lambda")
plot(lasso_fit, xvar ="lambda")
Same simulated data, large \(\lambda\) on the left: \(n = 100\), \(p = 20\), 8 nonzero coefficients. The lasso’s coefficients are exactly zero over whole ranges of \(\lambda\); ridge’s never are
High dimensional example
\(p = 200\) inputs, \(n = 100\) rows, 21 true coefficients nonzero; least squares has no unique fit. Ridge and lasso at the \(\lambda\) chosen on 100 fresh rows
Black dots the truth, hollow pink circles the lasso, blue crosses ridge. The lasso keeps 71 of 200 (17 of the 21 true ones; its extras are below 0.22); ridge keeps all 200
Lasso: cool or extremely cool?
High-dimensional: \(p > n\) means we can’t fit OLS
But all is not lost! Penalized regression to the rescue
True model is sparse
Only 21 of 200 variables have nonzero coefficients
Ridge estimates are dense
All coefficients nonzero 😰
Lasso estimates are sparse
Nonzero estimates largely coincide with true model 😎
Part 3: additive models
Three ways to estimate a curve
Week 1’s data: life expectancy on \(\log_{10}\) GDP per capita, 142 countries. Three ways to turn a line into a curve
Average nearby (\(k\)-NN; a kernel weights by distance instead of cutting at \(k\)). Fit a line nearby (loess). Fit a basis: fixed curves, combined by least squares. The course examines the third
A spline is regression on basis functions
A basis: fixed functions \(b_1(x), \ldots, b_J(x)\) of one input. The fit is their combination, \[
f(x) = \sum_{j=1}^J \beta_j\, b_j(x)
\] linear in \(\beta\), so least squares fits it: lm(lifeExp ~ ns(lgdp, df = J)), where ns() builds the \(J\) columns
A natural cubic spline: cubic pieces joined smoothly at knots, linear beyond the ends. Each \(b_j\) is local: it is zero far from its knots
The basis size is the knob
Small \(J\): almost a line. Large \(J\): the curve follows the points, and at \(J = n\) it passes through every one (training error zero). Which \(J\)? The selection set, as for \(k\) and \(\lambda\). Week 7: a penalty on wiggliness instead, with \(J\) as a ceiling
Additive modeling assumption
Linearity assumption: each predictor has a coefficient
Includes linear models as special case if \(f_j(\mathbf x_j) = \beta_j \mathbf x_j\)
Today each \(f_j\) is a spline with \(J\) basis functions, so the whole model is one least-squares fit on \(pJ\) columns: lm(y ~ ns(x1, J) + ns(x2, J) + ns(x3, J)). The GAM in the reading is this model, with a link
One least-squares fit, 17 coefficients, 53,388 films (data note). No coefficient to read, so how do we interpret?
Replace each linear coefficient with a 2d plot
Each curve is what the model adds to a film’s rating for that input, with the other held fixed: a coefficient’s job, done by a shape. The model is the sum of the two curves plus a constant
Part 4: trees and forests
Decision trees
Are you eligible for the COVID-19 vaccine?
If Age >= 50 then yes, otherwise continue
If HighRisk == TRUE then yes, otherwise continue
If Job == CareWorker then yes, otherwise no
This is (arguably) more interpretable than a linear model with multiple predictors
(Note: this is not the real vaccination criteria, but it was close to this in early 2021)
Input: subset of data
For each predictor variable x_j in subset
Split left: observations with x_j < cutoff
Split right: observations with x_j >= cutoff
Predict constants in each split
Compute model improvement
Scan cutoff value to find best split for x_j
Output: predictor and split with best improvement
Starting from full dataset, compute first split as above
Recurse: take the two subsets of data from each side of the split and plug them both back into the same function
Until some stopping rule prevents more splitting
Aggregating is… smoothing?
One tree on all the data, three on resamples of it. Predictions for one penguin
(“Smoothing” useful for low-bias, high-variance models)
Random forest: dropping predictors
Problem: aggregation does not increase information if the aggregates are highly correlated, e.g. averaging 1000 trees but each one uses the same small set of predictor variables
e.g. each split chooses among a random \(\sqrt{p}\) of the \(p\) predictors
Aggregate models which are now (typically) less correlated, achieving greater variance reduction
Part 5: which structure?
Three truths, one protocol
\(p = 10\) inputs, each uniform on \([-1, 1]\); \(y = f^\star(x) + \varepsilon\), noise variance \(\sigma^2 = 1\). Three truths, each using three (or two) of the ten inputs:
A, sparse linear:\(f^\star(x) = 2x_1 - 1.5x_2 + x_3\)
B, additive smooth:\(f^\star(x) = 1.5\,[f_1(x_1) + f_2(x_2) + f_3(x_3)]\), part 3’s three curves
One protocol. 300 training rows, 100 selection rows, 2,000 test rows. Six competitors: least squares; the lasso (\(\lambda\)); \(k\)-NN (\(k\)); the additive spline fit (\(J\)); a tree (depth); a forest (defaults). Every knob chosen on the selection rows, then scored once on the test rows, with the training MSE recorded beside it
Truth A: sparse linear, tall and then wide
Dotted line: the noise variance, the best any predictor can do. Sparsity costs nothing when \(n \gg p\) and is the only thing that works when \(p > n\)
Truth B: additive smooth
The additive fit was built for this truth. A forest can bend, with rectangles; a plane cannot bend at all
Truth C: thresholds with an interaction
A checkerboard: each input alone says nothing, so every additive model is blind to it. Two questions in a row see all of it
Four dimensions, not one
Axis
The question about \(f^\star\)
Who bets which way
wide or tall
how \(p\) compares with \(n\)
least squares and the additive fit need \(n \gg\) columns; ridge and the lasso fit at any \(p\)
sparse or dense
how many inputs matter
the lasso bets few; least squares and ridge spread the fit over all; a forest tolerates idle inputs, \(k\)-NN does not
smooth or discontinuous
how \(f^\star\) moves with each input
splines and \(k\)-NN bet smooth; trees bet on steps
additive or interactive
whether inputs act separately
the additive fit bets separately; trees, forests and \(k\)-NN allow interactions
Four separate questions, not one scale from simple to complex. A family is a bet on the answers; a selection set says which bet paid on these data
Week 1’s curse: \(k\)-NN lost on every truth, with ten inputs and 300 points. Week 4’s no free lunch: no family won every truth, and none can.
What the training error hides
On every leaderboard the training MSE sat below the test MSE, the forest’s by a factor of four. Luck, or a law?
Training data \((x_i, y_i)\), \(i = 1, \ldots, n\), independent draws from the population; a class \(\mathcal F\) (the planes; the additive fits with a given \(J\)); \(f_{\mathcal F}\) the best in \(\mathcal F\), the minimizer of \(R\) over it; \(\hat f\) the empirical risk minimizer, the minimizer of \(\hat R_n\) over \(\mathcal F\) (week 4)
Three numbers: \(\mathbb E[\hat R_n(\hat f)]\), \(R(f_{\mathcal F})\), \(\mathbb E[R(\hat f)]\). Put them in order, and write down what you used to do it.
Optimism theorem
A function fixed before the data has an unbiased training error. \(\hat f\) beats every such function in \(\mathcal F\) on the training data (it was chosen to) and cannot beat \(f_{\mathcal F}\) on the population
Training data i.i.d. from the population; \(\hat f\) the empirical risk minimizer over \(\mathcal F\); \(f_{\mathcal F}\) the best in \(\mathcal F\) (risks finite, minimizers exist). Then \[
\mathbb E\big[\hat R_n(\hat f)\big] \;\le\; R(f_{\mathcal F}) \;\le\; \mathbb E\big[R(\hat f)\big]
\]
Training error flatters the model: on average it sits below the best risk the class can reach, and the fitted model’s risk sits above it. The test set estimates the right end of this line.
Proof: exercise 1, three lines. Covers the minimizer over a fixed class (least squares; splines and penalized fits at a fixed knob); not a tree, a forest, \(k\)-NN, or a knob chosen on the data
Before the seminar
This week: exercises 1 and 2 at the top of the notebook. The optimism theorem’s proof, three lines with numbered hints; and ridge’s normal equations, with why \(\mathbf X^T\mathbf X + \lambda I\) is invertible
In the seminar: the Spotify task with every family. Each knob chosen on the selection tracks, then the vault opened once for the leaderboard
Conclusion: four bets, one protocol
Each family is a bet on the shape of \(f^\star\)
A plane, a sparse plane, a sum of curves, a set of rectangles, an average of many. Four questions about the world decide which bet pays
The protocol decides what you may believe
Fit on the training rows, choose on the selection rows, open the test set once
The training error flatters, by a theorem
On average, for every class and every minimizer. How much flexibility to allow, and how to choose it: week 7
