ST310 course page

A core set of commonly used machine learning tools, practiced in R, with attention to their mathematical foundations, their statistical limitations, and what they do and do not tell you.

Readings and lectures are complementary so do both, and bring a laptop with R and RStudio to the seminar (see about for setup). The seminar notebook for each week is linked below; the .qmd link downloads the source file to open in RStudio, the other link is the same notebook rendered so you can read it in a browser. Click a week’s heading, or its entry in the contents, to open it.

Week 1: prediction competitions and nearest neighbors

What machine learning is and where it came from: two lineages, one field, and the prediction competition (the common task framework) as its working definition, with the Netflix prize as the cautionary tale. A first prediction task, the rule that a prediction may use only what is available at prediction time, a held-out set as the score, and \(k\)-nearest neighbors as the first method. The bias–variance decomposition derived for \(k\)-NN at the board, and why nearest neighbors stop working in many dimensions.

Reading

  • ISLR Chapter 1 (Introduction), pp. 1–14. A quick read; don’t skip the section on notation and simple matrix algebra.
  • ISLR Chapter 2 (Statistical Learning), sections 2.1 and 2.2, pp. 15–42. If R is new to you, also do the lab in section 2.3, pp. 42–52.
  • PPA Chapter 1 (Introduction), pp. 1–10.

If you finish these quickly you may want to start Chapter 3, because it’s a long one.

Lecture: slides

Seminar: notebook (.qmd) · notebook with solutions

Week 2: linear regression

Regression as the first fitted model rather than a review: least squares and the hat matrix, and the conditional mean as the target that every regression method is aiming at (the optimal model theorem). Multiple regression and what a coefficient holds fixed: the partialling-out theorem, proved in the lecture and reused in weeks 8 and 10, and omitted-variable bias, what the same algebra says when a relevant column is left out.

Reading

  • ISLR Chapter 3 (Linear Regression), sections 3.1–3.3 and 3.5, pp. 61–110. Mostly review from your regression course; read 3.2.1 (estimating the coefficients in multiple regression) and 3.3.3 (potential problems) slowly. The lab in 3.6 goes with the seminar.
  • PPA Chapter 2 (Fundamentals of prediction), the sections “Modeling knowledge” and “Prediction via optimization”, pp. 13–21.

Optional, deeper: ESL section 2.4 (the optimal model theorem) and section 3.2, pp. 44–57 (3.2.3 is the partialling-out theorem).

Lecture: slides

Seminar: notebook (.qmd) · notebook with solutions. Before the seminar, do exercises 1 to 3 at the top of the notebook (the two proofs the lecture left to you, and the partialling-out proof again with the slides closed); the seminar’s problems start from them. Bring a dataset of your own if you have one: a numeric outcome and a few predictors, one row per unit.

Week 3: classification, and causality (part 1)

Classification: logistic regression, a threshold and the two kinds of error on either side of it, and the metrics that follow; the optimal threshold theorem says where the threshold goes, and the costs choose it. Causal block 1: prediction against intervention, a world in which the causation runs the other way from the correlation, and why a model that predicts well can still give the wrong answer to a what-if question. The adjustment theorem says when averaging over a measured variable recovers the effect of an intervention, and the inverse propensity weighting theorem, proved in the lecture, gets the same effect by weighting with the probability of treatment, a class probability that logistic regression estimates.

Reading

  • ISLR Chapter 4 (Classification), sections 4.1–4.3, pp. 130–141. Section 4.4 (LDA, QDA, naive Bayes) is not part of this course; skim 4.5, pp. 158–164, for the comparison with \(k\)-nearest neighbors.
  • PPA Chapter 9 (Causality), from “The limitations of observation” through “Confounding”, pp. 179–192.

Optional: PPA Chapter 2 (Fundamentals of prediction), “Types of errors and successes” and “The Neyman–Pearson lemma”, pp. 21–26, and “Decisions that discriminate”, pp. 26–31.

Lecture: slides

Seminar: notebook (.qmd) · notebook with solutions. Before the seminar, do exercises 1 to 4 at the top of the notebook (the threshold proof, the confounding gap, the adjustment proof, and the lecture’s inverse propensity weighting proof again with the slides closed); the seminar’s problems start from them. Data: the Palmer penguins (data note); the (fictional) attrition data as an alternative.

Week 4: optimization and model fitting

How a model actually gets fitted. Logistic regression first: maximum likelihood, the score equations read as week 2’s orthogonality, and why there is no closed form. Then the method that fits almost everything else: gradient descent, step sizes and the descent lemma, and its stochastic version, whose random steps are unbiased for the full gradient. Empirical risk minimization as the common form of every method so far, the bias–variance decomposition in that form, why no method is best on every problem, and early stopping as a first kind of regularization.

Reading

  • PPA Chapter 5 (Optimization), from “Optimization basics” through “The SGD quick start guide”, pp. 70–86, stopping at “Analysis of the stochastic gradient method”: gradient descent, the quadratic case, and stochastic gradient descent in practice.

Optional: PPA “Analysis of the stochastic gradient method”, pp. 86–88 (a convergence bound), and “Regularization” with “Implicit regularization by optimization”, pp. 92–94 (why gradient descent from zero lands on the minimum-norm solution, which the seminar meets); ESL section 4.4.1 for Newton-IRLS in full; the Wikipedia pages on Newton’s method and gradient descent for pictures and animations.

Lecture: slides

Seminar: notebook (.qmd). Before the seminar, do exercises 1 to 4 at the top of the notebook (the lecture’s descent lemma again with the slides closed, the ERM form of the bias–variance decomposition, the logistic Hessian and the Newton-Raphson update, and no free lunch on a finite domain); the seminar’s problems start from them. You write gradient descent once, as a function, and use it for least squares, logistic regression and random batches. Data: gapminder and the Palmer penguins (data note), then simulated data.

Week 5: what structures can we model?

The methods, introduced together and compared under one protocol. Ridge and lasso: what a penalty changes, what \(\lambda\) controls, fitting when \(p > n\). Additive models: a spline as regression on basis functions, one curve per input, the basis size as the knob. Trees and forests: how a small tree predicts, why it is unstable, what averaging does. Then one simulated dataset for each kind of truth (sparse linear, additive smooth, thresholds with an interaction), every family fitted to each, and the four dimensions on which a method bets: wide or tall, sparse or dense, smooth or discontinuous, additive or interactive. Training error flatters every one of them (the optimism theorem, stated; its proof is yours before the seminar). The three-way split, training, selection and a test set opened once, is used from the first slide.

Reading

  • ISLR Chapter 6 (Linear Model Selection and Regularization), sections 6.2.1–6.2.2, pp. 237–250: ridge regression and the lasso, and what each penalty does to the coefficients. Stop before 6.2.3; choosing \(\lambda\) by cross-validation is week 7.
  • ISLR Chapter 7 (Moving Beyond Linearity), sections 7.3–7.4, pp. 294–301 (basis functions and regression splines), and section 7.7, pp. 306–310 (generalized additive models; 7.7.1 fits one by least squares on spline basis functions, as the lecture does, and the GAM part of the lab in 7.8 has the lm and ns() code). Section 7.5, smoothing splines, is week 7. Skim 7.6, pp. 304–306: local regression is named in the lecture and not examined.
  • ISLR Chapter 8 (Tree-Based Methods), section 8.1.1, pp. 328–335 (regression trees), and sections 8.2.1–8.2.2, pp. 340–345 (bagging and random forests). Skim 8.1.2–8.1.4, pp. 335–340. Section 8.2.3, boosting, is week 8.

Optional: PPA Chapter 6 (Generalization), “Generalization gap”, pp. 100–101, the setting of the optimism theorem.

Lecture and seminar: released here at the start of the week. All three families and the baselines on one dataset, one leaderboard, the three-way split enforced. Data: Spotify tracks (what makes a song popular; a data note comes with the notebook), the penguins, the movies, simulated data.

Week 6: reading week

No lecture or seminar. Start week 7’s reading: ISLR section 5.1, pp. 197–209 (cross-validation), which week 7 opens with.

Week 7: how much flexibility should we allow?

Every method of week 5 had a knob. This week chooses it, and asks what the choice costs. \(K\)-fold cross-validation and the curve it draws against \(\lambda\), \(k\) and tree depth. Why a flexible method needs data: from week 1’s decomposition, the error of \(k\)-nearest neighbors in \(d\) dimensions, the rate \(n^{-2/(d+2)}\), and what an assumption buys (an additive truth, a sparse one). Ridge and lasso side by side in the simplest design: shrinkage against soft thresholding, prediction against selection; the smoothing penalty on a spline as ridge on its coefficients. What changes when \(p > n\): the minimum-norm interpolant, why one interpolant rather than another, and the double-descent curve that the bias–variance picture does not predict. The standard error of a held-out score.

Reading [PROVISIONAL]

  • ISLR Chapter 5 (Resampling Methods), section 5.1, pp. 197–209 (the validation set, leave-one-out and \(K\)-fold cross-validation, and the bias–variance trade-off in choosing \(K\)). Section 5.2, the bootstrap, is not part of this course.
  • ISLR Chapter 6, section 6.2.3, pp. 250–252 (choosing the tuning parameter by cross-validation), and section 6.4, from p. 261 (what changes when \(p > n\)). Section 6.3, which sits between them, is not part of this course.
  • ISLR Chapter 7, section 7.5, pp. 301–304 (smoothing splines: the roughness penalty, and \(\lambda\) chosen from the data).
  • PPA Chapter 6 (Generalization), “Generalization gap” and “Overparameterization: empirical phenomena”, pp. 100–106 (what over-parameterized models do in practice; double descent).

Deeper, optional: Belkin, Hsu, Ma and Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off”, PNAS 116(32), 2019 (the double-descent curve); ESL section 3.4, pp. 61–73 (ridge and the lasso in general designs); Györfi, Kohler, Krzyżak and Walk, A Distribution-Free Theory of Nonparametric Regression (2002), Chapter 6, for the rigorous form of the nearest-neighbor rate the lecture derives from a stated lemma.

Lecture and seminar: released here at the start of the week. A cross-validation pipeline on week 5’s data; learning curves for \(k\)-nearest neighbors in one, two, five and ten dimensions read against the rate; then, after cross-validation has worked for you, two ways it goes wrong: standardizing before the split against inside the folds, and a random split against a split in time. Data: week 5’s datasets, ISLR’s Hitters (baseball salaries, 1987), simulated data.

Week 8: how are flexible models constructed and improved?

How the fits of week 5 are built, and what makes them better. An additive model by backfitting: each component fitted to what the others leave, which is week 2’s partialling-out theorem one input at a time; and an interaction that no additive fit can see. A tree: one split found by hand, then greedy recursion, depth and pruning, and why a tree is unstable. Bagging: the variance of an average of correlated predictions, proved, and why it stops short of \(1/B\); random forests as the way to lower the correlation; the out-of-bag error as a held-out score that comes free. Boosting as gradient descent one small tree at a time (week 4’s method on week 2’s residuals), its three knobs, and why a sum of trees is additive in trees and not in the predictors. Two slides on kernels: a locally weighted average with a bandwidth, and ridge on features you never write down.

Reading [PROVISIONAL]

  • ESL Chapter 9, the opening of section 9.1, pp. 295–298 (additive models and the backfitting algorithm, Algorithm 9.1). ISLR section 7.7, pp. 306–310, read in week 5, describes the same model.
  • ISLR Chapter 8 (Tree-Based Methods), section 8.1, pp. 327–340, in full this time (recursive binary splitting and pruning; classification trees; trees against linear models), and sections 8.2.1–8.2.3, pp. 340–348 (bagging, random forests, boosting). Section 8.2.4, Bayesian additive regression trees, is not part of this course. The lab in section 8.3 goes with the seminar.
  • ESL Chapter 15, section 15.4, pp. 597–602 (why random forests work: the variance of a correlated average, which is the lecture’s proof).

Deeper, optional: ESL sections 10.9–10.10, pp. 353–361 (boosting trees as gradient descent); ESL Chapter 6, section 6.1 (one-dimensional kernel smoothers), for the kernel slides; Breiman, “Random forests”, Machine Learning 45, 2001; Friedman, “Greedy function approximation: a gradient boosting machine”, Annals of Statistics 29(5), 2001.

Lecture and seminar: released here at the start of the week. One split by hand, then a recursive tree; backfitting in ten lines against mgcv; bagging by hand with its variance measured against the theorem and an out-of-bag error; one boosting update by hand, then xgboost. Data: the penguins, the movies, gapminder, simulated data.

Week 9: what can we learn from a fitted prediction?

What a forecast is for. The decision principle: choose the action that minimizes expected loss given the inputs. The mean, the median and week 3’s threshold were three cases of it; the conditional quantile is the fourth, and the lecture proves it. Uncertainty about a future outcome, which is not uncertainty about a score: a prediction interval from estimated quantiles, by linear quantile regression and then by a method of week 5, with a Gaussian mean-and-variance model as the comparison; coverage checked on fresh data across the range of the inputs, as a diagnostic and not a certificate; why no distribution-free method can guarantee coverage at every \(x\), and what conformal prediction guarantees instead; a calibration check for a classifier. Then interpretation of the models you can now fit: a coefficient under correlated inputs, a GAM’s components, permutation importance and where it fails, partial dependence and ICE as questions put to the model (SHAP named in one slide), and why a model’s response curve is not an effect.

Reading [PROVISIONAL]

  • Koenker and Hallock, “Quantile regression”, Journal of Economic Perspectives 15(4), 2001, pp. 143–156: the first half, for what a conditional quantile is and the loss that finds it.
  • ISLR Chapter 3, section 3.2.2, the passage on confidence and prediction intervals (p. 82): the distinction the lecture starts from.
  • IML, the chapters “Partial dependence plot (PDP)”, “Individual conditional expectation (ICE)” and “Permutation feature importance”: the definitions and the pictures. The chapter “SHAP” is for recognition only.

Deeper, optional: Angelopoulos and Bates, “Conformal prediction: a gentle introduction”, Foundations and Trends in Machine Learning 16(4), 2023, sections 1–2; Barber, Candès, Ramdas and Tibshirani, “The limits of distribution-free conditional predictive inference”, Information and Inference 10(2), 2021 (the impossibility result the lecture states); Zhao and Hastie, “Causal interpretations of black-box models”, Journal of Business and Economic Statistics 39(1), 2021.

Lecture and seminar: released here at the start of the week. A simulation whose noise grows with one input: quantile fits, interval widths and coverage by range on fresh data; the calibration of week 3’s logistic fit; partial dependence, ICE and permutation importance by hand on week 8’s models. Data: supplied simulations, the penguins, the movies.

Week 10: how do prediction methods support other statistical tasks?

A fitted model put to a use other than predicting. A partial dependence curve as an estimate of week 3’s interventional mean, exact at the population level when the adjustment condition holds and the model is right, and an estimate otherwise. One effect estimated with machine learning doing the nuisance work: residual on residual, the two lines that show why residualizing the treatment makes the nuisance error second order (the lecture’s derivation), what cross-fitting does, a standard error that can be trusted, and honest trees as the same split-sample idea; nothing in any of it makes the effect causal without week 3’s condition. Then the data move: covariate shift and reweighting by a ratio of probabilities (week 3’s inverse propensity weighting again), label shift and which factorization of the joint distribution you believe, and adaptive drift when the model has been tuned to the score it is judged by.

Reading [PROVISIONAL]

  • PPA Chapter 10 (Causal inference in practice), pp. 204–217: design and inference, adjustment and controls, reductions to model fitting (double machine learning in a page), quasi-experiments, and the limitations.
  • PPA Chapter 8 (Datasets), “Longevity of benchmarks”, pp. 158–167 (what happens when the same held-out set is used many times, and the CIFAR-10 and ImageNet replications).
  • Zhao and Hastie, “Causal interpretations of black-box models”, Journal of Business and Economic Statistics 39(1), 2021 (the back-door statement behind the lecture’s corollary).

Deeper, optional: Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey and Robins, “Double/debiased machine learning for treatment and structural parameters”, Econometrics Journal 21(1), 2018, sections 1–2; Shimodaira, “Improving predictive inference under covariate shift by weighting the log-likelihood function”, Journal of Statistical Planning and Inference 90(2), 2000; Lipton, Wang and Smola, “Detecting and correcting for label shift with black box predictors”, ICML 2018 (the setting; the correction itself is not taught); PPA Chapter 13 (Epilogue), pp. 268–271.

Lecture and seminar: released here at the start of the week. One effect estimated by residual on residual with cross-fitting, two wrong versions and a hidden confounder; a partial dependence curve against the true interventional mean on a simulation; reweighting under covariate shift and under label shift, and a shift diagnosed on gapminder 1952 against 2007. Data: supplied simulations, gapminder.

Week 11: what changes when the representation is learned?

Every method in this course was a predictor on features someone chose. What changes when the features are built by hand (week 8’s bases and kernels), learned together with the task (a network with one hidden layer as a basis the data chooses, fitted by week 4’s stochastic gradient descent; everything beyond that is ST311’s), or transferred from a model trained on something else (embeddings, with week 1’s nearest neighbors and week 5’s ridge run on them). Then the course’s frame read again with everything since: what the held-out leaderboard delivered, and where it stops, each limit being a result from the term. The theorems in one view, as a reference sheet.

Reading [PROVISIONAL]

  • PPA Chapter 4 (Representations and features), pp. 49–67: measurement, and “Nonlinear predictors” (polynomials, kernels and random features, and a neural network as a learned representation).
  • ISLR Chapter 10, sections 10.1–10.2, pp. 404–411 (single- and multilayer neural networks), for the one network the lecture builds. The rest of Chapter 10 is ST311’s.

Delivered in lecture, for reference rather than preparation:

  • Rodu and Baiocchi, “When black box algorithms are (not) appropriate”, Observational Studies 9(2), 2023 (arXiv 2001.07648): the common task framework and what it does and does not establish.
  • Wang, Kapoor, Barocas and Narayanan, “Against predictive optimization”, ACM Journal on Responsible Computing 1(1), 2024.

Optional: ISLR Chapter 12, section 12.2, from p. 498 (principal components, the simplest learned representation); Belkin, Hsu, Ma and Mandal, PNAS 2019, if not read in week 7.

Lecture and seminar: released here at the start of the week, together with the theorems in one view (statements only, with the week each was proved or derived; bring it to the seminar and try to reproduce each statement from memory first, checking second). Supplied text embeddings in R: \(k\)-nearest neighbors, ridge and logistic regression fitted on them, what the representation preserves and where evaluation fails; then one unfamiliar problem taken from target to evaluation. Data: a supplied embedding file, and your project data.