Part 1: classification, logistic regression, and asymmetric costs
Where week 2 left the target
Week 2, the optimal model theorem: under squared loss the best predictor of \(Y\) from \(X\) is the conditional mean, \(f^\star(x) = \mathbb E[Y \mid X = x]\), and any other \(f\) pays \(\mathbb E[(f^\star(X) - f(X))^2]\) on top of it.
Today the outcome is a label: \(Y \in \{0, 1\}\).
Write down what \(\mathbb E[Y \mid X = x]\) is when \(Y\) takes only the values \(0\) and \(1\). Then write one thing a straight line \(x^T\beta\) gets wrong about it.
The target is a probability
For a \(0/1\) outcome the conditional mean is a probability: \[
p(x) = P(Y = 1 \mid X = x) = \mathbb E[Y \mid X = x]
\]
So the optimal model theorem’s target is the class probability\(p(x)\), and a fitted model gives an estimate \(\hat p(x)\)
A line \(x^T\beta\) does not stay inside \([0, 1]\): that is the first thing logistic regression fixes
Notation for today: \(p(x)\) the class probability, \(\hat p(x)\) its estimate, and a threshold\(c\): predict \(1\) when \(\hat p(x) > c\)
Classification
Supervised learning with categorical/qualitative outcomes
(in contrast to regression, with numeric outcomes)
Often called “labels”, \(K\) = number of unique classes
Binary: positive/negative or 0/1 or yes/no or success/fail etc
Label names not mathematically important - e.g. use \(1, ..., K\)
Limitations: labels already defined (not learned from data– that would be unsupervised learning), \(K\) is fixed
Plots: often use color/point shape for categorical variables
Because \(y\) is 0 or 1, residual plots will show patterns, not as easy to interpret geometrically
Challenges
Separable case (guaranteed if \(p \ge n\) and the rows of \(\mathbf X\) are linearly independent)
If classes can be perfectly separated, there is no finite MLE: the likelihood keeps increasing as \(\hat \beta\) runs off to infinity along a separating direction
Awkwardly, classification is too easy(!?) for this probabilistic approach
Curse of dimensionality
With Gaussian inputs: biased MLE and wrong variance/asymp. dist. if \(n/p \to \text{const}\), even if \(> 1\)
Move \(c\) and the two kinds of mistake trade against each other. The two numbers that move: the true positive rate (TPR, sensitivity, recall) \(= \text{TP}/(\text{TP} + \text{FN})\) and the false positive rate (FPR, one minus specificity) \(= \text{FP}/(\text{FP} + \text{TN})\)
Accuracy \(= (\text{TP} + \text{TN})/n\) hides the split. With 95% zeros, predicting \(0\) for everyone already scores 95%
ROC: the curve invented for radar
The receiver operating characteristic: one point per threshold \(c\), from \(c = 1\) (predict \(0\) for everyone, bottom left) to \(c = 0\) (top right)
Lower \(c\): catch more of the 1s, raise more false alarms
AUC, the area under it: the probability that a random 1 scores above a random 0 (a tie counts half), a ranking quality that never asks where \(c\) is
Named for radar receivers in the 1940s. It does not answer the operator’s question: which point?
What does a mistake cost?
Predicting \(1\) when the truth is \(0\) costs \(c_{FP}\); predicting \(0\) when the truth is \(1\) costs \(c_{FN}\); a correct prediction costs nothing
A miss costs ten times a false alarm and the model says \(\hat p = 0.3\). Predict \(1\) or \(0\)?
Optimal threshold theorem
Optimal threshold theorem. Let \(p(x) = P(Y = 1 \mid X = x)\), and let a false positive cost \(c_{FP}\) and a false negative \(c_{FN}\), both \(\ge 0\) and not both \(0\). The rule that minimizes expected cost at \(x\) predicts \(1\) exactly when \[
p(x) > c^\star = \frac{c_{FP}}{c_{FP} + c_{FN}},
\] and at \(p(x) = c^\star\) either prediction is optimal. Equal costs give \(c^\star = \tfrac12\): predict the more probable class, the Bayes classifier.
The model gives \(p\). The costs give \(c^\star\), and somebody chose the costs. With \(\hat p\) in place of \(p\), the rule is only as good as the estimate.
Proof: exercise 1 in the notebook, before the seminar.
Classification summary
Numeric prediction \(\to\) classification
\[\hat y = \mathbb I(\hat p > c) = \begin{cases} 0 & \text{ if } \hat p \leq c \\ 1 & \text{ if } \hat p > c \end{cases}\]
Log-odds function is monotonic, so (hyperplanes)
\[\hat p > c \leftrightarrow x^T \hat\beta > c' = \log\frac{c}{1 - c}, \quad 0 < c < 1\]
More classes: transform to binary, predict using largest \(\hat p_k\)
Non-linear boundaries: transformation of predictors, or use methods other than GLMs (we’ll learn more soon)
Some classification methods output categorical classes, not probabilities (or other numeric scores)
A model predicts life expectancy from GDP per capita well on held-out countries. A minister raises GDP per capita by ten percent. What does the model say happens to life expectancy? Do you believe it?
History: from the stars to “Poor Law Statistics”
Almost a century after Gauss
Scientists correlating/regressing anything
Problem: what does it mean?
e.g. Francis Galton correlated numeric traits between generations of organisms…
Multiple regression developed in 1897, applied to poverty, welfare and age in 1899
Association between poverty and welfare while “controlling for” age
History: Yule, 1897 and 1899
Instead of speaking of “causal relation,” … we will use the terms “correlation,” …
Variables, roughly:
\(Y =\) prevalence of poverty
\(X_1 =\) generosity of welfare policy
\(X_2 =\) age
Positive correlations:
\(\text{cor}(Y, X_1) > 0\)
\(\text{cor}(X_2, X_1) > 0\)
Do more people enter/stay in poverty if welfare is more generous?
Or is this association “due to” age?
History: Yule, in 1897
The investigation of causal relations between economic phenomena presents many problems of peculiar difficulty, and offers many opportunities for fallacious conclusions.
Since the statistician can seldom or never make experiments for himself, he has to accept the data of daily experience, and discuss as best he can the relations of a whole group of changes; he cannot, like the physicist, narrow down the issue to the effect of one variation at a time. The problems of statistics are in this sense far more complex than the problems of physics.
[We] cannot […] narrow down the issue to the effect of one variation at a time
but… isn’t this how almost everyone interprets regression coefficients?…
🤔 🤨
(yes! and they are wrong!!!!)
Misinterpreting regression coefficients
People want these things to be true:
“The linear model and our estimates are both good”
“We can interpret \(\beta_j\) as a causal parameter,” i.e. intervening to increase \(x_j\) by 1 unit would result in conditional average of \(y\) changing by \(\beta_j\) units
This is not guaranteed to work, and usually it’s wrong
Ceteris paribus
Many textbooks tell us something like:
“The coefficient \(\hat \beta_j\) estimates the relationship between the (conditional mean of the) outcome variable and \(x_j\)while holding all other predictors constant”
i.e. “ceteris paribus” or “other things equal” (unchanged)
Fundamental problem of interpreting regression coefficients:
“holding all other predictors constant” is (almost) never applicable in the real world, i.e. ceteris is (almost) never paribus
Reasons we’ll highlight: causality (today) and nonlinearity (the additive-models week)
Interpreting causality
Back to Yule. What does \(\hat \beta_\text{welfare}\) mean? (Simulated data below, not Yule’s.)
… the B.B.C. gave me the opportunity of putting forward examples of the two classes of alternative theories which any statistical association, observed without the predictions of a definite experiment, allows–namely, (1) that the supposed effect is really the cause, or in this case that incipient cancer, or a pre-cancerous condition with chronic inflammation, is a factor in inducing the smoking of cigarettes, or (2) that cigarette smoking and lung cancer, though not mutually causative, are both influenced by a common cause, in this case the individual genotype …
Graphical notation for causality
Variables: vertices (or nodes)
Relationships: directed edges (arrows)
Shaded node / dashed edges: unobserved variable
Two of Fisher’s theories
Cancer causes smoking?
Genotype is a common cause?
Yule: speak of correlation, not causal relations
Fisher: association is not causation
So what can we say about causation?
Idea: adjusting for confounders
Confounders: other variables that obscure the (causal) relationship from \(X\) to \(Y\), e.g.
\(Y\): health outcome
\(X\): treatment dose
\(Z\): disease severity
Without considering \(Z\), it might seem like larger doses of \(X\) correlate with worse health outcomes
Solution: add more variables to the model
Including (measured) confounders in the regression model may give us a more accurate estimate
Confounder adjustment is why some people think multiple regression is One Weird Trick that lets us make causal conclusions
It’s not that simple (as Yule already knew in 1897)
Use a simple argument with two DAGs to explain why this might not work.
Simple models for causality
Interventions are a new type of operation we may use in probability models. Consider an intervention that changes some target variable \(T\)
Forget about the arrows pointing into \(T\) (intervention makes them irrelevant)
Change \(T\), e.g. setting it to some arbitrary new value \(T = t\)
This change propagates along directed paths out of \(T\) to all descendant variables of \(T\) in the graph, which can change their values
Writing an intervention down
Behind a DAG is a structural model: one equation per variable, from its parents and its own noise. With \(T\) the target, \(Y\) the outcome and \(Z\) the other variables that matter: \[
T = h(Z, \varepsilon_T), \qquad Y = g(T, Z, \varepsilon_Y), \qquad Z,\ \varepsilon_T,\ \varepsilon_Y \text{ independent}
\]
Intervene on \(T\), written \(do(T = t)\): delete \(T\)’s equation and set \(T = t\). Every other equation stays, and the noises keep their distribution. \(Z\) keeps its distribution
Recovering the effect from what we observe
Under \(do(T = t)\), \(Y = g(t, Z, \varepsilon_Y)\) with \((Z, \varepsilon_Y)\) distributed as before, so \(\mathbb E[Y \mid do(T = t)] = \mathbb E\, g(t, Z, \varepsilon_Y)\). Compare \(\mathbb E[Y \mid T = t]\), the average over the units that happened to have \(T = t\)
From observed \((T, Z, Y)\): write down what you would compute to get \(\mathbb E[Y \mid do(T = t)]\), and what has to be true of \(Z\) for it to work.
The confounding gap
The same DAG, made linear: \(T = \alpha Z + \varepsilon_T\), \(\;Y = \tau T + \beta Z + \varepsilon_Y\), with \(Z\), \(\varepsilon_T\), \(\varepsilon_Y\) independent normal and the noises mean \(0\)\[
\mathbb E[Y \mid do(T = 1)] - \mathbb E[Y \mid do(T = 0)] = \tau
\]\[
\mathbb E[Y \mid T = 1] - \mathbb E[Y \mid T = 0] = \tau + \beta\,\big(\mathbb E[Z \mid T = 1] - \mathbb E[Z \mid T = 0]\big)
\]
Week 2, omitted-variable bias, inputs fixed: regressing on \(\mathbf x_1\) alone gives \(\mathbb E\hat\beta_{\text{short}} = \beta_1 + \beta_2\hat\delta\), with \(\hat\delta\) the slope of \(\mathbf x_2\) on \(\mathbf x_1\).
The naive difference is the effect plus what \(Z\) contributes through selection: omitted-variable bias, read causally. Derivation: exercise 2.
Adjustment theorem
Adjustment theorem. Structural model with independent noises, \(Z\) observed, \(T\) taking finitely many values: \(T = h(Z, \varepsilon_T)\), \(Y = g(T, Z, \varepsilon_Y)\). If \(P(T = t \mid Z) > 0\) with probability one, \[
\mathbb E[Y \mid do(T = t)] = \mathbb E_Z\Big[\, \mathbb E[Y \mid T = t, Z]\, \Big]
\]
Fix \(T\) at \(t\), average the conditional mean over the population’s \(Z\). In this model the condition, in words: \(Z\) holds every common cause of \(T\) and \(Y\) and no consequence of \(T\).
Proof: exercise 3 in the notebook, before the seminar.
The propensity score
From here on \(T \in \{0, 1\}\): treated or not. The propensity score is the class probability from the first half of the lecture, with \(T\) as the label and \(Z\) as the input: \[
\pi(z) = P(T = 1 \mid Z = z)
\]
Logistic regression of \(T\) on \(Z\) estimates it: glm(T ~ Z, family = binomial())
Suppose you can estimate \(\pi(z)\) well, but you have no regression model for \(\mathbb E[Y \mid T, Z]\). How would you use \(\hat\pi\) to estimate \(\mathbb E[Y \mid do(T = 1)]\)?
Hint: make the data look as if \(T\) had been assigned by a coin flip, not by the confounders
Weight each treated unit by \(1/\pi(Z)\): it stands in for \(1/\pi(Z)\) units with the same \(Z\). The adjustment theorem needs a regression model for \(Y\). This one needs a regression model for \(T\), and logistic regression gives one.
Inverse propensity weighting theorem, proof (1 of 2)
Start from what we want, and change one term at a time
For \(T = 0\): the same steps with \(1 - T\) and \(1 - \pi(Z)\)
Where inverse propensity weighting fails
\(\pi(Z)\) near \(0\) or \(1\): a few units get huge weights, and the estimate is unstable. The condition \(0 < \pi(Z) < 1\) is about the population; in a sample it shows up as extreme weights
Causal machine learning replaces logistic regression with more flexible models. If these models achieve high accuracy, they can output probabilities close to \(0\) or \(1\), and the weights explode: a better classifier can make a worse weighting estimate
A common cause left out of \(Z\): \(\varepsilon_T\) and \(\varepsilon_Y\) are dependent, and step (c) of the proof breaks
\(\hat\pi\) in place of \(\pi\): the weights are only as good as the logistic regression that produced them
Experiments
Actually do interventions while collecting data
Observational studies
Try to infer causal relationships without interventions, by using dark arts more/specialized assumptions/methods that require careful interpretation
(increasingly common due to superabundance of data)
Scientific progress: be wrong in more interesting/specific ways
Before the seminar
This week: exercises 1 to 4 at the top of the notebook. The proof of the optimal threshold theorem; the confounding gap; the proof of the adjustment theorem; and today’s inverse propensity weighting proof again, with the slides closed
Conclusions
Wisdom from one of the great early statistical explorers, Udny Yule:
Measurement does not necessarily mean progress. Failing the possibility of measuring that which you desire, the lust for measurement may, for example, merely result in your measuring something else - and perhaps forgetting the difference - or in your ignoring some things because they cannot be measured.
Remember: regression coefficients do not necessarily mean causal relationships. Causal inference: an exciting interdisciplinary field, practically important, with connections to ML
“Data scientists have hitherto only predicted the world in various ways; the point is to change it” - Joshua Loftus
