15.097 Lecture 9: Logistic Regression

Logistic Regression MIT 15.097 Course Notes Cynthia Rudin Thanks to Ashia Wilson Credit: J.S. Cramer’s “The Origin of Logistic Regression” Origins: 19th Century. • Studying growth of populations and the course of chemical reactions using d W (t) = βW (t) dt ⇒ W (t) = Aeβt which is a good model for unopposed growth, like the US population’s growth at the time. • Adolphe Quetelet (1796 - 1874), Belgian astronomer turned statistician, knew it produced impossible values and asked his pupil Pierre-François Verhulst (1804-1849) to help him work on a more realistic model. They chose d W (t) = βW (t) − Φ(W (t)) dt to resist further growth, and with the choice of Φ to be a quadratic function, they got: d W (t) = βW (t)(Ω − W (t)), dt where Ω is the saturation limit of W. Writing P (t) = WΩ(t) as the proportion of saturation limit: d P (t) = βP (t)(1 − P (t)) dt e(α+βt) P (t) = 1 + e(α+βt) where α is the constant from the integration. Verhulst’s “logistic” function has values between 0 and 1. Draw the logistic function 1 He published in 3 papers between 1838 and 1847. The first paper demonstrated that the curve agrees very well with the actual course of the population in France, Belgium, Essex, and Russia for periods up to 1833. He did not say how he fitted the curves. In the second paper he tried to fit the logistic to 3 points using 20-30 years of data (which in general is a not a great way to get a good model). His estimates of the limiting population Ω of 6.6 million for Belgium and 40 million for France were a little off - these populations are now 11 million for Belgium and 65 million for France. In another paper, the estimate was corrected, and they estimated 9.5 million for Belgium (pretty close!). Verhulst died young in poor heath, and his work was forgotten about, but reinvented in 1920 by Raymond Pearl and Lowell Reed who were studying population growth of the US. They also tried to fit the logistic function to population growth, and estimated Ω for the US to be 197 million (the current population is 312 million). Actually, Pearl and collaborators spent 20 years applying the logistic growth curve to almost any living population (fruit flies, humans in North Africa, cantaloupes). Verhulst’s work was rediscovered just after Pearl and Reed’s first paper in 1920, but they didn’t acknowledge him in their second paper, and only in a footnote in a third paper (by Pearl) in 1922. They cited him in 1923, but didn’t use his terminology and called his papers “long since forgotten.” The name logistic was revived by Yule, in a presidential address to the Royal Statistical Society in 1925. There was a lot of debate over whether the logistic function could replace the cdf of the normal distribution. The person who really showed this could happen was Joseph Berkson (1899-1982), who was the chief statistician at the Mayo Clinic, and who was a collaborator of Reed’s. But because of his personality and attacks on the method of maximum likelihood, controversy ensued... it took until around the 1960’s to resolve it! Let’s start with a probabilistic model. Assume x ∈ Rn , and that x is deterministic (chosen, not random): Y ∼ Bernoulli(P (Y = 1|x)). 2 Here Y takes either 0 or 1, but we need a ±1 for classification. I’ll write 0̃ for now to stand for either 0 or −1. The model is:   P (Y = 1|x, λ) ln = λT x P (Y = 0̃|x, λ) | {z } “odds ratio” Why does this model make any sense at all? We want to make a linear combination of feature values, like in regular regression, which explains the right hand side. But that can take any real values. We need to turn it into a model for P (Y = 1|x, λ), which takes values only between 0 and 1. The odds ratio turns those probabilities into positive real numbers, then the log turns it into any real numbers. P (Y = 1|x, λ) T = eλ x P (Y = 0̃|x, λ) T T P (Y = 1|x, λ) = eλ x P (Y = 0̃|x, λ) = eλ x (1 − P (Y = 1|x, λ)) h i T λT x P (Y = 1|x, λ) 1 + e = eλ x T eλ x P (Y = 1|x, λ) = ← logistic function 1 + eλT x We’ll use maximum likelihood estimation. We choose parameters λ to maximize the likelihood of the data given the model. iid L(λ) := P (Y1 = y1 , . . . , Ym = ym |λ, x1 , . . . , xm ) = m Y P (Yi = yi |λ, xi ). i=1 Choose λ∗ ∈ argmax L(λ) = argmax log L(λ) | {z } λ λ log-likelihood where we take the log for convenience since it doesn’t effect the argmax. But first, we’ll simplify. We need P (Y = yi |λ, xi ) for both yi = 1 and yi = −1.  T eλ xi 1 1  y = 1, need P (Y = 1|λ, x ) := p = = = T T T i i   i 1+eλ xi 1+e−λ xi 1+e−yi λ xi λT xi λT xi If yi = −1 (i.e., 0̃), need P (Y = 0̃|λ, xi ) = 1 − pi = 1+eλT x − e λT x = 1λT x i i i 1+e 1+e 1+e    1 = . Tx 1+e−yi λ 3 i So, we can just write P (Y = yi |λ, xi ) = 1 1 + e−yi λT xi . Then, λ∗ ∈ argmax log L(λ) λ = argmax λ = argmin λ m X i=1 m X log 1 1 + e−yi λT xi T log(1 + e−yi λ xi ). i=1 This agrees with the “frequentist” derivation we had before. The loss is convex in λ so we can minimize by gradient descent. 4