University:
University of Michigan
Course:
STATS 500 | DATASCI 500. Statistical Learning I: Regression
Academic year:

2021
Views:

181

Pages:

20
Author:

Senqanawa2k

Generalized Linear Models 2021 1 / 52 Motivation for nonlinear models The key properties of a linear model are that E [y |X ] = X β and var[y |X ] ∝ I . In some cases where these conditions are not met, we can transform y so that the properties of a linear model are well-satisfied. However it is often difficult to find a transformation that simultaneously linearizes the mean and gives constant variance. Also, if y lies in a restricted domain (e.g. yi ∈ {0, 1}), parameterizing E [y |X ] as a linear function of X violates the domain restriction. Generalized linear models (GLM’s) are a class of nonlinear regression models that can be used in certain cases where linear models do not fit well. 2 / 52 Logistic regression Logistic regression is a specific type of GLM. We will develop logistic regression from first principles before discussing GLM’s in general. Logistic regression is used for binary outcome data, where yi = 0 or yi = 1. It is defined by the probability mass function P(yi = 1|xi = x) = 1 exp(β 0 x) = , 0 1 + exp(β x) 1 + exp(−β 0 x) which implies that P(yi = 0|xi = x) = 1 − P(yi = 1|xi = x) = 1 , 1 + exp(β 0 x) where in most cases, xi0 = 1 so β0 is the intercept. 3 / 52 Logistic regression This plot shows P(y = 1|x) and P(y = 0|x), plotted as functions of β 0 x: 1.0 Probability 0.8 ′ P (Y=1| β X) 0.6 ′ P (Y=0 | β X) 0.4 0.2 0.0 -6 -4 -2 0 ′ 2 4 6 βX 4 / 52 Logistic regression The logit function logit(x) = log(x/(1 − x)) maps the unit interval onto the real line. The inverse logit function, or expit function expit(x) = logit−1 (x) = exp(x) 1 + exp(x) maps the real line onto the unit interval. In logistic regression, the logit function is used to map the linear predictor β 0 x to a probability. 5 / 52 Logistic regression The linear predictor in logistic regression is the conditional log odds: P(y = 1|x) = β 0 x. log P(y = 0|x) Thus one way to interpret a logistic regression model is that a one unit increase in xj (the j th covariate) results in a change of βj in the conditional log odds. Or, a one unit increase in xj results in a multiplicative change of exp(βj ) in the conditional odds. 6 / 52 Latent variable model for logistic regression It may make sense to view the binary outcome y as being a dichotomization of a latent continuous outcome yc , y = I(yc ≥ 0). Suppose yc |X follows a logistic distribution, with CDF F (yc |x) = exp(yc − β 0 x) . 1 + exp(yc − β 0 x) In this case, y |x follows the logistic regression model: P(y = 1|x) = P(yc ≥ 0|x) = 1− exp(0 − β 0 x) exp(β 0 x) = . 1 + exp(0 − β 0 x) 1 + exp(β 0 x) 7 / 52 Mean/variance relationship for logistic regression Since the mean and variance of a Bernoulli trial are linked, the mean structure E [y |x] = P(y = 1|x) = exp(β 0 x) 1 + exp(β 0 x) also determines the variances var[y |x] = P(y = 1|x) · P(y = 0|x) = 2+ 1 . + exp(−β 0 x) exp(β 0 x) Since the variance depends on x, logistic regression models are always heteroscedastic. 8 / 52 Logistic regression and case-control studies Suppose we sample people based on their disease status D (D = 1 is a case, D = 0 is a control). We are interested in a binary marker M ∈ {0, 1} that may predict a person’s disease status. The prospective log odds log P(D = 1|M = m) P(D = 0|M = m) measures how informative the marker is for the disease. 9 / 52 Logistic regression and case-control studies Suppose we model P(M = m|D = d) using logistic regression, so P(M = 1|D = d) = exp(α + βd) 1 + exp(α + βD) P(M = 0|D = d) = 1 . 1 + exp(α + βd) More generally, P(M = m|D = d) = exp(m(α + βd)) . 1 + exp(α + βd) 10 / 52 Logistic regression and case-control studies Since log P(M = 1|D = d) = α + βd P(M = 0|D = d) we see that β is the coefficient of d in the retrospective log odds. 11 / 52 Logistic regression and case-control studies The prospective log odds can be written log P(D = 1|M = m) P(D = 0|M = m) P(M P(M P(M = log P(M = log = m|D = m|D = m|D = m|D = 1)P(D = 0)P(D = 1)P(D = 0)P(D = 1)/P(M = m) = 0)/P(M = m) = 1) = 0) 12 / 52 Logistic regression and case-control studies Continuing from the previous slide, we have log P(M = m|D = 1)P(D = 1) = P(M = m|D = 0)P(D = 0) exp(m · (α + β))/(1 + exp(α + β)) P(D = 1) · , log exp(m · α)/(1 + exp(α)) P(D = 0) which equals 1 + exp(α) P(D = 1) βm + log · . 1 + exp(α + β) P(D = 0) Thus β is both the coefficient of d in the retrospective log odds, and it is the coefficient of m in the prospective log odds. This is sometimes called case/control convertibility. 13 / 52 Estimation and inference for logistic regression Assuming independent cases, the log-likelihood for logistic regression is Y exp(yi · β 0 xi ) 1 + exp(β 0 xi ) i X X β 0 xi − log(1 + exp(β 0 xi )). L(β|y , X ) = log = i:yi =1 i This likelihood is for the conditional distribution of y given X . As in linear regression, we do not model the marginal distribution of x (a row of X ). 14 / 52 Estimation and inference for logistic regression Logistic regression models are usually fit using maximum likelihood estimation. This means that the parametric likelihood above is maximized as a function of β. The gradient of the log-likelihood function (the score function) is G (β|y , X ) = ∂ L(β|y , X ) ∂β X X xi − exp(β 0 xi ) xi 1 + exp(β 0 xi ) i:yi =1 i X exp(β 0 xi ) = yi − xi . 1 + exp(β 0 xi ) = i 15 / 52 Estimation and inference for logistic regression The Hessian of the log-likelihood is H(β|y , X ) = X ∂2 exp(β 0 xi ) L(β|y , X ) = − xi x 0 . ∂ββ 0 (1 + exp(β 0 xi ))2 i i The Hessian is strictly negative definite as long as the design matrix has independent columns. Therefore L(β|y , X ) is a concave function of β, so has a unique maximizer, and hence the MLE is unique. 16 / 52 Estimation and inference for logistic regression From the general theory of the MLE, the Fisher information I (β) = −(E [H(β|y , X )|X ])−1 is the asymptotic sampling covariance matrix of the MLE β̂. Since H(β|y , X ) does not depend on y , I (β) = −H(β|y , X )−1 . Since β̂ is an MLE for a regular problem, it is consistent, asymptotically unbiased, and asymptotically normal if the model is correctly specified. 17 / 52 Poisson regression The Poisson distribution is a single-parameter family of distributions on the sample space {0, 1, 2, . . .}. A key property of the Poisson distribution is that the mean is equal to the variance. The Poisson distribution is usually parameterized in terms of a parameter λ that is equal to the common mean and variance. In regression, we don’t want just a single distribution. Instead we want a family of distributions indexed by the covariate vector x. To create a regression methodology based on the Poisson distribution, we can formulate a regression model in which y |x is Poisson, with mean and variance equal to λ(x) = exp(β 0 x). 18 / 52 Poisson regression Since the mean function in a Poisson distribution has an exponential form, the covariates are related multiplivatively to the mean. If we contrast the mean value for two different covariate vectors, (1) (2) (1) (2) x (1) and x (2) , such that xj − xj = 1, and xk = xk for k 6= j, then the means at these two points are related through: λ(x (1) ) = exp(βj )λ(x (2) ). 19 / 52 Poisson regression Setting the mean to be λi = exp(β 0 xi ), the PMF for one observation in a Poisson regression model is exp−λi λyi i /yi ! The corresponding contribution to the log likelihood is −λi + yi log(λi ) − log(yi !) = − exp(β 0 xi ) + yi · β 0 xi − log(yi !), and the contribution to the score function is −xi exp(β 0 xi ) + yi · xi = (yi − exp(β 0 xi ))xi . 20 / 52

Generalized Linear Models-1

Related Documents

Recommended Documents

Description

Related Documents

Free up your schedule!

Take 5 seconds to unlock