University:
University of Michigan
Course:
STATS 500 | DATASCI 500. Statistical Learning I: Regression
Academic year:

2021
Views:

60

Pages:

23
Author:

herchychlx

Score equations The MLE is a stationary point of the score function. Thus, for logistic regression, the following equation is satisfied at the MLE: X yi − i exp(β 0 xi ) 1 + exp(β 0 xi ) xi = 0. For Poisson regression, this equation is satisfied at the MLE: X (yi − exp(β 0 xi ))xi i We also know that for OLS (viewed here as a Gaussian regression model), this equation is satisfied at the MLE X (yi − β 0 xi )xi = 0. i 21 / 52 Score equations Writing µi = E [yi |xi ], we see that for all three types of regression models, the following equation is satisfied. X (yi − µi )xi = 0. i This shows that the residuals are orthogonal to each covariate in all of these models, and that achieving this orthogonality characterizes the MLE. This turns out to be a useful generic framework for regression, as many different mean functions µ(β) can be substituted into this equation, and the solution of the equation can be used to estimate β. 22 / 52 Relationship between the mean and variance We have seen three parametric regression models, each of which expresses the mean in terms of the linear predictor. The family is the distributional family used to form the log-likelihood and score functions. For each of these models, the variance can also be related to the mean. Family Gaussian Binomial Poisson Mean (µ) β0x 1/(1 + exp(−β 0 x)) exp(β 0 x) Variance (v (µ)) 1 µ(1 − µ) µ 23 / 52 Relationship between the mean and variance The variance functions here are only specified up to a constant of proportionality. That is, var(yi |xi ) = φv (µi ), where φ is the scale parameter. In any single index model, µi = µ(β 0 xi ), so ∂µi /∂β = µ0 (β 0 xi ) · xi ∝ x. Note that in each case above, ∂µi /∂β is proportional to vi · xi , where vi ∈ R is the variance (but this is not always the case). 24 / 52 Estimating equations For the three examples we are focusing on here, the MLE can be defined as the solution to the estimating equations: X ∂µi /∂β · (yi − µi (β))/vi (β) = 0 i which can also be expressed X µ0 (ηi ) · (yi − µi (β)) · xi /vi (β) = 0 i where ηi = β 0 xi is the linear predictor. 25 / 52 Estimating equations The score equations constitute a system of p = dim(x) equations in p unknowns. It should be solvable unless there is some degeneracy in the equations. In the “canonical setting”, (∂µi /∂β)/vi (β) ∝ xi , so these equations are equivalent to the orthogonality between residuals and covariates. But we will see below that the form given here extends to some “non-canonical” settings and hence is somewhat more general. 26 / 52 Development of GLM’s using likelihoods A GLM is based on the following conditions: I The yi are conditionally independent given X . I The probability mass function or density can be written log p(yi |θi , φ, xi ) = wi (yi θi − γ(θi ))/φ + τ (yi , φ/wi ), where wi is a known weight, θi = g (β 0 xi ) for an unknown vector of regression slopes β, g (·) and γ(·) are smooth functions, φ is the “scale parameter” (which may be either known or unknown), and τ (·) is a known function. 27 / 52 Development of GLM’s using likelihoods The log-likelihood function is L(β, φ|y , X ) = X wi (yi θi − γ(θi ))/φ + τ (yi , φ/wi ). i The score function with respect to θi is wi (yi − γ 0 (θi ))/φ. 28 / 52 Development of GLM’s using likelihoods Next we need a fundamental fact about score functions. Let fθ (y ) be a density in y with parameter θ. The score function is ∂ ∂ log fθ (y ) = fθ (y )−1 fθ (y ). ∂θ ∂θ The expected value of the score function is ∂ log fθ (y ) = E ∂θ Z −1 ∂ fθ (y ) fθ (y )dy ∂θ fθ (y ) Z ∂ = fθ (y )dy ∂θ = 0. Thus the score function has expected value 0 when θ is at its true value. 29 / 52 Development of GLM’s using likelihoods Since the expected value of the score function is zero, we can conclude that E [wi (yi − γ 0 (θi ))/φ|X ] = 0, so E [yi |X ] = γ 0 (θi ) = γ 0 (g (β 0 xi )). Note that this relationship does not depend on φ or τ . 30 / 52 Development of GLM’s using likelihoods Using a similar approach, we can relate the variance to wi , φ, and γ 0 . By direct calculation, ∂ 2 L(θi |yi , xi , φ)/∂θi2 = −wi γ 00 (θi )/φ. Returning to the general density fθ (y ), we can write the Hessian as ∂2 ∂fθ (y ) ∂fθ (y ) ∂ −2 . log fθ (y ) = fθ (y ) fθ (y ) fθ (y ) − · ∂θθ0 ∂θθ0 ∂θ ∂θ0 31 / 52 Development of GLM’s using likelihoods The expected value of the Hessian is ∂ E log fθ (y ) = ∂θθ0 Z ∂ log fθ (y ) · fθ (y )dy ∂θθ0 Z Z ∂ ∂fθ (y )/∂θ ∂fθ (y )/∂θ0 · fθ ( = f (y )dy − θ ∂θθ0 fθ (y ) fθ (y ) ∂ = −cov log fθ (y )|X . ∂θ Therefore wi γ 00 (θi )/φ = var wi (yi − γ 0 (θi ))/φ|X so var[yi |X ] = φγ 00 (θi )/wi . 32 / 52 Examples of GLM’s Gaussian linear model: The density of y |X can be written 1 (yi − β 0 xi )2 2σ 2 = − log(2πσ 2 )/2 − yi2 /2σ 2 + (yi β 0 xi − (β 0 xi )2 /2)/σ 2 . log p(yi |xi ) = − log(2πσ 2 )/2 − This can be put into GLM form by setting g (x) = x, γ(x) = x 2 /2, wi = 1, φ = σ 2 , and τ (yi , φ) = − log(2πφ)/2 − yi2 /2φ. 33 / 52 Examples of GLM’s Logistic regression: The mass function of y |x can be written log p(yi |xi ) = yi log(pi ) + (1 − yi ) log(1 − pi ) = yi log(pi /(1 − pi )) + log(1 − pi ), where pi = logit−1 (β 0 xi ) = exp(β 0 xi ) . 1 + exp(β 0 xi ) Since log(pi /(1 − pi )) = β 0 x, this can be put into GLM form by setting g (x) = x, γ(x) = − log(1 − logit−1 (x)) = log(1 + exp(x)), τ (yi , φ) ≡ 0, w = 1, and φ = 1. 34 / 52 Examples of GLM’s Poisson regression: In Poisson regression, the distribution of y |x follows a Poisson distribution, with the mean response related to the covariates via log E [y |x] = β 0 x. It follows that log var[y |x] = β 0 x as well. The mass function can be written log p(yi |xi ) = yi β 0 xi − exp(β 0 xi ) − log(yi !), so in GLM form, g (x) = x, γ(x) = exp(x), w = 1, τ (yi ) = − log(yi !), and φ = 1. 35 / 52 Examples of GLM’s Negative binomial regression: In negative binomial regression, the probability mass function for the dependent variable Y is Γ(y + 1/α) P(yi = y |x) = Γ(y + 1)Γ(1/α) 1 1 + αµi 1/α αµi 1 + αµi y . The mean of this distribution is µi and the variance is µi + αµ2i . If α = 0 we get the same mean/variance relationship as the Poisson model. As α increases, we get increasingly more overdispersion. 36 / 52 Examples of GLM’s Negative binomial regression (continued): The log-likelihood (dropping terms that do not involve µ) is log P(yi = y |xi ) = y log( αµi ) − α−1 log(1 + αµi ) 1 + αµi Suppose we model the mean as µi = exp(β 0 xi ). Then in the standard GLM notation, we have θi = log α exp(β 0 Xi ) 1 + α exp(β 0 xi ) , so g (x) = log(α) + x − log(1 + α exp(x)), and γ(x) = −α−1 log(1 − exp(x)). 37 / 52 Link functions In a GLM, the link function maps the mean to the linear predictor ηi = xi0 β. Since E [yi |xi ] = γ 0 (g (η)), it follows that the link function is the inverse of γ 0 ◦ g . For example, in the case of logistic regression, γ 0 (g (η)) = exp(η)/(1 + exp(η)), which is the expit function. The inverse of this function is the logit function log(p/(1 − p)), so the logit function is the link in this case. 38 / 52 Link functions When g (x) = x, the resulting link function is called the canonical link function. In the examples above, linear regression, logistic regression, and Poisson regression all used the canonical link function, but negative binomial regression did not. The canonical link function for negative binomial regression is 1/x, but this does not respect the domain and is harder to interpret than the usual log link. Another setting where non-canonical links arise is the use of the log link function for logistic regression. In this case, the coefficients β are related to the log relative risk rather than to the log odds. 39 / 52 Estimating equations and quasi-likelihood As noted above, the regression parameters in a GLM can be estimated by solving these estimating equations: X ∂µi /∂β · (yi − µi (β))/vi (β) = 0 i Note that we only need to correctly specify vi (β) up to a constant. For example, in the Gaussian case, we can set vi (β) = 1. 40 / 52 Estimating equations and quasi-likelihood This opens up the possibility of specifying a large class of regression models through their first two moments, represented by the functions µ(β) and v (β). For example, we can get quasi-Poisson regression by specifying µi (β) = exp(xi0 β) and vi (β) = µi (β). This formulation of quasi-Poisson regression never refers to the Poisson distribution directly, it only depends on moments. It can be shown that solving the quasi-likelihood equations generally gives consistent estimates of β, as long as the data are sample from a population in which the specified mean and variance functions are correct. 41 / 52 Estimating equations and quasi-likelihood Wedderburn introduced a “quasi-likelihood” function that can be used when working with estimating equations. It has the form Z Q(y ; µ, v ) = 0 µ y −u du v (u) Since ∂Q/∂β = ∂µ/∂β · ∂Q/∂µ, and ∂Q/∂µ = (y − µ)/v (µ) by the fundamental theorem of calculus, we ssee that ∂Q/∂β gives the estimating equations discussed above. 42 / 52 Estimating equations and quasi-likelihood In some cases the quasi-likelihood is an actual likelihood, but even if it is not, we can use it in place of a likelihood for many purposes. For example, we can define a “Quasi Information Criterion” QIC, analogous to AIC for model selection as X Q(yi , µ̂i , v ) − p, i where p = dim(β). This quantity is to be maximized, or alternatively we can minimize −2 X Q(yi , µ̂i , v ) + 2p. i 43 / 52

Generalized Linear Models-2

Related Documents

Recommended Documents

Description

Related Documents

Free up your schedule!

Take 5 seconds to unlock