Lecture 7 Notes: Maximum Likelihood Estimation

Lecture 7 Maximum Likelihood Estimation. 1 MLE Let f (·|θ) θ∈Θ X = (X1 , ..., Xn ) be a random sample from distribution ∏n f (x|θ) = i=1 f1 (xi |θ) where x = (x1 , ..., xn ). The log-likelihood be a parametric family. Let θ0 ∈ Θ. Then the joint pdf is ∑n ℓ(θ|x) = i=1 log f1 (xi |θ). The maximum f1 (·|θ0 ) is with with likelihood estimator is, by de˝nition, θ̂M L = arg max ℓ(θ|x). θ∈Θ The FOC is 1 ∑ ∂ℓ1 (θ̂M L |xi ) = 0. n i=1 ∂θ n Note that the ˝rst information equality is E[∂ℓ1 (θ0 |Xi )] = 0. Thus MLE is the method of moments estimator corresponding to the ˝rst information equality. So we can expect that the MLE is consistent. Indeed, the theorem below gives the consistency result for MLE: ̸ θ0 , In the setting above, assume that (1) θ0 is identi˝able, i.e. for any θ = there exists x such that f (x|θ) = ̸ f (x|θ0 ), (2) the support of f (·|θ) does not depend on θ, and (3) θ0 is an interior point of parameter space Θ. Then θ̂M L →p θ0 . Theorem 1 (MLE consistency). The proof of MLE consistency will be given in 14.382 and 14.385. What the proof does, it shows that function g(θ) = Eθ0 ℓ1 (θ|Xi ) converges to the function process θˆM L (here g(θ) will converge to Xi ∼ f1 (xi |θ0 )) is maximized at θ = θ0 and random process 1 n ℓ(θ|X) in a uniform manner in probability. Then it argues that the maximizer of the θ0 . Once we know that the estimator is consistent, we can think about the asymptotic distribution of the estimator. The next theorem gives the asymptotic distribution of MLE: In the setting above, assume that conditions (1)-(3) in the MLE consistency theorem hold. In addition, assume that (4) f1 (xi |θ) is thrice di˙erentiable with respect to θ and we can interchange integration with respect to x and di˙erentiation with respect to θ, and (5) |∂ 3 log f1 (xi |θ)/∂θ3 | ≤ M (x) and E[M (Xi )] < ∞. Then Theorem 2 (MLE asymptotic normality). √ n(θ̂M L − θ0 ) ⇒ N (0, I1−1 (θ0 )) 1 Proof. This is a sketch of the proof as it misses and important step. By de˝nition, Taylor theorem with a remainder, there is some random variable θ̃ ∂ℓ(θ̂M L ,x) ∂θ with value between θ0 and = 0. θ̂M L By the such that ∂ℓ(θ̂M L )|X ∂ℓ(θ0 |X) ∂ 2 ℓ(θ̃|X) = + (θ̂M L − θ0 ). ∂θ ∂θ ∂θ2 So, √ Since θ̂M L →p θ0 and θ̃ is between θ0 n(θ̂M L − θ0 ) = and − √1n ∂ℓ(θ∂θ0 |X) θ̂M L , θ̃ →p θ0 1 ∂ 2 ℓ(θ̃|X) n ∂θ 2 . as well. From θ̃ →p θ0 , one can prove that 1 ∂ 2 ℓ(θ0 |X) 1 ∂ 2 ℓ(θ̃|X) − = op (1). 2 n ∂θ2 n ∂θ We will not discuss this result here since it requires knowledge of the concept of asymptotic equicontinuity which we do not cover in this class. You will learn it in 14.385. Note, however, that this result does not follow from the Continuous mapping theorem since we have a sequence of random functions ℓ(θ|X) instead of just one non-random function. Suppose we believe in this result. Then, by the Law of large numbers, [ 2 ] n 1 ∑ ∂ 2 log f1 (Xi |θ0 ) ∂ log f1 (Xi |θ0 ) 1 ∂ 2 ℓ(θ0 |X) = → E = −I1 (θ0 ). p n i=1 ∂θ2 ∂θ2 n ∂θ2 [ Next, by the ˝rst information equality, E ∂ log f1 (Xi |θ0 ) ∂θ ] [ =0 while V ar ∂ log f1 (Xi |θ0 ) ∂θ ] = I1 (θ0 ). Thus, by the Central limit theorem, 1 ∂ℓ(θ0 |X) 1 ∑ ∂ log f1 (Xi |θ0 ) √ =√ ⇒ N (0, I(θ0 )). ∂θ ∂θ n n i=1 n Finally, by the Slutsky theorem, √ n(θ̂M L − θ0 ) ⇒ N (0, I −1 (θ0 )). One interpretation of MLE asymptotics is that the MLE is asymptotically e°cient (hit Rao-Cramer bound in very large samples). Example Let X1 , ..., Xn be a random sample from a distribution with pdf distribution is called exponential. Its log-likelihood for one draw is 1/λ − 2 1 (λ|xi ) xi and ∂ ℓ∂λ 2 = −1/λ log-likelihood for the whole is Its asymptotic distribution is 2 . So Fisher information is ∑n 2 ℓ1 (λ|xi ) = log λ − I1 (λ) = 1/λ ℓ(θ|x) = n log λ − λ i=1 xi . The FOC √ given by n(λ̂M L − λ) ⇒ N (0, λ2 ). f (x|λ) = λ exp(−λx). is 2 . Let us ˝nd the MLE for n λ̂M L − ∑n i=1 This (λ|xi ) λxi . So ∂ℓ1 ∂λ xi = 0. So λ. λ̂M L = = The 1 . Xn 2 Inference using MLE We will have a longer discussion about how to estimate the asymptotic variance of the MLE (I1 (θ0 )) −1 later when we will discuss asymptotic tests. Right now I want to mention several suggestions. I1 (θ) is a continuous function in θ (which is needed for asymptotic results), ( )−1 ( −1 )−1 consistent for θ0 , the quantity I1 (θ̂M L ) is consistent for I1 (θ0 ) . First of all, if θ̂M L is then given that Second, by de˝nition of Fisher information, it equals to the expectation of either negative second derivative of the likelihood or of the squared score. Instead of taking expectation one may approximate it by taking averages. For example 1 Iˆ = − n n ∑ ∂ 2 ℓ1 (θ̂|Xi ) ∂θ2 i=1 will be a consistent estimator of the Fisher information. The third idea to be used in this context is parametric bootstrap. Assume from our sample of size • Simulate sample n. For b = 1, ..., B θ̂M L is the MLE we obtained do the following: ∗ ∗ Xb∗ = (X1b , ..., Xnb ) as i.i.d. draws from f1 (xi |θ̂M L ) (that is, assuming that θˆM L is the true parameter value). • Find MLE using sample Xb∗ , Calculate the sample variance of denote it ∗ (θ1∗ , ..., θB ), θb∗ . it gives the bootstrap approximation to (nI1 (θ0 ))−1 . You may also do bootstrap- bias correction using similar procedure. 3 When MLE asymptotic theory fails us... A word of caution. For asymptotic normality of MLE, we should have common support. Let Example us see what might happen otherwise. Let So √ n(θ̂M L − θ) X1 , ..., Xn be a random sample from U [0, θ]. V (X(n) ) = θ2 n/((n + 1)2 (n + 2)) ≈ θ2 /n2 . and worked, we would have V (X(n) ) ≈ 1/(nI(θ)). On the other hand, if the theorem The MLE happens to be super-consistent here, means it converges to the true value at a faster speed than the regular parametric speed of of Θ. Let Now, let us consider what might happen if the true parameter value X1 , ..., Xn µ̂M L = X n if θ̂M L = X(n) . is always nonpositive. So it does not converge to mean zero normal distribution. In fact, E[X(n) ] = (n/(n + 1))θ Example Then be a random sample from distribution Xn ≥ 0 and 0 otherwise. Suppose that N (µ, 1) µ0 = 0 . Then with √ µ ≥ 0. √ 1/ n. θ0 were on the boundary As an exercise, check that n(µ̂M L − µ0 ) is always nonnegative. So it does not converge to mean zero normal distribution. Example Finally, note that it is implicitly assumed both in the consistency and asymptotic normality theorems that parameter space should not depend on n. Θ is ˝xed, i.e. independent of n. In particular, the number of parameters Indeed, let ( Xi = X1i X2i ) (( ∼N µi µi 3 ) ( , σ2 0 0 σ2 )) for i = 1, ..., n, and X1 , ..., Xn to in˝nity, the MLE for σ 2 be mutually independent. One can show that if the sample size (n) increases is inconsistent in this case, though a consistent estimator for σ2 exists. What is interesting, though we won't show it here is that a bootstrap does not help in this cases, that is, the bootstrap approximation to the distribution of of 4 θ̂M L is not close to the true ˝nite-sample distribution θ̂M L . Pseudo-MLE Let us have a sample X = (X1 , ..., Xn ) is, let's assume it has pdf g(xi ). i.i.d from some distribution. We do not know what distribution it But we wrongly assumed a speci˝c parametric family, that is, we assumed Xi ∼ f1 (xi |θ). What would happen if we do MLE. Apparently, MLE will be estimating a pseudo-true parameter value θ0 with minimizes in some sense the distance between g(·) and family f (·|θ). In particular: ∫ log[f1 (xi |θ)]g(xi )dxi = arg max E log f1 (Xi |θ). θ0 = arg max θ Parameter θ0 θ may be of interest or may be not. Under some regularity condition θ̂M L →p θ0 , and in most parts the logic of the proof of theorem about normality will hold. However, the the information equality [( would fail. De˝ne Σ1 = E ∂ log f1 (Xi |θ0 ) ∂θ0 [ Σ2 = −E Σ1 ̸= Σ2 . Xi ∼ g(·). If g is not in the parametric family, But using the logic of the proof, we can prove that √ This asymptotic variance , ] ∂ 2 log f1 (Xi |θ0 ) , ∂θ02 where expectations in both cases are taken assuming that then in general )2 ] −1 Σ−1 2 Σ1 Σ2 −1 n(θ̂M L − θ0 ) ⇒ N (0, Σ−1 2 Σ1 Σ2 ) is often called White's due to White's (1980) paper and thus White's standard errors. 4