Point Estimators and Point Estimation Methods

Lecture Note 8 ∗ Point Estimators and Point Estimation Methods MIT 14.30 Spring 2006 Herman Bennett Given a parameter with unknown value, the goal of point estimation is to use a sample to compute a number that represents in some sense a good guess for the true value of the parameter. 19 Definitions 19.1 Parameter A pmf/pdf can be equivalently written as fX (x) or fX (x|θ), where θ represents the constants that fully define the distribution. For example, if X is a normally distributed RV, the constants µ and σ will fully define the distribution. These constants are called parameters and are generally denoted by the Greek letter θ. 1 Example 19.1. – Normal distribution: f (x|θ) ≡ f (x|µ, σ), 2 parameters: θ1 = µ and θ2 = σ; – Binomial distribution: f (x|θ) ≡ f (x|n, p), 2 parameters: θ1 = n and θ2 = p; – Poisson distribution: f (x|θ) ≡ f (x|λ), 1 parameter: θ = λ; – Gamma distribution: f (x|θ) ≡ f (x|α, β), 2 parameters: θ1 = α and θ2 = β. 19.2 (Point) Estimator A (point) estimator of θ, denoted by θ̂, is an statistic (a function of the random sample): θ̂ = r(X1 , X2 , ..., Xn ). ∗ (63) Caution: These notes are not necessarily self-explanatory notes. They are to be used as a complement to (and not as a substitute for) the lectures. 1 Another way of saying this: parameters are constants that index a family of distributions. 1 Herman Bennett LN8—MIT 14.30 Spring 06 • Note that θ̂ does not depend directly on θ, but only indirectly through the random process of each Xi . A (point) estimate of θ is a realization of the estimator θ̂ (i.e.: a function of the realization of the random sample): θ̂ = r(x1 , x2 , ..., xn ). (64) Example 19.2. Assume we have a random sample X1 , ..., X10 from a normal distribution N (µ, σ 2 ) and we want to have an estimate of the parameter µ (which is unknown). There is an infinite number of estimators of µ that we could construct. In fact, any function of the random sample could classify as an estimator of µ, for example: ⎧ ⎪ X10 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 2 ⎪ ⎪ ⎪ ⎨ X10 +X1 θ̂ = r(X1 , X2 , ..., X10 ) = 2 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ¯= X 1 10 �10 i=1 Xi 1.5X2 etc. Example 19.3. Assume a random sample X1 , ..., Xn from a U[0, θ] distribution, where θ is unknown. Propose 3 different estimators. 2 Herman Bennett 20 LN8—MIT 14.30 Spring 06 Evaluating (Point) Estimators Since there are many possible estimators, we need to define some characteristic in order to evaluate and rank them. 20.1 Unbiasedness An estimator θ̂ is said to be an unbiased estimator of the parameter θ, if for every possible value of θ: E(θ̂) = θ (65) If θ̂ is not unbiased, it is said to be a biased estimator, where the difference E(θ̂) − θ is called the bias of θ̂. • If the random sample X1 , ..., Xn is iid with E(Xi ) = θ, then the sample mean estimator n � ¯n = 1 θ̂ = X Xi n i=1 (66) ¯ n ) = θ (see Example 18.1 in Lecture is an unbiased estimator of the population mean: E(X Note 7). • If the random sample X1 , ..., Xn is iid with E(Xi ) = µ and V ar(Xi ) = θ, then the sample variance estimator n θ̂ = Sn2 = 1 � (Xi − X̄n )2 n − 1 i=1 (67) is an unbiased estimator of the population variance: E(S 2 ) = θ (see Example 18.1 in Lec­ ture Note 7). Example 20.1. Let Xi ∼ U [0, θ]. Assume a random sample of size n and define an estimator of θ as: n 2� Xi . θ̂ = n i=1 Is θ̂ biased? 3 Herman Bennett 20.2 LN8—MIT 14.30 Spring 06 Efficiency Let θ̂1 and θ̂2 be two unbiased estimators of θ. θˆ1 is said to be more efficient than θ̂2 , if for a given sample size n, V ar(θ̂1 ) < V ar(θ̂2 ) (68) Where V ar(θ̂i ) is the variance of the estimator. Let θ̂1 be an unbiased estimator of θ. θ̂1 is said to be efficient, or minimum variance unbiased estimator, if for any unbiased estimators of θ, θ̂k , V ar(θ̂1 ) ≤ V ar(θ̂k ) (69) • Do not confuse the variance of the estimator θ̂, V ar(θ̂), with the sample variance esti­ mator S 2 , which is an unbiased estimator of the population variance σ 2 (!). Example 20.2. How would you compare the efficiency of the estimators in Example 19.2? Which of these estimators is unbiased? 20.3 Mean Squared Error Why restrict ourselves to the class of unbiased estimators? The mean square error (MSE ) specifies for each estimator θ̂ a trade off between bias and efficiency. M SE(θ̂) = E[(θ̂ − θ)2 ] = V ar(θ̂) + (bias(θ̂))2 (70) θ̂ is the minimum mean square error estimator of θ if, among all possible estimators of θ, it has the smaller MSE for a given sample size n. 4 Herman Bennett LN8—MIT 14.30 Spring 06 Example 20.3. Graph the pdf of two estimators such that the bias of the first estimator is less of a problem than inefficiency (and vice versa for the other estimator). 20.4 Asymptotic Criteria 20.4.1 Consistency p Let θ̂ be an estimator of θ. θ̂ is is said to be consistent if θ̂ −→ θ (Law of Large Numbers; Lecture Note 7). Example 20.4. Assume a random sample of size n from a population f (x), where E(X) = µ (unknown). Which if the following estimators is consistent? n 1� µˆ1 = Xi n i=1 n 1 � µˆ2 = Xi n − 5 i=1 n−5 1 � µˆ3 = Xi n − 5 i=1 • MSE → 0 as n → ∞ =⇒ consistency. 20.4.2 Asymptotic Efficiency Let θ̂1 be an estimator of θ. θ̂1 is is said to asymptotically efficient if it satisfies the definition of an efficient estimator, equation (69), when n → ∞. 5 Herman Bennett 21 LN8—MIT 14.30 Spring 06 Point Estimation Methods The following are two standard methods used to construct (point) estimators. 21.1 Method of Moments (MM) Let X1 , X2 , ..., Xn be a random sample from a population with pmf/pdf f (x|θ1 , ..., θk ), where θ1 , ..., θk are unknown parameters. One way to estimate these parameters is by equating the first k population moments to the corresponding k sample moments. The resulting k estimators are called method of moments (MM) estimators of the parameters θ1 , ..., θk . 2 The procedure is summarized as follows: ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ Sample moment System of equations: ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ k equations and ⎪ ⎪ ⎪ �n ⎪ 1 k unknowns ⎪ ⎪ i=1 Xi n ⎪ ⎪ ⎪ ⎪ =⇒ yields θˆ1 , ..., θˆk ⎪ ⎪ ⎪ �n ⎪ 1 2 ⎨ i=1 Xi n ⎪ ⎪ ⎪ ⎪ Note that �n ⎪ 1 3 ⎪ ⎪ i=1 Xi j n ⎪ ⎪ i) E(Xi ) = gj (θ1 , ..., θk ) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ii) the realization of ⎪ ... ⎪ ⎪ ⎪ the sample moment ⎪ ⎪ ⎪ is a scalar. ⎪ ⎪ ⎪ �n ⎩ 1 k i=1 Xi n 2 Population moment (theoretical) = E(Xi1 ) First moment = E(Xi2 ) Second moment = E(Xi3 ) Third moment . .. . .. = E(Xik ) This estimation method was introduced by Karl Pearson in 1894. 6 k th moment Herman Bennett LN8—MIT 14.30 Spring 06 Example 21.1. Assume a random sample of size n from a N (µ, σ 2 ) population, where µ and σ 2 are unknown parameters. Find the MM estimator of both parameters. Example 21.2. Assume a random sample of size n from a gamma distribution: f (x) = 1 xα−1 e−x/β for 0 < x < ∞. Assume also that the random sample realization is Γ(α)β α � � characterized by n1 ni=1 xi = 7.29 and n1 ni=1 xi2 = 85.59. Find the MM estimators of α and β (parameters). Remember that E(Xi ) = αβ and Var(Xi ) = αβ 2 . 7 Herman Bennett 21.2 LN8—MIT 14.30 Spring 06 Maximum Likelihood Estimation (MLE) Let X1 , X2 , ..., Xn be a random sample from a population with pmf/pdf f (x|θ1 , ..., θk ), where θ1 , ..., θk are unknown parameters. Another way to estimate these parameters is finding the values of θˆ1 , ..., θˆk that maximize the likelihood that the observed sample is generated by f (x|θˆ1 , ..., θˆk ). The joint pmf/pdf of the random sample, f (x1 , x2 , ..., xn |θ1 , ..., θk ), is called the likelihood function, and it is denoted by L(θ|x). ⎧ ⎪ ⎪ ⎨ f (x1 , ..., xn |θ1 , ..., θk ) generally L(θ|x) = L(θ1 , ..., θk |x1 , ..., xn ) = ⎪ ⎪ ⎩ �n f (x |θ , ..., θ ) i 1 k i=1 random sample (iid ) Where θ and x are vectors such that θ = (θ1 , ..., θk ) and x = (x1 , ..., xn ). For a given sample vector x, denote θ̂M LE (x) the parameter value of θ at which L(θ|x) attains its maximum. Then, θ̂M LE (x) is the maximum likelihood estimator (MLE) of the (unknown) parameters θ1 , ..., θk . 3 • Intuition: Discrete case. 3 This estimation method was introduced by R.A. Fisher in the 1912. 8 Herman Bennett LN8—MIT 14.30 Spring 06 • Does a global maximum exist? Can we find it? Unique?...Back to Calculus 101: f oc : ∂L(θ|x) = 0, ∂θi i = 1, ..k (for a well behave function.) Also need to check that is really a maximum and not a minimum (2nd derivative). • Many times it is easier to look for the maximum of LnL(θ|x) (same maximum since is a monotonic transformation...back to Calculus 101). • Invariance property of MLE: τ̂M LE (θ) = τ (θ̂M LE ). • For large samples MLE yields an excellent estimator of θ (consistent and asymptotically efficient). No wonder it is widely used. • But... i) Numerical sensitivity (robustness). ii) Not necessary unbiased. iii) Might be hard to compute. 9 Herman Bennett LN8—MIT 14.30 Spring 06 Example 21.3. Assume a random sample from a N (µ, σ 2 ) population, where the parame­ ters are unknown. Find the ML estimator of both parameters. Example 21.4. Assume a random sample from a U(0, θ) population, where θ is unknown. Find θ̂M LE . 10