Lecture 4 Notes: Sufficient Statistics, Introduction to Estimation

Lecture 4 Su°cient Statistics. Introduction to Estimation 1 Su°cient statistics Let f (x|θ) with θ ∈ Θ be some parametric family. f (x|θ). Let X = (X1 , ..., Xn ) be a random sample from distribution Suppose we would like to learn parameter value X statistic allows us to separate information contained in information as long as we are concerned with parameter θ from our sample. The concept of su°cient into two parts. One part contains all the valuable θ, while the other part contains pure noise in the sense that this part has no valuable information. Thus, we can ignore the latter part. De˝nition 1. depend on Let T (X) Statistic θ is su°cient for if the conditional distribution of be a su°cient statistic. Consider the pair θ information about as X alone, since T (X) T (X)), T (X), we can discard X = (X1 , ..., Xn ) only parameter is T (X) X. Obviously, µ (θ = µ). conditional distribution of X X T (X) does not is independent of We have already seen that T (X) = t. n ∑ (xi − µ)2 − n(xn − µ)2 θ θ. T (X), i=1 = X itself has X (in is more likely than another. N (µ, σ 2 ). Suppose that σ2 T (X) = X n ∼ N (µ, σ /n). 2 is known. Thus, the Let us calculate the n ∑ (xi − xn + xn − µ)2 − n(xn − µ)2 i=1 = then Thus, by observing First, note that = contains the same completely. be a random sample from given (X, T (X)) But if we know we cannot say whether one particular value of parameter Therefore, once we know Let (X, T (X)). is a function of no value for us since its conditional distribution given Example given θ. T (X) addition to X n ∑ (xi − xn )2 + 2 i=1 n ∑ n ∑ i=1 (xi − xn )2 . i=1 1 (xi − xn )(xn − µ) Therefore fX|T (X) (x|T (X) = T (x)) fX (x) fT (T (x)) ∑n exp{− i=1 (xi − µ)2 /(2σ 2 )}/((2π)n/2 σ n ) = exp{−n(xn − µ)2 /(2σ 2 )}/((2π)1/2 σ/n1/2 ) n ∑ = exp{− (xi − xn )2 /(2σ 2 )}/((2π)(n−1)/2 σ n−1 /n1/2 ), = i=1 which is independent of Note, however, that Xn µ. T (X) = X n We conclude that is not su°cient if σ 2 is a su°cient statistic for our parametric family. is not known. 2 Factorization Theorem The Factorization Theorem gives a general approach for how to ˝nd a su°cient statistic: Theorem 2 (Factorization Theorem). Let f (x|θ) be the pdf of X . Then T (X) is a su°cient statistic if and only if there exist functions g(t|θ) and h(x) such that f (x|θ) = g(T (x)|θ)h(x). Proof. Let l(t|θ) Suppose depend on be the pdf of T (X) θ. T (X). is a su°cient statistic. Denote it by h(x). Then Then fX|T (X) (x|T (X) = T (x)) = fX (x|θ)/l(T (x)|θ) f (x|θ) = l(T (x)|θ)h(x). Denoting l by g yields the result in one direction. In the other direction we will give a sloppy proof. Denote ∫ A(x) = {y : T (y) = T (x)}. ∫ l(T (x)|θ) = A(x) g(T (y)|θ)h(y)dy = g(T (x)|θ) A(x) fX|T (X) (x|T (X) = T (x)) = = = θ. We conclude that T (X) h(y)dy. A(x) So which is independent of Then ∫ f (y|θ)dy = f (x|θ) l(T (x)|θ) g(T (x)|θ)h(x) ∫ g(T (x)|θ) A(x) h(y)dy h(x) , h(y)dy A(x) ∫ is a su°cient statistic. 2 does not Example from Let us show how to use the factorization theorem in practice. Let 2 N (µ, σ ) where both µ and f (x|θ) = σ 2 are unknown, i.e. exp{− exp{−[ x2i − 2µ i=1 Thus, ∑n ∑n T (X) = ( i=1 Xi2 , i=1 Xi ) θ = (µ, σ ). X1 , ..., Xn be a random sample Then n ∑ (xi − µ)2 /(2σ 2 )}/((2π)n/2 σ n ) i=1 n ∑ = 2 n ∑ xi + nµ2 ]/(2σ 2 )}/((2π)n/2 σ n ). i=1 is a su°cient statistic (here h(x) = 1 and g is the whole thing). Note that in this example we actually have a pair of su°cient statistics. In addition, as we have seen before, f (x|θ) = exp{−[ n ∑ (xi − xn )2 + n(xn − µ)2 ]/(2σ 2 )}/((2π)n/2 σ n ) i=1 = exp{−[(n − 1)s2n + n(xn − µ)2 ]/(2σ 2 )}/((2π)n/2 σ n ). Thus, T (X) = (X n , sn2 ) Note that Example Xn is another su°cient statistic. Yet another su°cient statistic is T (X) = (X1 , ..., Xn ). is not su°cient in this example. A less trivial example: let θ ≤ mini Xi ≤ maxi Xi ≤ 1 + θ T (X) = (X(1) , X(n) ) and 0 X1 , ..., Xn be a random sample from otherwise. In other words, U [θ, 1 + θ]. Then f (x|θ) = 1 f (x|θ) = I{θ ≤ X(1) }I{1 + θ ≥ X(n) }. if So is su°cient. 3 Minimal Su°cient Statistics Could we reduce su°cient statistic say, T (X) ⋆ T (X). and ⋆ T (X) = r(T (X)). T ∗ We say that information reduction than some function T is not bigger than such that ⋆ T (X) T Suppose we have two statistics, if there exists some function whenever we know must change its value as well. In this sense T T (X). ∗ r such that In this case when does not give less of an T. A su°cient statistic r T ⋆ In other words, we can calculate changes its value, statistic De˝nition 3. T (X) in the previous example even more? T ⋆ (X) is called minimal if for any su°cient statistic T (X) there exists ⋆ T (X) = r(T (X)). Thus, in some sense, the minimal su°cient statistic gives us the greatest data reduction without a loss of information about parameters. The following theorem gives a characterization of minimal su°cient statistics: Theorem 4. Let f (x|θ) be the pdf of X and T (X) be such that, for any x, y, statement {f (x|θ)/f (y|θ) does not depend on θ} is equivalent to statement {T (x) = T (y)}. Then T (X) is minimal su°cient. We will leave this statement unproven here. Example Let us now go back to the example with independent of Therefore θ if and only if T (X) = (X(1) , X(n) ) x(1) = y(1) and X1 , ..., Xn ∼ U [θ, 1 + θ]. x(n) = y(n) is minimal su°cient. 3 Ratio f (x|θ)/f (y|θ) which is the case if and only if is T (x) = T (y). Example X1 , ..., Xn Let distribution with the pdf theorem above, θ, be a random sample from the Cauchy distribution with parameter f (x|θ) = 1/(π(x − θ) ). 2 T (X) = (X(1) , ..., X(n) ) Then f (x1 , ..., xn |θ) = 1/(π n ∏n i=1 (xi i.e. the − θ) ). By the θ, then an 2 is minimal su°cient. 4 Estimators. Properties of estimators. An estimator is a function of the data (statistic). If we have a parametric family with parameter estimator of Example θ is usually denoted by For example, if θ̂. X1 , ..., Xn is a random sample from some distribution with mean 2 σ , then sample average µ̂ = X n is an estimator of the population mean, ∑n 2 i=1 (Xi − X n ) /(n − 1) is an estimator of the population variance. µ and variance and sample variance σ̂ 2 = s2 = 4.1 Unbiasness Let X be our data. Let We say that θ̂ is θ expectation when θ̂ = T (X) unbiased for be an estimator where θ if Eθ [T (X)] = θ is the true parameter value. The T is some function. for all possible values of bias of θ̂ θ is de˝ned by Bias(θ̂) where sample X1 , ..., Xn E[µ̂] = µ and 2 from some distribution with mean E[s ] = σ 2 µ and variance σ denotes the = Eθ [θ̂] − θ. Thus, the concept of unbiasness means that we are on average correct. For example, if 2 Eθ X is a random , then, as we have already seen, . Thus, sample average and sample variance are unbiased estimators of population mean and population variance correspondingly. 4.2 E°ciency: MSE Another of the concepts that evaluates performance of estimators is the MSE (Mean Squared Error). By de˝nition, MSE(θ̂) = Eθ [(θ̂ − θ)2 ]. The theorem below gives a useful decomposition for MSE: Theorem 5. MSE(θ̂) = Bias2 (θ̂) + V (θ̂). Proof. E[(θ̂ − θ)2 ] = E[(θ̂ − E[θ̂] + E[θ̂] − θ)2 ] = E[(θ̂ − E[θ̂])2 + (E[θ̂] − θ)2 + 2(θ̂ − E[θ̂])(E[θ̂] − θ)] = V (θ̂) + Bias2 (θ̂) + 2E[θ̂ − E[θ̂]](E[θ̂] − θ) = V (θ̂) + Bias2 (θ̂). Estimators with smaller MSE are considered to be better, or more 4 e°cient. 4.3 Connection between e°ciency and su°cient statistics Let X = (X1 , ..., Xn ) T (X) be a random sample from distribution be a su°cient statistic for θ. fθ . Let θ̂ = δ(X) be an estimator of θ. Let As we have seen already, an MSE provides one way to compare the quality of di˙erent estimators. In particular, estimators with smaller MSE are said to be more e°cient. On the other hand, once we know T (X), X. we can discard θ̂ = δ(X), ˆ: e°cient as θ theorem below shows that for any estimator only through T (X) Theorem 6 (Rao-Blackwell) and is at least as How do these concepts relate to each other? The there is another estimator which depends on data . In the setting above, de˝ne X ϕ(T ) = E[δ(X)|T ]. Then θ̂2 = ϕ(T (X)) is an estimator for θ and MSE(θ̂2 ) ≤ MSE(θ̂). In addition, if θ̂ is unbiased, then θ̂2 is unbiased as well. Proof. To show that δ(X) given depend on is an estimator, we have to check that it does not depend on θ, the conditional distribution of X su°cient for of θ̂2 θ. T is independent of Thus, ϕ(T (X)) θ given T θ. X and θ̂2 Indeed, since T is So the conditional distribution E[δ(X)|T ] as well. In particular, the conditional expectation depends only on the data MSE(θ̂) is independent of θ. does not is an estimator. = E[(θ̂ − θ̂2 + θ̂2 − θ)2 ] = E[(θ̂ − θ̂2 )2 ] + 2E[(θ̂ − θ̂2 )(θ̂2 − θ)] + E[(θ̂2 − θ)2 ] = E[(θ̂ − θ̂2 )2 ] + 2E[(θ̂ − θ̂2 )(θ̂2 − θ)] + MSE(θ̂2 ) = E[(θ̂ − θ̂2 )2 ] + MSE(θ̂2 ), where in the last line we used E[(θ̂ − θ̂2 )(θ̂2 − θ)] = since E[(δ(X) − ϕ(T (X)))(ϕ(T (X)) − θ)] = E[E[(δ(X) − ϕ(T (X)))(ϕ(T (X)) − θ)|T ]] = E[(ϕ(T (X)) − θ)E[(δ(X) − ϕ(T (X)))|T ]] = E[(ϕ(T (X)) − θ) · (E[δ(X)|T ] − ϕ(T (X)))] = 0, E[δ(X)|T ] = ϕ(T (X)). To show the last result, we have E[ϕ(T (X))] = E[E[δ(X)|T ]] = E[δ(X)] = θ by the law of iterated expectations. Example m)!))p (1 − X1 , ..., Xn be a random sample from Binomial(p, k), i.e. P {Xj = m} = (k!/(m!(k − k−m p) for any integer m ≥ 0. Suppose our parameter of interest is the probability of one success, i.e. θ = P {Xj = 1} = kp(1 − p)k−1 . m Let One possible estimator is 5 θ̂ = ∑n i=1 I(Xi = 1)/n. This estimator is unbiased, i.e. E[θ̂] = θ. Let us ˝nd a su°cient statistic. The joint density of the data is f (x1 , ..., xn ) = n ∏ (k!/(xi !(k − xi )!))pxi (1 − p)k−xi i=1 = Thus, T = ∑n i=1 Xi function(x1 , ..., xn )p ϕ = E[θ̂|T ] ϕ(t) ∑ (1 − p)nk− θ̂ = = = = = = = = . T. t, n n ∑ ∑ Xi = t] E[ I(Xi = 1)/n| n ∑ i=1 P {Xi = 1| i=1 = xi by considering its conditional expectation given denote this estimator. Then, for any nonnegative integer i=1 n ∑ Xj = t}/n j=1 P {X1 = 1| n ∑ Xj = t} j=1 ∑n P {X1 = 1, j=1 Xj = t} ∑n P { j=1 Xj = t} ∑n P {X1 = 1, j=2 Xj = t − 1} ∑n P { j=1 Xj = t} ∑n P {X1 = 1}P { j=2 Xj = t − 1} ∑n P { j=1 Xj = t} kp(1 − p)k−1 · (k(n − 1))!/((t − 1)!(k(n − 1) − (t − 1))!)pt−1 (1 − p)k(n−1)−(t−1) (kn)!/(t!(kn − t)!)pt (1 − p)kn−t k(k(n − 1))!/((t − 1)!(k(n − 1) − (t − 1))!) (kn)!/(t!(kn − t)!) k(k(n − 1))!(kn − t)!t (kn)!(kn − k + 1 − t)! where we used the fact that Binomial(k(n xi is su°cient. In fact, it is minimal su°cient. Using the Rao-Blackwell theorem, we can improve Let ∑ − 1), p). X1 is independent of (X2 , ..., Xn ), ∑n i=1 Xi ∼ Binomial(kn, p), ∑n and i=2 Xi ∼ So our new estimator is ∑n ∑n k(k(n − 1))!(kn − i=1 Xi )!( i=1 Xi ) ∑n θˆ2 = ϕ(X1 , ..., Xn ) = . (kn)!(kn − k + 1 − i=1 Xi )! By the theorem above, it is unbiased and at least as e°cient as θˆ. The procedure we just applied is sometimes informally referred to as Rao-Blackwellization. Note on implementation. t) One may say this is too complicated. We have derived analytically in order to calculate a new estimate θ̂2 = ϕ(T ) = ϕ( ∑n i=1 Xi ), ϕ(t) = E(θ̂| depend on ϕ(T ), p, that is evaluated in one point (the realized value of so we are free to choose any p (choosing 6 p close to T ). T /(kn) i=1 Xi = but in real life you may just do this with Monte-Carlo simulations. Note, that we do not need to calculate the whole need only ∑n ϕ(t) function we Note also, that the result does not will give faster calculations). Choose some • Draw p ∈ (0, 1). ∗ ∗ X1b , ..., Xnb For b = 1, ..., B as independent variables from Binomial (p, k ), if Repeat drawing samples until you get • Calculate Yb = The new estimator is 1 n repeat the following: ∑n i=1 θ̂2 ≈ 1 B ∑n i=1 ∑n i=1 ∗ Xib ̸= T , discard this sample. ∗ Xib = T. ∗ I(Xib = 1). ∑B b=1 Yb . The accuracy is better for larger number of simulation 7 B.