Lecture 9 Notes: Uniformly Most Powerful Tests

Lecture 9 Uniformly Most Powerful Tests 1 Some more on bootstrap in testing Example. Assume you have a random sample from some unknown distribution ˝nite moments. Denote 2 µ = EXi , σ = V ar(Xi ). of both arguments with a full rank derivative. against X = (X1 , ..., Xn ) with four 2 h(µ, σ ) be a twice-continuously-di˙erentiable function Let Assume we wish to test the hypothesis H0 : h(µ, σ 2 ) = 0 H1 : h(µ, σ 2 ) ̸= 0. One way to approach this problem is to de˝ne a new parameter consistent estimate ¯ s2 ). γ̂ = h(X, γ = h(µ, σ 2 ) for which we have a We can even prove asymptotic gaussianity via the delta-method: √ n(γ̂ − γ) ⇒ N (0, (∇h)′ V ∇h), where 2 √ ) ∂h(µ,σ 2 ) ′ n(X̄ − µ, s2 − σ 2 )′ ⇒ N (0, V ) and ∇h = ( ∂h(µ,σ , ∂σ2 ) . ∂µ estimate of the asymptotic variance of γ̂ and construct a regular So, in principle, one may get a natural t-statistic to conduct a test. Alternatively one can employ a non-parametric bootstrap in the way we discussed in the previous lecture by approximating √ ¯ ∗ , s∗2 ) − γ̂). Let us n(h(X ∫ ∗ denote all parameters and statistics related to the bootstraped distribution with stars, that is µ = xdF̂ = ∗ ∗ ¯ E X = X, etc. Several comments are needed: an unknown, ˝nite-sample distribution of a statistic z= √ n(γ̂ − 0) with z∗ = i • Notice a re-centering of the null hypothesis. We approximate the unknown distribution of empirical distribution function distribution of Xi is such that for the empirical distribution (almost s2 ; F̂ . However we know for sure that this null does not hold as the mean for it, the di˙erence is of order Op (1/n) this perspective the true value of parameter ¯, µ∗ = X and the variance is σ ∗2 = 1 n ∑ i (Xi ¯ 2 − X) and will be ignored here as too `smallish' to care). From γ for the empirical distribution is bootstrapped samples we should be testing the true null hypothesis • with The size is de˝ned under the null, so, we assume that the unknown H0 : h(µ, σ 2 ) = 0. F̂ Xi ∗ γ̂ (almost). So, for the ∗2 H0 : h(µ , σ ) = γ̂ . The justi˝cation of bootstrap validity here is based on the validity of asymptotic approximation, both for the statistic itself and for its bootstrapped version: √ n(γ̂ ∗ − γ̂) ⇒ N (0, (∇h∗ )′ V ∗ ∇h∗ ), 1 where √ n(X̄ ∗ − µ∗ , s∗2 − σ ∗2 )′ ⇒ N (0, V ∗ ) ∗ V = V + op (1) • and and ∗ ∇h∗ = ( ∂h(µ∂µ,σ ∗2 ) ∂h(µ∗ ,σ ∗2 ) ′ , ). ∂σ 2 And noticing that ∗ ∇h = ∇h + op (1). In the vast majority of cases the justi˝cation for the bootstrap comes from the validity of some asymptotic approximation to the test statistic, and so the asymptotic method can be used. There are only a couple of cases I know of when we do not know if a statistic has an asymptotic limit and, if so, what it is, but the bootstrap magically works. But there is a large number of cases (weak identi˝cation, moment inequality, boundary problem, post-selection inferences) when there are di°culties in carrying out the asymptotic approach, and when the bootstrap does not work either. • Another thing the bootstrap is implicitly doing here is bias correction. Remember our discussion about bias of the order • O(1/n) present in estimate γ̂ ? In the example above by using the bootstrap we avoided calculating the standard error for γ̂ . The question arises, if we are willing to calculate standard errors, would still using the bootstrap have any bene˝ts? The answer is yes. Imagine we construct a natural consistent estimator considered t-statistic: t= where The bootstrapped statistic Thus the distribution of t t We know that under the null is constructed in an analogous way (using re-centering) and is close to the distribution of ∗ t suggesting that the distance between the distributions of distance between the distribution of t for V and √ γ̂ − 0 n√ , (∇ĥ)′ V̂ ∇ĥ ¯ s2 . ∇ĥ is the corresponding derivative evaluated at X, ∗ V̂ t ⇒ N (0, 1). ∗ t ⇒ N (0, 1). in large samples. However, there are results t and t∗ are asymptotically smaller than the and the standard normal cdf. That is, the bootstrap provides better approximation, or so called second-order re˝nement. Statements like this one are based on Edgeworth's expansion, which in general is quite complicated to arrive at and we will not derive it. But it is a statement of the following sort: 1 1 P {t ≤ x} = Φ(x) + √ ϕ(x, F ) + O( ), n n here ϕ(x, F ) will depend on F only through several of the ˝rst moments of this distribution (I think in our case on ˝rst 6). Similarly, 1 1 P ∗ {t∗ ≤ x} = Φ(x) + √ ϕ(x, F̂ ) + Op ( ). n n One may show that ϕ(x, F̂ ) − ϕ(x, F ) = Op ( √1n ) (CLT and delta-method). Then 1 P {t ≤ x} − P ∗ {t∗ ≤ x} = Op ( ), n while 1 P {t ≤ x} − Φ(x) = Op ( √ ). n 2 2 Uniformly Most Powerful Test Let Θ = Θ0 ∪ Θ1 be a parameter space. Consider a parametric family test the null hypothesis, H0 , that θ ∈ Θ0 against the alternative, Ha , {f (x|θ), θ ∈ Θ}. that set. Then the probability that the null hypothesis is rejected is given by test based on test. Let C ′ C is of level θ ∈ Θ1 . if α ≥ supθ∈Θ0 β(θ). The restriction of β(·) α. Then the test based on C on Θ1 C G ′ be some critical Recall that the is known as the power of the by β ′ (θ). is more powerful than the test based on Any test which is more powerful than any other test in some class most powerful test in the class C Let / C}. β(θ) = Pθ {X ∈ be another critical set. Denote the power of the test based on tests are of level all α θ ∈ Θ1 . Suppose we want to G C Suppose that both ′ if β(θ) ≥ β ′ (θ) for will be called uniformly the (UMP test). As follows from the theorem below, the UMP test exists if both the null and the alternative are simple. Theorem 1 (Neyman-Pearson Lemma). Let f (x|θ) with Θ = {θ0 , θ1 } be some parametric family. Suppose we want to test the null hypothesis, H0 , that θ = θ0 against the alternative hypothesis, Ha , that θ = θ1 . Assume that some critical set C satis˝es (1) x ∈ C if kf (x|θ0 ) > f (x|θ1 ) and (2) x ∈/ C if kf (x|θ0 ) < f (x|θ1 ) where / C). Then the test based on C is the UMP among all tests of level α. k ≥ 0 is chosen so that α = Pθ0 (X ∈ In addition, any UMP test of level α satis˝es (1) and (2). Proof. Denote ϕ(x) = I(x ∈ / C), that the test based on α. of level x. value Let ϕ̃(x) C i.e. ϕ(x) = 1 if x∈ /C and 0 otherwise. Thus, ϕ(x) denotes the probability x. Consider any other test rejects the null hypothesis upon observing data of value denote the probability that this test rejects the null hypothesis upon observing data Since this test is of level α, ∫ ˜ 0) = β(θ where is β̃(θ) ˜ ϕ(x)f (x|θ0 )dx ≤ α, denotes the probability that this test rejects the null hypothesis when the true parameter value θ. Note that (ϕ(x) − ϕ̃(x))(f (x|θ1 ) − kf (x|θ0 )) ≥ 0 for any then x. f (x|θ1 ) − kf (x|θ0 ) ≥ 0, Indeed, if ϕ(x) = 0 and ϕ(x) − ϕ̃(x) ≤ 0. then ϕ(x) = 1 and ϕ(x) − ϕ̃(x) ≥ 0. If f (x|θ1 ) − kf (x|θ0 ) < 0, So, ∫ (ϕ(x) − ϕ̃(x))(f (x|θ1 ) − kf (x|θ0 ))dx = β(θ1 ) − β̃(θ1 ) − k(β(θ0 ) − β̃(θ0 )), 0≤ where β(θ) denotes the probability that the test based on parameter value is θ. C rejects the null hypothesis when the true Therefore, β(θ1 ) − β̃(θ1 ) ≥ k(β(θ0 ) − β̃(θ0 )) ≥ k(α − α) = 0, since β̃(θ0 ) ≤ α and β(θ0 ) = α. So the test based on C is more powerful than any other test of level it is the UMP which proves the ˝rst statement of the theorem. 3 α. So ϕ̃(·) If is also a UMP among all tests of level β̃(θ0 ) ≥ α. On the other hand, β̃(θ0 ) ≤ α, α, then β̃(θ1 ) = β(θ1 ). since this test is of level α. So, k(α − β̃(θ0 )) ≤ 0. We conclude that Therefore β̃(θ0 ) = α. It ∫ follows that (ϕ(x) − ϕ̃(x))(f (x|θ1 ) − kf (x|θ0 ))dx = 0. Since the integrand is nonnegative for all x, (ϕ(x) − ϕ̃(x))(f (x|θ1 ) − kf (x|θ0 )) = 0. Thus, ϕ̃(x) = ϕ(x) whenever f (x|θ1 ) − kf (x|θ0 ) ̸= 0. So, ϕ̃(·) also satis˝es conditions (1) and (2). Recall from the proof of the factorization theorem in lecture 4 that if f (x|θ) = g(T (x)|θ)h(x) the critical set C g(·) where denotes pdf of of the UMP test satis˝es (1) T (X). x ∈ C T (X) is a su°cient statistic, then So, in terms of the pdf of su°cient statistics, if kg(T (x)|θ0 ) > g(T (x)|θ1 ) and (2) x ∈ / C if kg(T (x)|θ0 ) < g(T (x)|θ1 ). Example Let X1 , ..., Xn be a random sample from the we want to test the null hypothesis, H0 , µ = θ0 that Without loss of generality we can assume that this example is given by Xn = ∑n i=1 Xi /n. N (µ, σ 2 ) distribution with known against the alternative hypothesis, θ0 > θ 1 . Ha , σ2 . that Suppose µ = θ1 . We have already seen that the su°cient statistic in We know that X n ∼ N (µ, σ 2 /n). So, g(t|θ) = C exp{−(n/(2σ 2 ))(t − θ)2 }. From the Neyman-Pearson lemma, the UMP test among all tests of level α accepts the null hypothesis if and only if kg(X n |θ0 ) > g(X n |θ1 ) or, equivalently, k exp{(n/(σ 2 ))X n (θ0 − θ1 ) − (n/(2σ 2 ))(θ02 − θ12 )} > 1. θ0 > θ1 , the test accepts the null hypothesis if and only if X n > k̃ where k̃ is such that Pθ0 (X ≤ k̃) = α. √ k̃ = θ0 + σzα / n where zα denotes an α-quantile of the standard normal distribution. Since So, 3 UMP tests with composite hypotheses The idea of the Neyman-Pearson lemma is to consider an optimization problem subject to ∫ ϕ(x)f (x|θ0 )dx ≤ α and 0 ≤ ϕ ≤ 1. ∫ ϕ(x)f (x|θ1 )dx → max The solution to this problem gives the UMP test of level α. This problem looks like maximization of utility given some budget constraint. We want to choose the most valuable items with the lowest price. In some special cases we can extend this idea to the case of complex hypotheses. Suppose we want to test the null hypothesis, Ha , that θ > θ0 . De˝nition 2. Then the UMP test exists if A family f (x|θ) H0 , that θ ≤ θ0 against the alternative hypothesis, satis˝es the monotone likelihood ratio property. f (x|θ) with θ ∈ R satis˝es the monotone likelihood ratio if there exists some function 4 T (x) such that for any θ < θ′ , Pθ′ (x)/Pθ (x) is a nondecreasing function of Theorem 3. Let x depends on T (x) only through and, moreover, Pθ′ (x)/Pθ (x) T (x). f (x|θ) with θ ∈ R be some parametric family that satis˝es the monotone likelihood ratio with function T (x). Suppose we want to test the null hypothesis, H0 , that θ ≥ θ0 against the alternative hypothesis, Ha , that θ < θ0 . Then a UMP test of level α exists. It is given by ϕ(X) = 1 if T (X) > C , ϕ(X) = γ if T (X) = C , and 0 otherwise for some constants c and γ such that Eθ0 [ϕ(X)] = α. In addition, the power of this test β(θ) = Eθ [ϕ(X)] for θ > θ0 is strictly decreasing in θ. Proof. Choose a simple alternative accepts the null hypothesis if θ1 < θ0 . f (X|θ0 )/f (X|θ1 ) > k and rejects it if the test rejects with probability, say, Eθ0 [ϕ(X)] = α. α for the null hypothesis to show that the same test is UMP of level α, enough to show that this test is of level α i.e. Since there always exists a test of level α independently of the data), argument yields β(θ1 ) ≤ β(θ2 ) C θ = θ0 θ ≥ θ0 α and γ against θ1 By the monotone T (X) < C . When should be chosen such that for any other alternative against the alternative against the alternative θ2 < θ 0 θ < θ0 . θ < θ0 , as Thus, it will be supθ≥θ0 β(θ) ≤ α. α with power θ = θ1 θ1 ≥ θ2 for any statement of the theorem follows from for the null β(θ0 ) ≤ β(θ1 ). UMP (of some level) for the null hypothesis and rejects it if The constants Now, note that the same test will be UMP of level well. So, this test is UMP of level probability γ. f (X|θ0 )/f (X|θ1 ) < k . T (X) > C likelihood ratio, this test accepts the null hypothesis if T (X) = C , θ0 By the Neyman-Pearson lemma, the UMP test of α (this test rejects the null hypothesis with Since the test based on against the alternative θ = θ2 T (x), C , for any and θ2 < θ1 , which is the second statement of the theorem. supθ≥θ0 β(θ) = β(θ0 ) = α since β(θ) is decreasing in γ is also the same The ˝rst θ. In many cases a UMP test does not exist. Below is an example of such a situation. Example Let X1 , ..., Xn be a random sample from an want to test the null hypothesis, θ1 < θ0 . Consider some if and only if for any Indeed, β(θ) < α always exists a test of level 3.1 that θ = θ0 α distribution with known against the alternative hypothesis, α The only UMP test of level √ X n < θ0 + σzα / n θ > θ0 . H0 , N (θ, σ 2 ) of θ = θ0 against θ = θ1 σ2 . Ha , Suppose we that θ ̸= θ0 . rejects the null hypothesis as we have already seen. But this test has little power in our situation for all θ > θ0 . with power So, this test cannot be UMP in this situation, since there α. Unbiased tests α Since there are no UMP tests among all tests of level in many situations, the question arises whether we can ˝nd UMP tests in some smaller, but still reasonably large, classes of tests. The de˝nition below gives a property that reasonable tests should have. De˝nition 4. some Any test of the null hypothesis α ∈ [0, 1], β(θ) ≤ α Let Θ=R for all θ ∈ Θ0 and θ ∈ Θ0 β(θ) ≥ α against the alternative for all θ ∈ Θ1 θ ∈ Θ1 . be a parameter space. Suppose we want to test the null hypothesis, the alternative hypothesis, Ha , that θ ̸= θ0 . is called unbiased if for H0 , that θ = θ0 against Consider a test which rejects the null hypothesis with probability 5 ϕ(x) upon observed data value x. As before, denote ′ β (θ0 ) = 0. then for any unbiased test, we necessarily have there exists a point θ θ0 in the neighborhood of β(θ) = such that ∫ ϕ(x)f (x|θ)dx. 4 β(θ) is di˙erentiable in θ, Indeed, if this condition does not hold, then β(θ) ≤ β(θ0 ) situations, there exists a UMP test among all unbiased tests of level among all tests of level If α by de˝nition of derivative. In some even though there are no UMP tests α. Likelihood Ratio Test Suppose we want to test the null hypothesis, θ ∈ Θ1 . Denote De˝nition 5. Θ = Θ 0 ∪ Θ1 . Let L(θ|x) H0 , that θ ∈ Θ0 against the alternative hypothesis, Ha , that denote likelihood function. Then we have A Likelihood ratio test (LRT) statistic is λ(x) = By de˝nition, 0 ≤ λ(x) ≤ 1. alternative hypothesis Θ1 supθ∈Θ0 L(θ|x) . supθ∈Θ L(θ|x) Small values of the LRT statistic imply that there is a value θ in the which gives much greater likelihood than all values in the null hypothesis. likelihood ratio tests reject the null hypothesis if and only if λ(x) ≤ c for some c. So, As usual, the constant c is chosen according to the desired level of the test. Let θ̂r = arg maxθ∈Θ0 L(θ|x) be the ML estimator of the restricted model. Let θ̂ur = arg maxθ∈Θ L(θ|x) be the ML estimator of the unrestricted model. Then an equivalent way to de˝ne LRT statistic is to set λ(x) = Example Let X1 , ..., Xn L(θ̂r |x) L(θ̂ur |x) be a random sample from an H0 , that θ = θ0 against the ˆ ˆ θur = θM LE = X n . So, the LRT statistic is null hypothesis, λ(x) = . N (θ, 1) distribution. Suppose we want to test the alternative hypothesis, Ha , that θ = ̸ θ0 . Then θ̂r = θ0 and L(θ0 |x) L(θ̂M LE |x) ∑n (2π)−n/2 exp{−(1/2) i=1 (Xi − θ0 )2 } = ∑n (2π)−n/2 exp{−(1/2) i=1 (Xi − X n )2 } n ∑ = exp{−(1/2) [(Xi − X n + X n − θ0 )2 − (Xi − X n )2 ]} i=1 = exp{−(n/2)(X n − θ0 )2 } So, the LRT rejects the null hypothesis if and only if |X n −θ0 | > c. null hypothesis if and only if or √ X n ∼ N (θ0 , 1/ n). √ X n − θ0 > z1−α/2 / n 6 Speci˝cally, the LRT of level √ X n − θ0 < zα/2 / n α rejects the since under the null hypothesis,