Fundamentals of Statistical Hypothesis Testing

Lecture Note 10 ∗ Hypothesis Testing MIT 14.30 Spring 2006 Herman Bennett Hypothesis testing: Given a random sample from a certain population, Is the sample evidence enough to disregard a particular belief about that population? (E.g.: the value of a parameter.) 25 Definitions 25.1 Hypothesis Testing A (parametric) hypothesis is a statement about one or more population parameters 1 . This hypothesis can be tested using a hypothesis test. A hypothesis test consists of: 1. Two complementary hypotheses: the null hypothesis and the alternative hypothesis, denoted H0 and H1 respectively. 2. A decision rule that specifies for which sample values the null hypothesis is not rejected (‘accepted’), and for which sample values the null hypothesis is rejected in favor of the alternative hypothesis. The set of sample values for which H0 is rejected is called the rejection or critical region. The complement of the critical region is called the acceptance region (where H0 is accepted). ∗ Caution: These notes are not necessarily self-explanatory notes. They are to be used as a complement to (and not as a substitute for) the lectures. 1 An example of a nonparametric hypothesis would be to make an statement about the distribution of the RV X. E.g.: X ∼ N ( ). 1 Herman Bennett LN10—MIT 14.30 Spring 06 A hypothesis that if true completely specifies the population distribution, is called a simple hypothesis; one that does not is called a composite hypothesis. 25.2 General Setting for Hypothesis Testing Let X1 , ..., Xn be a random sample from a population with pmf/pdf f (x|θ). Define the following hypothesis test about the parameter θ ∈ Ω,: H0 : θ ∈ Ω0 H1 : θ ∈ Ω1 , where Ω0 ∪ Ω1 = Ω and Ω0 ∩ Ω1 = ∅. H0 is rejected if the random sample X1 , ..., Xn lies in the n−dimensional space C. The space C is the critical region defined in terms of x, the n−dimensional vector that contains the random sample. The two complementary hypotheses, H0 and H1 , usually take one of the following five structures: 1.- Singleton H0 and singleton H1 : H0 : θ = θ0 H1 : θ = θ1 (76) 2.- Singleton H0 and composite 2-sided H1 : H0 : θ = θ0 H1 : θ �= θ0 (77) 3.- Singleton H0 and composite 1-sided H1 : H0 : θ = θ0 H1 : θ < θ0 (or θ > θ0 ) (78) 4.- Composite 1-sided H0 and composite 1-sided H1 : H0 : θ ≤ θ0 (or θ ≥ θ0 ) H1 : θ > θ0 (or θ < θ0 ) (79) 5.- Composite 2-sidedH0 and composite 2-sided H1 : H0 : θn1 ≤ θ ≤ θn2 H1 : θ < θn1 and θ > θn2 (80) 2 Herman Bennett 25.3 LN10—MIT 14.30 Spring 06 Type of Errors in Hypothesis Testing A type I error occurs when H0 is rejected when indeed is true. The probability that this error occurs, denoted αθ , is defined as follows: � � � αθ = P (type I error) = P rejecting H0 � θ ∈ Ω0 (81) A type II error occurs when H0 is not rejected when indeed H1 is true. The probability that this error occurs, denoted βθ , is defined as follows: � � � βθ = P (type II error) = P accepting H0 � θ ∈ Ω1 (82) • Wrap up: 25.3.1 Level of Significance and Optimal Tests The level of significance, or size, of a hypothesis test is the highest type I error. The level of significance is denote by α. 2 Formally: α = sup αθ . (83) θ∈Ω0 If Ω0 is singleton: α = αθ . For a given pair of null and alternative hypotheses, and a given level of α, an optimal hypothesis test is defined as a test that minimizes βθ ∀θ. Note that optimal tests do not exist for many hypothesis test structures (more on this later). 2 There is a technical difference between the level and the size, which in practice becomes only relevant in complicated testing situations. For the purpose of this course we will use them interchangeably. 3 Herman Bennett LN10—MIT 14.30 Spring 06 Example 25.1. Assume a random sample of size n from a normal population N (µ, 4). ¯ to construct a hypothesis test with H0 : µ = 0, H1 : µ = 1, and i) Use the statistic X a decision rule of the form “reject H0 when x¯ > k”, such that the probability of type I error is 5%. ii) Compute the probability of type II error. What is the size of the test? iii) What happens to α and β as k ↑ or k ↓? Which is the trade-off? iv) What happens if the sample size n increases? v) How would the answers change if we redefine the hypotheses � 0. as H0 : µ = 0 and H1 : µ = 4 Herman Bennett LN10—MIT 14.30 Spring 06 • Be careful when interpreting the results of a hypothesis test: accepting v/s failing to reject H0 . 5 Herman Bennett 25.4 LN10—MIT 14.30 Spring 06 Power Function Let’s denote the characteristics of a hypothesis test (the null hypothesis, the alternative hypothesis, and the decision rule) by the letter δ. The power function of a hypothesis test δ is the probability of rejecting H0 given that the true value of the parameter is θ ∈ Ω. � � � π(θ|δ) = P rejecting H0 � θ ∈ Ω = P (X ∈ C |θ) for all θ ∈ Ω. (84) Thus, π(θ|δ) = αθ (δ) if θ ∈ Ω0 1 − π(θ|δ) = βθ (δ) if θ ∈ Ω1 (85) Example 25.2. Ideal power function...a=? b=? � a if θ ∈ Ω0 π(θ|δ) = b if θ ∈ Ω1 . • If Ω0 is singleton: α = π(θ|δ). • For a given pair of null and alternative hypotheses, and a given level of α, an optimal hypothesis test, δ ∗ , is a test that minimizes β(δ) for all θ ∈ Ω1 . In other words, δ ∗ maxi­ mizes the power function for all θ ∈ Ω1 . Example 25.3. Assume a random sample of size n from a U[0, θ], where θ is unknown. Suppose the following hypothesis test δ: H0 : 3≤θ≤4 H1 : θ < 3 or θ > 4 Decision rule: Accept H0 if θ̂M LE ∈ [2.9, 4.1], and reject H0 otherwise. Find the power function π(θ|δ) (note: ∀θ). Which is the size of this test? 6 Herman Bennett LN10—MIT 14.30 Spring 06 7 Herman Bennett 25.5 LN10—MIT 14.30 Spring 06 p-value The p-value describes the minimum level of significance α that would have implied, given the particular realization of the random sample (x), a rejection of H0 . Thus, the p-value, as well as whether H0 is rejected or not, are ex-post calculations. 26 26.1 (Four) Most Common Hypothesis Tests Structures Likelihood Ratio Test (LRT): H0 : θ = θ 0 H1 : θ = θ 1 Decision Rule form: “Reject H0 if f1 (x)/f0 (x) > k”. (86) Where k > 0 is a constant chosen according to the size of the test (α0 ), such that P (f1 (x)/f0 (x) > k |θ0 ) = α0 . The statistic f1 (x)/f0 (x) is given by: fi (x) = f (x1 , x2 , ..., xn |θi ) = f (x1 |θi )f (x2 |θi )...f (xn |θi ) (iid sample) (87) • The ratio f1 (x)/f0 (x) is called the likelihood ratio of the sample. Optimality of the LRT Minimize the probability of type II error given the probability of type I error: min β ; δ given α0 . (α0 is the size imposed on the test.) (Neyman-Pearson lemma) Let δ ∗ be a hypothesis test where H0 and H1 are simple hy­ potheses, and where H0 is accepted if kf0 (x) > f1 (x) (k > 0). Otherwise, H1 is accepted, except if kf0 (x) = f1 (x) where both H0 and H1 may be accepted. Then, for every other hypothesis test δ: β(δ) < β(δ ∗ ) ←→ α(δ) > α(δ ∗ ) 8 (88) Herman Bennett LN10—MIT 14.30 Spring 06 Example 26.1. Assume a random sample of size n = 20 from a Bernoulli distribution, where p is unknown. Suppose the following hypotheses: H0 : p = 0.2 H1 : p = 0.4 Find the optimal test procedure δ ∗ with α(δ ∗ ) = 0.05. 9 Herman Bennett LN10—MIT 14.30 Spring 06 • For the case of a normal random sample, the hypothesis test (86) implies the following decision rule: - “Reject H0 if x̄ > k � ” when θ1 > θ0 . - “Reject H0 if x̄ < k � ” when θ1 < θ0 . (For the derivation of this result check DeGroot and Schervish (2002) page 465.) 26.2 One-sided Test: H0 : µ = µ0 H1 : µ > µ0 or H1 : µ < µ 0 Decision Rule form: “Reject H0 if x¯ > c” or “Reject H0 if x¯ < c”. (89) ¯ > c|µ0 ) = α0 Where c is a constant chosen according to the size of the test (α0 ), such that P (X ¯ < c|µ0 ) = α0 . or P (X Optimality of One-sided tests What does it mean to be optimal in these cases? Should we use non optimal tests? A generalization of the relevant optimality results for these cases is out of the scope of this course. 3 However, we can handily state the following result: - Assume a random sample from a binomial or normal distribution, a null and an alternative hypotheses given by (89), and a level of significance α0 . Then, the optimal test δ ∗ , which minimizes β(δ) for all θ ∈ Ω1 , is given by test (89). • The decision rule of test (89) is widely use for cases (78) and (79), even if it is not an optimal test. 3 An excellent reference is DeGroot and Schervish (2002) Ch. 8.3. 10 Herman Bennett LN10—MIT 14.30 Spring 06 Example 26.2. Assume a random sample of size n = 100 from a N(µ, 1), where µ is unknown and x̄ = 1.13. Suppose the following hypotheses: H0 : µ = 1 H1 : µ > 1 Construct a one-sided hypothesis test of size 0.05. Test the null hypothesis and find the p-value. 26.3 Two-sided Test: H 0 : µ = µ0 H1 : µ = � µ0 Decision Rule form: “Reject H0 if x̄ is outside the interval [c1 , c2 ]. (90) Where c1 and c2 are a constants chosen according to the size of the test (α0 ), such that P (x̄ ∈ / [c1 , c2 ] |µ0 ) = α0 . Usually, hypothesis tests are constructed in a symmetric way, which ¯ < c1 |µ0 ) = α0 /2 and P (X ¯ > c2 |µ0 ) = α0 /2. means that P (X 11 Herman Bennett LN10—MIT 14.30 Spring 06 Optimality of Two-sided tests Unfortunately, there is no robust result regarding optimality in this case. No test pro­ cedure δ ∗ will minimize βθ (δ) for all θ ∈ Ω1 . However, the optimality results for the 1-sided hypothesis test case suggest that a reasonable decision rule for the hypotheses described in (90), could be given by the decision rule of test (90). 4 • The decision rule of test (90) is widely use for cases (77) and (80). Example 26.3. A candle producer company claims that their candles last for 60 minutes on average. One consumer, curious about this claim, bought 40 candles and tested them. He found that on average they last for 65.22 minutes. With the data collected he also computed the statistic s2 = 225. Can the consumer say, with 99% of significance, that the company is wrong in its claim? (Assume the sample is iid.) Also, compute the p-value and the limiting n such that H0 is rejected at α = 0.01 (assume s2 and x̄ keep their value). 4 In fact, this is what most researchers do. 12 Herman Bennett 26.4 LN10—MIT 14.30 Spring 06 Generalized Likelihood Ratio Test (GLRT): H0 and H1 : any composite or simple hypothesis Decision Rule form: “Reject H0 if W > k”. (91) Where k > 0 is a constant chosen according to the size of the test (α0 ), such that P (W > k |H0 ) = α0 . The statistic W is given by: W = supθ∈Ω1 L(θ1 , ..., θk |x1 , ..., xn ) supθ∈Ω1 f (x|θ ∈ Ω1 ) = . supθ∈Ω0 L(θ1 , ..., θk |x1 , ..., xn ) supθ∈Ω0 f (x|θ ∈ Ω0 ) (92) • As with previous tests, the constant k will depend on the distribution of the statistic W and α0 . If computing the distribution of W becomes a nightmare, it is possible to use an equivalent definition of the GLRT, (93), which has a known limiting distribution. T = supθ∈Ω0 L(θ1 , ..., θk |x1 , ..., xn ) supθ∈Ω0 f (x|θ ∈ Ω0 ) = supθ∈Ω L(θ1 , ..., θk |x1 , ..., xn ) supθ∈Ω f (x|θ ∈ Ω) (93) Decision Rule form: “Reject H0 if T < d”; where d > 0 is a constant chosen according to the test size (α0 ), such that P (T < d|H0 ) = α0 . The limiting distribution of −2lnT is known: −2lnT n→∞ ∼ χ2(r) ; (94) where r is the # of free parameters in Ω minus the # of free parameters in Ω0 . Reject H0 if −2lnT > χ2(r),α . 5 If it is possible to compute directly the distribution of W or T , then is better to use that distribution instead of the limiting χ2 . Optimality of GLRT The GLRT is a generalization of the LRT; it works for any case where either H0 or/and H1 are composite hypotheses. However, GLRT is not necessarily optimal, as the LRT is. In particular, it will depend on the case at hand (further details on this issue are out of the scope of this course 6 ). 5 6 The technical result says that the distribution is a χ2(r) with degrees of freedom r = dimΩ − dimΩ0 . An excellent reference is DeGroot and Schervish (2002) Ch. 8. 13 Herman Bennett 27 LN10—MIT 14.30 Spring 06 Hypothesis Testing Based on Two Normal Samples Example 27.1. Assume 2 random samples: 2 Xi ∼ N (µX , σX ) of sample size nX Yi ∼ N (µY , σY2 ) of sample size nY , and the following hypotheses to be tested: a) H0 : µX = µY 2 b) H0 : σX = σY2 H1 : µX �= µY 2 H1 : σX � σY2 = For each case, construct a hypothesis test of size 95%. In part a) assume that you know 2 and σY2 . σX That’s all Folks! 14