18.05 S22 Class 18: Null Hypothesis Significance Testing II

Null Hypothesis Significance Testing II Class 18, 18.05 1 Learning Goals 1. Be able to list the steps common to all null hypothesis significance tests. 2. Be able to define and compute the probability of Type I and Type II errors. 3. Be able to look up and apply one- and two-sample 𝑡-tests. 2 Introduction We continue our study of significance tests. In these notes we will introduce two new tests: one-sample 𝑡-tests and two-sample 𝑡-tests. You should pay careful attention to the fact that every test makes some assumptions about the data – often that is drawn from a normal distribution. You should also notice that all the tests follow the same pattern. It is just the computation of the test statistic and the type of the null distribution that changes. 3 Review 3.1 Setting up and running a significance test There is a fairly standard set of steps one takes to set up and run a null hypothesis significance test. 1. Design an experiment to collect data and choose a test statistic 𝑥 to be computed from the data. The key requirement here is to know the null distribution 𝜙(𝑥|𝐻0 ). To compute power, one must also know the alternative distribution 𝜙(𝑥|𝐻𝐴 ). 2. Decide if the test is one or two-sided based on 𝐻𝐴 and the form of the null distribution. 3. Choose a significance level 𝛼 for rejecting the null hypothesis. 4. Decide how much data you need to collect to achieve the desired power for the test. 5. Run the experiment to collect data 𝑥1 , 𝑥2 , … , 𝑥𝑛 . 6. Compute the test statistic 𝑥. 7. Compute the 𝑝-value corresponding to 𝑥 using the null distribution. 8. If 𝑝 < 𝛼, reject the null hypothesis in favor of the alternative hypothesis. 1 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 2 Notes. 1. Rather than choosing a significance level, you could instead choose a rejection region and reject 𝐻0 if 𝑥 falls in this region. The corresponding significance level is then the probability, assuming 𝐻0 , that 𝑥 falls in the rejection region. 2. The null hypothesis is often the ‘cautious hypothesis’. The lower we set the significance level, the more “evidence” we will require before rejecting our cautious hypothesis in favor of a more sensational alternative. It is standard practice to publish the 𝑝 value itself so that others may draw their own conclusions. 3. A key point of confusion: A significance level of 0.05 does not mean the test only makes mistakes 5% of the time. It means that if the null hypothesis is true, then the probability the test will mistakenly reject it is 5%. The power of the test measures the accuracy of the test when the alternative hypothesis is true. Namely, the power of the test is the probability of rejecting the null hypothesis if the alternative hypothesis is true. Therefore the probability of falsely failing to reject the null hypothesis is 1 minus the power. 4. Another key point of confusion: We use 𝑝-values, but conceptually the 𝑝-value is just a computational trick. After choosing a test statistic, the conceptual order is: first pick a significance level, then use this to define the rejection region. We reject the null hypothesis if the test statistic is in the rejection region. All the 𝑝-value does is tell us in one computation whether or not the test stastic is in the rejection region. Errors. We can summarize these two types of errors and their probabilities as follows: Type I error Type II error = = rejecting 𝐻0 when 𝐻0 is true. failing to reject 𝐻0 when 𝐻𝐴 is true. P(type I error) = = = probability of falsely rejecting 𝐻0 P(test statistic is in the rejection region | 𝐻0 ) significance level of the test P(type II error) = = = probability of falsely not rejecting 𝐻0 P(test statistic is in the acceptance region | 𝐻𝐴 ) 1 - power. Helpful analogies. In terms of medical testing for a disease: a Type I error is a false positive and a Type II error is a false negative. In a jury trial, a Type I error is convicting an innocent defendant and a Type II error is acquitting a guilty defendant. 3.2 Power We discussed power in the Class 17 notes. Power is the probabilitiy of correctly rejecting the null hypothesis. It depends on the alternative hypothesis 𝐻𝐴 being considered. The ideal test has power equal to 1.0 and significance equal to 0.0. Of course, in general, this is impossible. And we want to find some compromise where power is high and signficance is low. In symbols: power = 𝑃 (data is in the rejection region | 𝐻𝐴 ). 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 3 Compare this with: signficance = 𝑃 (data is in the rejection region | 𝐻0 ). 4 Understanding a significance test Questions to ask: 1. How did they collect data? What is the experimental setup? 2. What are the null and alternative hypotheses? 3. What type of significance test was used? Does the data match the criteria needed to use this type of test? How robust is the test to deviations from these criteria? 4. For example, some tests comparing two groups of data assume that the groups are drawn from distributions that have the same variance. This needs to be verified before applying the test. Often the check is done using another significance test designed to compare the variances of two groups of data. 5. How is the 𝑝-value computed? A significance test comes with a test statistic and a null distribution. In most tests the 𝑝-value is 𝑝 = 𝑃 (data at least as extreme as what we got | 𝐻0 ) What does ‘data at least as extreme as the data we saw’ mean? For example, is the test one or two-sided? 6. What is the significance level 𝛼 for this test? If 𝑝 < 𝛼 then the experimenter will reject 𝐻0 in favor of 𝐻𝐴 . 7. What is the power of the test? 5 𝑡 tests Many significance tests assume that the data are drawn from a normal distribution, so before using such a test you should examine the data to see if the normality assumption is reasonable. We will describe how to do this in more detail later, but plotting a histogram is a good start. Like the 𝑧-test, the one-sample and two-sample 𝑡-tests that we consider below start from this normality assumption. We don’t expect you to memorize all the computational details of these tests and those to follow. In real life, you have access to textbooks, google, and wikipedia; on the exam, you’ll have your notecard. Instead, you should be able to identify when a 𝑡-test is appropriate and apply this test after looking up the details and using a table or software like R. 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 5.1 4 𝑧-test Let’s first review the 𝑧-test. • Data: we assume 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ 𝑁 (𝜇, 𝜎2 ), where 𝜇 is unknown and 𝜎 is known. • Null hypothesis: 𝜇 = 𝜇0 for some specific value 𝜇0 • Test statistic: 𝑧= 𝑥 − 𝜇0 √ = standardized mean 𝜎/ 𝑛 • Null distribution: 𝜙(𝑧 | 𝐻0 ) is the pdf of 𝑍 ∼ 𝑁 (0, 1) • One-sided 𝑝-value (right side): 𝑝 = 𝑃 (𝑍 ≥ 𝑧 | 𝐻0 ) One-sided 𝑝-value (left side): 𝑝 = 𝑃 (𝑍 ≤ 𝑧 | 𝐻0 ) 2𝑃 (𝑍 ≥ 𝑧) if 𝑧 > 0 Two-sided 𝑝-value: 𝑝 = { . 2𝑃 (𝑍 ≤ 𝑧) if 𝑧 < 0. Because of the symmetry of the distribution around 0, we can also write this as 𝑝 = 𝑃 (|𝑍| ≥ |𝑧|). See Example 1b for the rationale for this. Example 1. Suppose that we have data that follows a normal distribution of unknown mean 𝜇 and known variance 4. Let the null hypothesis 𝐻0 be that 𝜇 = 2. Let the alternative hypothesis 𝐻𝐴 be that 𝜇 > 2. Suppose we collect the following data: 3, 2, 5, 7, 1 At a significance level of 𝛼 = 0.05, should we reject the null hypothesis? Solution: There are 5 data points with average 𝑥 = 3.6. Because we have normal data with a known variance we should use a 𝑧 test. Our 𝑧 statistic is 𝑧 = 3.6 − 2 𝑥 − 𝜇0 √ = √ = 1.79 𝜎/ 𝑛 2/ 5 Our test is one-sided because the alternative hypothesis is one-sided. So (using R) our 𝑝-value is 𝑝 = 𝑃 (𝑍 > 𝑧) = 𝑃 (𝑍 > 1.79) = 0.037 Since 𝑝 < 𝛼 = 0.05, we reject the null hypothesis in favor of the alternative hypothesis that 𝜇 > 2. We can visualize the test as follows: 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 Rejection region = shaded orange (starts at 𝑞0.95 = 𝑧0.05 = 1.64) 𝛼 = shaded orange area = 0.05 𝑧 = statistic = black dot = 1.79 𝑝 = blue stripe area = 0.037 𝜙(𝑧|𝐻0 ) ∼ N(0, 1) -1 5 0 non-reject 𝐻0 1 1.64 1.79 reject 𝐻0 𝑥 Example 1b. Repeat Example 1 as a two-sided test, i.e. with 𝐻𝐴 being 𝜇 ≠ 2. Solution: Let’s do the test and then we’ll explain the rationale behind the computation of the 𝑝-value. Since 𝑧 > 0, 𝑝 = 2𝑃 (𝑍 > 𝑧) = 0.074. Since, 𝑝 > 𝛼 = 0.05, the data does not support rejecting the null hypothesis in favor of 𝐻𝐴 . Reason for the factor of 2 in the computation of 𝑝 The reason is essentially arithmetic. Remember, the purpose of the 𝑝-value is that 𝑝 ≤ 𝛼 indicates that the test statistic is in the rejection region. The picture below illustrates the following. For a two-sided test, each side of the rejection region has probability 𝛼/2. So, if the test statistic is on the right, then it is in the rejection region if 𝑃 (𝑍 > 𝑧) ≤ 𝛼/2, i.e. if 𝑝 = 2𝑃 (𝑍 > 𝑧) ≤ 𝛼 Left rejection region = shaded orange (starts at 𝑞0.025 = 𝑧0.925 = −1.96) 𝛼/2 = left shaded orange area = 0.025 −𝑧 = black dot = 1.79 𝑝/2 = left blue stripe area = 0.037 −1.79 −1.96 -1 reject 𝐻0 5.2 Right rejection region = shaded orange (starts at 𝑞0.975 = 𝑧0.025 = 1.96) 𝛼/2 = right shaded orange area = 0.025 𝑧 = statistic = black dot = 1.79 𝑝/2 = right blue stripe area = 0.037 𝜙(𝑧|𝐻0 ) ∼ N(0, 1) 0 non-reject 𝐻0 1 1.79 𝑥 1.96 reject 𝐻0 The Student 𝑡 distribution ‘Student’ is the pseudonym used by the William Gosset who first described this test and distribution. See https://en.wikipedia.org/wiki/Student's_t-test The 𝑡-distribution is symmetric and bell-shaped like the normal distribution. It has a parameter 𝑑𝑓 which stands for degrees of freedom. For 𝑑𝑓 small the 𝑡-distribution has more probability in its tails than the standard normal distribution. As 𝑑𝑓 increases 𝑡(𝑑𝑓) becomes more and more like the standard normal distribution. 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 6 0.4 Here is a simple applet that shows 𝑡(𝑑𝑓) and compares it to the standard normal distribution: https://mathlets.org/mathlets/t-distribution/ 0.0 0.2 N(0,1) m=8 m=4 m=2 −4 −2 0 2 4 t densities for m degrees of freedom As degrees of freedom increases the t-distribution becomes normal 5.3 R As usual in R, the functions pt, dt, qt, rt correspond to cdf, pdf, quantiles, and random sampling for a 𝑡 distribution. Remember that you can type ?dt in RStudio to view the help file specifying the parameters of dt. For example, pt(1.65,3) computes the probability that 𝑥 is less than or equal 1.65 given that 𝑥 is sampled from the 𝑡 distribution with 3 degrees of freedom, i.e. 𝑃 (𝑥 ≤ 1.65) given that 𝑥 ∼ 𝑡(3)). 5.4 One sample 𝑡-test For the 𝑧-test, we assumed that the variance of the underlying distribution of the data was known. However, it is often the case that we don’t know 𝜎 and therefore we must estimate it from the data. In these cases, we use a one sample 𝑡-test instead of a 𝑧-test and the studentized mean in place of the standardized mean • Data: we assume 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ 𝑁 (𝜇, 𝜎2 ), where both 𝜇 and 𝜎 are unknown. • Null hypothesis: 𝜇 = 𝜇0 for some specific value 𝜇0 • Test statistic: 𝑡= where 𝑠2 = 𝑥 − 𝜇0 √ 𝑠/ 𝑛 𝑛 1 ∑(𝑥𝑖 − 𝑥)2 . 𝑛 − 1 𝑖=1 Here 𝑡 is called the Studentized mean and 𝑠2 is called the sample variance. The latter is an estimate of the true variance 𝜎2 . • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of 𝑇 ∼ 𝑡(𝑛 − 1), the 𝑡 distribution with 𝑛 − 1 degrees of freedom.* 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 7 • One-sided 𝑝-value (right side): 𝑝 = 𝑃 (𝑇 ≥ 𝑡 | 𝐻0 ) One-sided 𝑝-value (left side): 𝑝 = 𝑃 (𝑇 ≤ 𝑡 | 𝐻0 ) 2𝑃 (𝑇 ≥ 𝑡) if 𝑡 > 0 Two-sided 𝑝-value: 𝑝 = { 2𝑃 (𝑇 ≤ 𝑡) if 𝑡 < 0. Because of the symmetry of the distribution around 0, we can also write this as 𝑝 = 𝑃 (|𝑇 | ≥ |𝑡|). *Important note. This is a good example of how we will work with significance tests. Once we know the distribution of the test statistic, all the tests have the same basic form. In this case, we make use of a theorem that says, for normal data the Studentized mean follows a 𝑡-distribution. We will not prove this in 18.05, but you can look up the proof if you want: https://en.wikipedia.org/wiki/Student's_t-distribution#Derivation Example 2. Now suppose that in Example 1 the variance is unknown. That is, we have data that follows a normal distribution of unknown mean 𝜇 and and unknown variance 𝜎. Suppose we collect the same data as before: 1, 2, 3, 6, −1 As above, let the null hypothesis 𝐻0 be that 𝜇 = 0 and the alternative hypothesis 𝐻𝐴 be that 𝜇 > 0. At a significance level of 𝛼 = 0.05, should we reject the null hypothesis? Solution: There are 5 data points with average 𝑥 = 2.2. Because we have normal data with unknown mean and unknown variance we should use a one-sample 𝑡 test. Computing the sample variance we get 𝑠2 = 1 ((1 − 2.2)2 + (2 − 2.2)2 + (3 − 2.2)2 + (6 − 2.2)2 + (−1 − 2.2)2 ) = 6.7 4 Our 𝑡-statistic is the Studentized mean: 𝑡 = 2.2 − 0 𝑥 − 𝜇0 √ = √ √ = 1.901 𝑠/ 𝑛 6.7/ 5 Our test is one-sided because the alternative hypothesis is one-sided. So (using R) the 𝑝-value is 𝑝 = 𝑃 (𝑇 > 𝑡) = 𝑃 (𝑇 > 1.901) = 1-pt(1.901,4) = 0.065 Since 𝑝 > 0.05, we do not reject the null hypothesis. We can visualize the test as follows: 𝜙(𝑦|𝐻0 ) ∼ 𝑡(4) -1 0 non-reject 𝐻0 Rejection region = shaded orange (starts at 𝑞0.95 = 𝑡0.05 = 2.13) 𝛼 = shaded orange area = 0.05 𝑡 = statistic = black dot = 1.90 𝑝 = blue stripe area = 0.065 1 1.9 𝑥 2.13 reject 𝐻0 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 5.5 8 Two-sample 𝑡-test with equal variances We next consider the case of comparing the means of two samples. For example, we might be interested in comparing the mean efficacies of two medical treatments. • Data: We assume we have two sets of data drawn from normal distributions 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ 𝑁 (𝜇1 , 𝜎2 ) 𝑦1 , 𝑦2 , … , 𝑦𝑚 ∼ 𝑁 (𝜇2 , 𝜎2 ) where the means 𝜇1 and 𝜇2 and the variance 𝜎2 are all unknown. Notice the assumption that the two distributions have the same variance. Also notice that there are 𝑛 samples in the first group and 𝑚 samples in the second. • Null hypothesis: 𝜇1 = 𝜇2 (the values of 𝜇1 and 𝜇2 are not specified) • Test statistic: 𝑡= 𝑥−𝑦 , 𝑠𝑝 where 𝑠2𝑝 is the pooled variance 𝑠2𝑝 (𝑛 − 1)𝑠2𝑥 + (𝑚 − 1)𝑠2𝑦 1 1 ( + ) = 𝑛+𝑚−2 𝑛 𝑚 Here 𝑠2𝑥 and 𝑠2𝑦 are the sample variances of the 𝑥𝑖 and 𝑦𝑗 respectively. The expression for 𝑡 is somewhat complicated, but the basic idea remains the same and it still results in a known null distribution. • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of 𝑇 ∼ 𝑡(𝑛 + 𝑚 − 2). • One-sided 𝑝-value (right side): 𝑝 = 𝑃 (𝑇 > 𝑡 | 𝐻0 ) One-sided 𝑝-value (left side): 𝑝 = 𝑃 (𝑇 < 𝑡 | 𝐻0 ) Two-sided 𝑝-value: 𝑝 = 𝑃 (|𝑇 | > |𝑡|). Note 1: Some authors use a different notation. They define the pooled variance as 𝑠2𝑝-other-authors = (𝑛 − 1)𝑠2𝑥 + (𝑚 − 1)𝑠2𝑦 𝑛+𝑚−2 and what we called the pooled variance they point out is the estimated variance of 𝑥 − 𝑦. That is, 𝑠2𝑝 = 𝑠𝑝-other-authors × (1/𝑛 + 1/𝑚) ≈ 𝑠2𝑥−𝑦̄ Note 2: There is a version of the two-sample 𝑡-test that allows the two groups to have different variances. In this case the test statistic is a little more complicated but R will handle it with equal ease. Note 3: We reiterate our ‘important note’ from above: It can be proved that under the assumptions on the data (independent samples, normal data, equal variances), the null distribution is a 𝑡-distribution. We won’t prove this in 18.05. But knowing it, we can 18.05 Class 18, Null Hypothesis Significance Testing II, Spring 2022 9 work with and understand the gist of the two-sample 𝑡-test in exactly the same way we can understand other significance tests. Example 3. The following data comes from a real study in which 1408 women were admitted to a maternity hospital for (i) medical reasons or through (ii) unbooked emergency admission. The duration of pregnancy is measured in complete weeks from the beginning of the last menstrual period. We can summarize the data as follows: ̄ = 39.08 and 𝑠2𝑀 = 7.77. Medical: 775 observations with 𝑥𝑀 Emergency: 633 observations with 𝑥𝐸 ̄ = 39.60 and 𝑠2𝐸 = 4.95 Set up and run a two-sample 𝑡-test to investigate whether the mean duration differs for the two groups. What assumptions did you make? Solution: The pooled variance for this data is 𝑠2𝑝 = 774(7.77) + 632(4.95) 1 1 ( ) = 0.0187 + 1406 775 633 The 𝑡 statistic for the null distribution is 𝑥𝑀̄ − 𝑦𝐸̄ = −3.8064 𝑠𝑝 We have 1406 degrees of freedom. Using R to compute the two-sided 𝑝-value we get 𝑝 = 𝑃 (|𝑇 | > |𝑡|) = 2*pt(-3.8064, 1406) = 0.00015 𝑝 is very small, much smaller than 𝛼 = 0.05 or 𝛼 = 0.01. Therefore we reject the null hypothesis in favor of the alternative that there is a difference in the mean durations. Rather than compute the two-sided 𝑝-value exactly using a 𝑡-distribution we could have noted that with 1406 degrees of freedom the 𝑡 distribution is essentially standard normal and 3.8064 is almost 4 standard deviations. So 𝑃 (|𝑡| ≥ 3.8064) ≈ 𝑃 (|𝑧| ≥ 3.8064) < 0.001 We assumed the data was normal and that the two groups had equal variances. Given the large difference between the sample variances this assumption may not be warranted. In fact, there are other significance tests that test whether the data is approximately normal and whether the two groups have the same variance. In practice one might apply these first to determine whether a 𝑡 test is appropriate in the first place. We don’t have time to go into normality tests here, but we will see the 𝐹 distribution used for equality of variances next week. https://en.wikipedia.org/wiki/Normality_test https://en.wikipedia.org/wiki/F-test_of_equality_of_variances