18.05 S22 Class 19: Null Hypothesis Significance Testing III

Null Hypothesis Significance Testing III Class 19, 18.05 1 Learning Goals 1. Given hypotheses and data, be able to identify an appropriate significance test from a list of common ones. 2. Given hypotheses, data, and a suggested significance test, know how to look up details and apply the significance test. 2 Introduction In these notes we will collect together some of the most common significance tests, though by necessity we will leave out many other useful ones. Still, all significance tests follow the same basic pattern in their design and implementation, so by learning the ones we include you should be able to apply other ones as needed. Designing a null hypothesis significance test (NHST): • Specify null and alternative hypotheses. • Choose a test statistic whose null distribution and alternative distribution(s) are known. • Specify a rejection region. Very often this is done implicitly by specifying a significance level 𝛼 and a method for computing 𝑝-values based on the tails of the null distribution. • Compute power using the alternative distribution(s). Running a NHST: • Collect data and compute the test statistic. • Check if the test statistic is in the rejection region. Most often this is done implicitly by checking if 𝑝 < 𝛼. If so, we ‘reject the null hypothesis in favor of the alternative hypothesis’. Otherwise we conclude ‘the data does not support rejecting the null hypothesis’. Note the careful phrasing: when we fail to reject 𝐻0 , we do not conclude that 𝐻0 is true. The failure to reject may have other causes. For example, we might not have enough data to clearly distinguish 𝐻0 and 𝐻𝐴 , whereas more data might indicate that we should reject 𝐻0 . 1 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 3 2 Population parameters and sample statistics Example 1. If we randomly select 10 men from a population and measure their heights we say that we have sampled the heights from the population. In this case the sample mean, say 𝑥, is the mean of the sampled heights. It is a statistic and we know its value explicitly. On the other hand, the true average height of the population, say 𝜇, is unknown and we can only estimate its value. We call 𝜇 a population parameter. The main purpose of significance testing is to use sample statistics to draw conlusions about population parameters. For example, we might test if the average height of men in a given population is greater than 70 inches. 4 A gallery of common significance tests related to the normal distribution We will show a number of tests that all assume normal data. For completeness we will include the 𝑧 and 𝑡 tests we’ve already explored. You shouldn’t try to memorize these tests. It is a hopeless task to memorize the tests given here and even more hopeless to memorize all the tests we’ve left out. Rather, your goal should be to be able to find the correct test when you need it. Pay attention to the types of hypotheses the tests are designed to distinguish and the assumptions about the data needed for the test to be valid. We will work through the details of these tests in class and on homework. The null distributions for all of these tests are all related to the normal distribution by explicit formulas. We will not go into the details of these distributions or the arguments showing how they arise as the null distributions in our significance tests. However, the arguments are accessible to anyone who knows calculus and is interested in understanding them. Given the name of any distribution, you can easily look up the details of its construction and properties online. You can also use R to explore the distribution numerically and graphically. When analyzing data with any of these tests one thing of key importance is to verify that the assumptions are true or at least approximately true. For example, you shouldn’t use a test that assumes the data is normal unless you’ve checked that the data is approximately normal. The script class19.r contains examples of using R to run some of these tests. It is posted in our usual place for R code. 4.1 𝑧-test • Use: Test if the population mean equals a hypothesized mean. • Data: 𝑥1 , 𝑥2 , … , 𝑥𝑛 . • Assumptions: The data are independent normal samples: 𝑥𝑖 ∼ 𝑁 (𝜇, 𝜎2 ) where 𝜇 is unknown, but 𝜎 is known. • 𝐻0 : For a specified 𝜇0 , 𝜇 = 𝜇0 . 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 • 𝐻𝐴 : Two-sided: one-sided-greater: one-sided-less: 3 𝜇 ≠ 𝜇0 𝜇 > 𝜇0 𝜇 < 𝜇0 𝑥 − 𝜇0 √ 𝜎/ 𝑛 • Null distribution: 𝜙(𝑧 | 𝐻0 ) is the pdf of 𝑍 ∼ 𝑁 (0, 1). • Test statistic: 𝑧 = • 𝑝-value: Two-sided: one-sided-greater: one-sided-less: 𝑝 = 𝑃 (|𝑍| > 𝑧) 𝑝 = 𝑃 (𝑍 > 𝑧) 𝑝 = 𝑃 (𝑍 < 𝑧) = = = 2*(1-pnorm(abs(z), 0, 1)) 1 - pnorm(z, 0, 1) pnorm(z, 0, 1) • R code: There does not seem to be a single R function in the base R packages that runs a 𝑧-test. There are other packages you can install that have a z.test function. Of course, it is easy enough to get R to compute the 𝑧 score and 𝑝-value. There is an example of this in class19.r. Example 2. We quickly reprise our example from the class 17 notes. IQ is normally distributed in the population according to a N(100, 152 ) distribution. We suspect that most MIT students have above average IQ so we frame the following hypotheses. 𝐻0 = MIT student IQs are distributed identically to the general population = MIT IQ’s follow a N(100, 152 ) distribution. 𝐻𝐴 = MIT student IQs tend to be higher than those of the general population = the average MIT student IQ is greater than 100. Notice that 𝐻𝐴 is one-sided. Suppose we test 9 students and find they have an average IQ of 𝑥̄ = 112. Can we reject 𝐻0 at a significance level 𝛼 = 0.05? Solution: Our test statistic is 𝑧= 𝑥̄ − 100 36 √ = = 2.4. 15 15/ 9 The right-sided 𝑝-value is therefore 𝑝 = 𝑃 (𝑍 ≥ 2.4) = 1- pnorm(2.4,0,1) = 0.0081975. Since 𝑝 ≤ 𝛼 we reject the null hypothesis in favor of the alternative hypothesis that MIT students have higher IQs on average. 4.2 One-sample 𝑡-test of the mean • Use: Test if the population mean equals a hypothesized mean. • Data: 𝑥1 , 𝑥2 , … , 𝑥𝑛 . • Assumptions: The data are independent normal samples: 𝑥𝑖 ∼ 𝑁 (𝜇, 𝜎2 ) where both 𝜇 and 𝜎 are unknown. 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 4 • 𝐻0 : For a specified 𝜇0 , 𝜇 = 𝜇0 • 𝐻𝐴 : Two-sided: one-sided-greater: one-sided-less: • Test statistic: 𝑡 = 𝜇 ≠ 𝜇0 𝜇 > 𝜇0 𝜇 < 𝜇0 𝑥 − 𝜇0 √ , 𝑠/ 𝑛 where 𝑠2 is the sample variance: 𝑠2 = 𝑛 1 ∑(𝑥𝑖 − 𝑥)2 𝑛 − 1 𝑖=1 • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of 𝑇 ∼ 𝑡(𝑛 − 1). (Student 𝑡-distribution with 𝑛 − 1 degrees of freedom) • 𝑝-value: Two-sided: one-sided-greater: one-sided-less: 𝑝 = 𝑃 (|𝑇 | > 𝑡) 𝑝 = 𝑃 (𝑇 > 𝑡) 𝑝 = 𝑃 (𝑇 < 𝑡) = = = 2*(1-pt(abs(t), n-1)) 1 - pt(t, n-1) pt(t, n-1) • R code example: For data 𝑥 = 1, 3, 5, 7, 2 we can run a one-sample 𝑡-test with 𝐻0 : 𝜇0 = 2.5 using the R command: t.test(x, mu = 2.5, alternative=t́wo.sided)́ This will return a several pieces of information including the mean of the data, 𝑡-value and the two-sided 𝑝-value. See the help for this function for other argument settings. Example 3. Look in the class 18 notes or slides for an example of this test. The class 19 example R code also gives an example. 4.3 Two-sample 𝑡-test for comparing means 4.3.1 The case of equal variances We start by describing the test assuming equal variances. • Use: Test if the population means from two populations differ by a hypothesized amount. • Data: 𝑥1 , 𝑥2 , … , 𝑥𝑛 and 𝑦1 , 𝑦2 , … , 𝑦𝑚 . • Assumptions: Both groups of data are independent normal samples: 𝑥𝑖 ∼ 𝑁 (𝜇𝑥 , 𝜎2 ) 𝑦𝑗 ∼ 𝑁 (𝜇𝑦 , 𝜎2 ) where both 𝜇𝑥 and 𝜇𝑦 are unknown and possibly different. The variance 𝜎2 is unknown, but the same for both groups. • 𝐻0 : For a specified Δ𝜇 the difference of means 𝜇𝑥 − 𝜇𝑦 = Δ𝜇 • 𝐻𝐴 : Two-sided: one-sided-greater: one-sided-less: 𝜇𝑥 − 𝜇𝑦 ≠ Δ𝜇 𝜇𝑥 − 𝜇𝑦 > Δ𝜇 𝜇𝑥 − 𝜇𝑦 < Δ𝜇 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 5 𝑥 − 𝑦 ̄ − Δ𝜇 , 𝑠𝑃 where 𝑠2𝑥 and 𝑠2𝑦 are the sample variances of the 𝑥 and 𝑦 data respectively, and 𝑠2𝑃 is (sometimes called) the pooled sample variance: • Test statistic: 𝑡 = 𝑠2𝑝 = (𝑛 − 1)𝑠2𝑥 + (𝑚 − 1)𝑠2𝑦 1 1 ( + ) 𝑛+𝑚−2 𝑛 𝑚 and 𝑑𝑓 = 𝑛 + 𝑚 − 2 • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of 𝑇 ∼ 𝑡(𝑑𝑓), the 𝑡-distribution with 𝑑𝑓 = 𝑛 + 𝑚 − 2 degrees of freedom. • 𝑝-value: Two-sided: one-sided-greater: one-sided-less: 𝑝 = 𝑃 (|𝑇 | > 𝑡) 𝑝 = 𝑃 (𝑇 > 𝑡) 𝑝 = 𝑃 (𝑇 < 𝑡) = = = 2*(1-pt(abs(t), df)) 1 - pt(t, df) pt(t, df) • R code: The R function t.test will run a two-sample 𝑡-test. See the example code in class19.r. In t.test the argument mu is used for what we have called Δ𝜇. Notes: 1. Most often the test is done with Δ𝜇 = 0. That is, the null hypothesis is the the means are equal, i.e. 𝜇𝑥 − 𝜇𝑦 = 0. 2. If the 𝑥 and 𝑦 data have the same length 𝑛 = 𝑚, then the formula for 𝑠2𝑝 becomes simpler: 𝑠2𝑥 + 𝑠2𝑦 𝑠2𝑝 = 𝑛 Example 4. Look in the class 18 notes or slides for an example of the two-sample 𝑡-test. 4.3.2 The case of unequal variances There is a form of the 𝑡-test for when the variances are not assumed equal. It is sometimes called Welch’s 𝑡-test. This looks exactly the same as the case of equal except for a small change in the assumptions and the formula for the pooled variance: • Use: Test if the population means from two populations differ by a hypothesized amount. • Data: 𝑥1 , 𝑥2 , … , 𝑥𝑛 and 𝑦1 , 𝑦2 , … , 𝑦𝑚 . • Assumptions: Both groups of data are independent normal samples: 𝑥𝑖 ∼ 𝑁 (𝜇𝑥 , 𝜎𝑥2 ) 𝑦𝑗 ∼ 𝑁 (𝜇𝑦 , 𝜎𝑦2 ) where both 𝜇𝑥 and 𝜇𝑦 are unknown and possibly different. The variances 𝜎𝑥2 and 𝜎𝑦2 are unknown and not assumed to be equal. • 𝐻0 , 𝐻𝐴 : Exactly the same as the case of equal variances. 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 6 𝑥 − 𝑦 ̄ − Δ𝜇 , 𝑠𝑃 where 𝑠2𝑥 and 𝑠2𝑦 are the sample variances of the 𝑥 and 𝑦 data respectively, and 𝑠2𝑃 is (sometimes called) the pooled sample variance: • Test statistic: 𝑡 = 𝑠2𝑝 = 𝑠2𝑥 𝑠2𝑦 + 𝑛 𝑚 and 𝑑𝑓 = (𝑠2𝑥 /𝑛 + 𝑠2𝑦 /𝑚)2 (𝑠2𝑥 /𝑛)2 /(𝑛 − 1) + (𝑠2𝑦 /𝑚)2 /(𝑚 − 1) • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of degrees of freedom. 𝑇 ∼ 𝑡(𝑑𝑓), the 𝑡 distribution with 𝑑𝑓 • 𝑝-value: Exactly the same as the case of equal variances. • R code: The function t.test also handles this case if you set the argument var.equal=FALSE. Notes. 1. In truth, the null distribution given above only approximates the exact null distribution. 2. Notice that the degrees of freedom are unlikely to be a whole number. 3. Some people recommend always using Welch’s t-test, even if the variances are believed to be equal. This avoids making the assumption that the variances are equal and has very little downside if they are equal. 4.3.3 The paired two-sample 𝑡-test When the data naturally comes in pairs (𝑥𝑖 , 𝑦𝑖 ), we can use the paired two-sample 𝑡-test. (After checking the assumptions are valid!) Example 5. To measure the effectiveness of a cholesterol lowering medication we might test each subject before and after treatment with the drug. So for each subject we have a pair of measurements: 𝑥𝑖 = cholesterol level before treatment 𝑦𝑖 = cholesterol level after treatment. Example 6. To measure the effectiveness of a cancer treatment we might pair each subject who received the treatment with one who did not. In this case we would want to pair subjects who are similar in terms of stage of the disease, age, sex, etc. • Use: Test if the average difference between paired values in a population equals a hypothesized value. • Data: 𝑥1 , 𝑥2 , … , 𝑥𝑛 and 𝑦1 , 𝑦2 , … , 𝑦𝑛 must have the same length. • Assumptions: The differences 𝑤𝑖 = 𝑥𝑖 −𝑦𝑖 between the paired samples are independent draws from a normal distribution N(𝜇, 𝜎2 ), where 𝜇 and 𝜎 are unknown. • 𝐻0 : For a specified 𝜇0 , 𝜇 = 𝜇0 . • 𝐻𝐴 : Two-sided: one-sided-greater: one-sided-less: 𝜇 ≠ 𝜇0 𝜇 > 𝜇0 𝜇 < 𝜇0 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 • Test statistic: 𝑡 = 7 𝑤 − 𝜇0 √ , 𝑠/ 𝑛 where 𝑠2 is the sample variance: 𝑠2 = 𝑛 1 ∑(𝑤𝑖 − 𝑤)2 𝑛 − 1 𝑖=1 • Null distribution: 𝜙(𝑡 | 𝐻0 ) is the pdf of 𝑇 ∼ 𝑡(𝑛 − 1). (Student 𝑡-distribution with 𝑛 − 1 degrees of freedom) • 𝑝-value: Two-sided: one-sided-greater: one-sided-less: 𝑝 = 𝑃 (|𝑇 | > 𝑡) 𝑝 = 𝑃 (𝑇 > 𝑡) 𝑝 = 𝑃 (𝑇 < 𝑡) = = = 2*(1-pt(abs(t), n-1)) 1 - pt(t, n-1) pt(t, n-1) • R code: The R function t.test will do a paired two-sample test if you set the argument paired=TRUE. You can also run a one-sample 𝑡-test on 𝑥−𝑦. There are examples of both of these in class19.r Notes. 1. This is just a one-sample 𝑡-test using 𝑤𝑖 . 2. Another way to write the assumption is that we assume a relation between 𝑥𝑖 and 𝑦𝑖 of the form 𝑦𝑖 = 𝑥𝑖 + 𝜇 + 𝑒. Here 𝜇 is some (unknown) constant, and 𝑒 is random error (noise) of mean 0 and (unknown) variance 𝜎2 . Example 7. The following example is taken from Rice 1 To study the effect of cigarette smoking on platelet aggregation Levine (1973) drew blood samples from 11 subjects before and after they smoked a cigarette and measured the extent to which platelets aggregated. Here is the data: Before After Difference 25 27 2 25 29 4 27 37 10 44 56 12 30 46 16 67 82 15 53 57 4 53 80 27 52 61 9 60 59 -1 28 43 15 The null hypothesis is that smoking had no effect on platelet aggregation, i.e. that the difference between before and after should have mean 𝜇0 = 0. We ran a paired two-sample 𝑡-test to test this hypothesis. Here is the R code: (It’s also in class19.r.) before.cig = c(25,25,27,44,30,67,53,53,52,60,28) after.cig = c(27,29,37,56,46,82,57,80,61,59,43) mu0 = 0 result = t.test(after.cig, before.cig, alternative="two.sided", mu=mu0, paired=TRUE) print(result) Here is the output: Paired t-test data: after.cig and before.cig t = 4.2716, df = 10, p-value = 0.001633 alternative hypothesis: true difference in means is not equal to 0 mean of the differences: 10.27273 We got the same results with the one-sample 𝑡-test: t.test(after.cig - before.cig, mu=0) 1 John Rice, Mathematical Statistics and Data Analysis, 2nd edition, p. 412. This example references P.H Levine (1973) An acute effect of cigarette smoking on platelet function. Circulation, 48, 619-623. 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 4.4 8 One-way ANOVA (𝐹 -test for equal means) • Use: Test if the population means from 𝑛 groups are all the same. • Data: (𝑛 groups, 𝑚 samples from each group) 𝑥1,1 , 𝑥2,1 , 𝑥1,2 , 𝑥2,2 , 𝑥1,𝑚 𝑥2,𝑚 …, …, … 𝑥𝑛,1 , 𝑥𝑛,2 , … , 𝑥𝑛,𝑚 • Assumptions: Data for each group is an independent normal sample drawn from distributions with (possibly) different means but the same variance: 𝑥1,𝑗 𝑥2,𝑗 𝑥𝑛,𝑗 ∼ 𝑁 (𝜇1 , 𝜎2 ) ∼ 𝑁 (𝜇2 , 𝜎2 ) … ∼ 𝑁 (𝜇𝑛 , 𝜎2 ) The group means 𝜇𝑖 are unknown and possibly different. The variance 𝜎 is unknown, but the same for all groups. • 𝐻0 : All the means are identical 𝜇1 = 𝜇2 = … = 𝜇𝑛 . • 𝐻𝐴 : Not all the means are the same. • Test statistic: 𝑓 = 𝑥𝑖̄ 𝑥 𝑠2𝑖 MS𝐵 MS𝑊 , where = mean of group 𝑖 𝑥𝑖,1 + 𝑥𝑖,2 + … + 𝑥𝑖,𝑚 = . 𝑚 = grand mean of all the data. = sample variance of group 𝑖 𝑚 1 = ∑(𝑥𝑖,𝑗 − 𝑥𝑖̄ )2 . 𝑚 − 1 𝑗=1 = between group variance = 𝑚 × sample variance of group means 𝑚 𝑛 = ∑(𝑥̄ − 𝑥)2 . 𝑛 − 1 𝑖=1 𝑖 MS𝑊 = average within group variance = sample mean of 𝑠21 , … , 𝑠2𝑛 𝑠2 + 𝑠22 + … + 𝑠2𝑛 = 1 𝑛 • Idea: If the 𝜇𝑖 are all equal, test statistic 𝑓, which is a ratio, should be near 1. If they are not equal then MS𝐵 should be larger while MS𝑊 should remain about the same, so 𝑓 should be larger. We won’t give a proof of this. MS𝐵 • Null distribution: 𝜙(𝑓 | 𝐻0 ) is the pdf of 𝐹 ∼ 𝐹 (𝑛 − 1, 𝑛(𝑚 − 1)). This is the 𝐹 -distribution with (𝑛 − 1) and 𝑛(𝑚 − 1) degrees of freedom. Several 𝐹 -distributions are plotted below. • 𝑝-value: 𝑝 = 𝑃 (𝐹 > 𝑓) = 1- pf(f, n-1, n*(m-1))) 9 1.0 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 0.0 0.2 0.4 0.6 0.8 F(3,4) F(10,15) F(30,15) 0 2 4 x 6 8 10 Notes: 1. ANOVA tests whether all the means are the same. It does not test whether some subset of the means are the same. 2. There is a test where the variances are not assumed equal. 3. There is a test where the groups don’t all have the same number of samples. 4. R has a function aov() to run ANOVA tests. 5. See: https://en.wikipedia.org/wiki/F-test Example 8. The table shows patients’ perceived level of pain (on a scale of 1 to 6) after 3 different medical procedures. 𝑇1 2 4 1 5 3 𝑇2 3 4 6 1 4 𝑇3 2 1 3 3 5 (1) Set up and run an F-test comparing the means of these 3 treatments. (2) Based on the test, what might you conclude about the treatments? Solution: Using the code below, the 𝐹 statistic is 0.325 and the 𝑝-value is 0.729 At any reasonable significance level we will fail to reject the null hypothesis that the average pain level is the same for all three treatments.. Note, it is not reasonable to conclude the the null hypothesis is true. With just 5 data points per procedure we might simply lack the power to distinguish different means. R code to perform the test # DATA ---T1 = c(2,4,1,5,3) T2 = c(3,4,6,1,4) T3 = c(2,1,3,3,5) procedure = c(rep('T1',length(T1)),rep('T2',length(T2)),rep('T3',length(T3))) pain = c(T1,T2,T3) data.pain = data.frame(procedure,pain) aov.data = aov(pain∼procedure,data=data.pain) # do the analysis of variance print(summary(aov.data)) # show the summary table # class19.r also shows code to compute the ANOVA by hand. 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 10 The summary shows a 𝑝-value (shown as Pr(>F)) of 0.729. Therefore we do not reject the null hypothesis that all three group population means are the same. 4.5 Chi-square test for goodness of fit This is a test of how well a hypothesized probability distribution fits a set of data. The test statistic is called a chi-square statistic and the null distribution associated to the chi-square statistic is the chi-square distribution. It is denoted by 𝜒2 (𝑑𝑓) where the parameter 𝑑𝑓 is called the degrees of freedom. Suppose we have an unknown probability mass function given by the following table. Outcomes Probabilities 𝜔1 𝑝1 𝜔2 𝑝2 … … 𝜔𝑛 𝑝𝑛 In the chi-square test for goodness of fit we hypothesize a set of values for the probabilities. Typically we will hypothesize that the probabilities follow a known distribution with certain parameters, e.g. binomial, Poisson, multinomial. The test then tries to determine if this set of probabilities could reasonably have generated the data we collected. • Use: Test whether discrete data fits a specific finite probability mass function. • Data: An observed count 𝑂𝑖 for each possible outcome 𝜔𝑖 . • Assumptions: None • 𝐻0 : The data was drawn from a specific discrete distribution. • 𝐻𝐴 : The data was drawn from a different distribution. • Test statistic: The data consists of observed counts 𝑂𝑖 for each 𝜔𝑖 . From the null hypothesis probability table we get a set of expected counts 𝐸𝑖 . There are two statistics that we can use: Likelihood ratio statistic 𝐺 = 2 ∗ ∑ 𝑂𝑖 ln ( Pearson’s chi-square statistic 𝑋 2 = ∑ 𝑂𝑖 ) 𝐸𝑖 (𝑂𝑖 − 𝐸𝑖 )2 . 𝐸𝑖 It is a theorem that under the null hypothesis 𝑋 2 ≈ 𝐺 and both are approximately chi-square. Before computers, 𝑋 2 was used because it was easier to compute. Now, it is better to use 𝐺 although you will still see 𝑋 2 used quite often. • Degrees of freedom 𝑑𝑓: For chi-square tests the number of degrees of freedom can be a bit tricky. In this case 𝑑𝑓 = 𝑛 − 1. It is computed as the number of cell counts that can be freely set under 𝐻𝐴 consistent with the statistics needed to compute the expected cell counts assuming 𝐻0 . • Null distribution: Assuming 𝐻0 , both statistics (approximately) follow a chi-square distribution with 𝑑𝑓 degrees of freedom. That is both 𝜙(𝐺 | 𝐻0 ) and 𝜙(𝑋 2 | 𝐻0 ) have (approximately) the same pdf as 𝑌 ∼ 𝜒2 (𝑑𝑓). • 𝑝-value: Extreme data means large values of 𝑋 2 , i.e. large differences between the observed and expected counts. So, 𝑝 = 𝑃 (𝑌 > 𝐺) = 1 - pchisq(G, df) 𝑝 = 𝑃 (𝑌 > 𝑋 2 ) = 1 - pchisq(𝑋 2 , df) 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 11 • R code: The R function chisq.test can be used to do the computations for a chisquare test use 𝑋 2 . For 𝐺 you either have to do it by hand or find a package that has a function. (It will probably be called likelihood.test or G.test. Notes. 1. When the likelihood ratio statistic 𝐺 is used the test is also called a 𝐺-test or a likelihood ratio test. Example 9. First chi-square example. Suppose we have an experiment that produces numerical data. For this experiment the possible outcomes are 0, 1, 2, 3, 4, 5 or more. We run 51 trials and count the frequency of each outcome, getting the following data: Outcomes Observed counts 0 3 1 10 2 15 3 13 4 7 ≥5 3 Suppose our null hypothesis 𝐻0 is that the data is drawn from 51 trials of a binomial(8, 0.5) distribution and our alternative hypothesis 𝐻𝐴 is that the data is drawn from some other distribution. Do all of the following: 1. 2. 3. 4. Make a table of the observed and expected counts. Compute both the likelihood ratio statistic 𝐺 and Pearson’s chi-square statistic 𝑋 2 . Compute the degrees of freedom of the null distribution. Compute the 𝑝-values corresponding to 𝐺 and 𝑋 2 . Solution: All of the R code used for this example is in class19.r. 1. Assuming 𝐻0 the data truly comes from a binomial(8, 0.5) distribution. We have 51 total observations, so the expected count for each outcome is just 51 times its probability. We computed the binomial(8, 0.5) probabilities and expected counts in R: Outcomes Observed counts 𝐻0 probabilities Expected counts 0 3 0.0039 0.20 1 10 0.0313 1.59 2 15 0.1094 5.58 3 13 0.2188 11.16 4 7 0.2734 13.95 ≥5 3 0.3633 18.53 2. Using the formulas above we compute that 𝑋 2 = 116.41 and 𝐺 = 66.08 3. The only statistic used in computing the expected counts was the total number of observations 51. So, the degrees of freedom is 5, i.e we can set 5 of the cell counts freely and the last is determined by requiring that the total number is 51. 4. The 𝑝-values are 𝑝𝐺 =1 - pchisq(G, 5) and 𝑝𝑋2 = 1 - pchisq(𝑋 2 , 5). Both 𝑝values are effectively 0. For almost any significance level we would reject 𝐻0 in favor of 𝐻𝐴 . 4.5.1 Degrees of freedom in chi-square tests We alreay gave a quick definition of degrees of freedom for a chi-square test. Here we will try to go a little slower in showing how to compute degrees of freedom. To start, recall that in a chi-square test, our table has 𝑛 observed counts. Then, we use observed counts and the null hypothesis to compute 𝑛 expected counts. This is typically done by computing some statistics and using them estimate the parameters needed to compute the expected counts. For example, in the previous example the statistic computed was the total number of counts. 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 12 Now, imagine that we are allowed to fabricate the 𝑛 observed counts, but we demand that our made up observations produce the same statistics as the true observed counts. That is, our imaginary observed counts need to produce the same expected counts as the true data. The degrees of freedom is the number of fake observed counts we can freely choose. The rest will be determined by our constraint that they produce the same statistics. Example 10. (Degrees of freedom.) Suppose we have the following observed counts Outcomes Observed counts 0 6 1 9 2 13 3 12 4 7 5 3 Suppose our null hypothesis 𝐻0 is that the data producing these counts was drawn from 50 trials of a binomial(5, 𝜃) distribution. Our alternative hypothesis 𝐻𝐴 is that the data is drawn from some other distribution. Run a chi-square test of these hypotheses with significance 0.05. Solution: To compute expected counts we need a value of 𝜃. Since it is not known, we have to use the data to estimate it. The total number of observations is 50. So, the mean of the data is 𝑚= 6 ⋅ 0 + 9 ⋅ 1 + 13 ⋅ 2 + 12 ⋅ 3 + 7 ⋅ 4 + 3 ⋅ 5 = 2.28. 50 The expected value of a binom(5,𝜃) distribution, is 5𝜃. The maximum likelihood estimate for 𝜃 is 𝜃 ̂ = 𝑚/5 = 0.456. Now, just like in the previous example, we can compute expected counts for each possible outcome. The expected count of outcome 𝑘 is 50 ⋅ 𝑝(𝑘). In R this is 50*dbinom(k, 50, ̂ We have the following table 𝜃). Outcomes Observed counts Expected counts 0 6 2.38 1 9 9.98 2 13 16.74 3 12 14.03 4 7 5.88 5 3 0.99 To determine the degrees of freedom: (i) We have 6 observed counts. (ii) To compute the expected counts we need the total number of counts = 50 and our estimate of 𝜃 ̂ = 𝑚/5 That is, we have two constraints: the total number of counts is 50, and mean 𝑚 = 2.28. So, to get the same expected counts, we could choose 4 of the observed counts freely and then set last two counts so the constraints are met. Thus, there are 4 degrees of freedom. More briefly: 6 observed counts - 2 constraints = 4 degrees of freedom. Using R we can compute 𝐺 = 8.01, with 𝑝-value 0.09. Thus, at significance level 0.05 we would not reject the null hypothesis. Note, the 𝑋 2 statistic is 11.05 with 𝑝-value 0.026. Clearly this example is a borderline case, since we reject 𝐻0 when using 𝑋 2 and we don’t when using 𝐺. It bears repeating, for reasons like this, we never say 𝐻0 is false. The most we can say is that the data does not support rejecting 𝐻0 . 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 13 4.5.2 More examples Example 11. Mendel’s genetic experiments (Adapted from Rice Mathematical Statistics and Data Analysis, 2nd ed., example C, p.314) In one of his experiments on peas Mendel looked at 2 genetic trait pairs: smooth/wrinkled and yellow/green. Symbolically we label a smooth gene 𝑆 and a wrinkled gene 𝑠. Likewise we use 𝑌 and 𝑦 for yellow and green respectively. Mendel started by selecting a parent generation of homozygous plants. They were either smooth/yellow (genes 𝑆𝑆𝑌 𝑌 ) and wrinkled/green (genes 𝑠𝑠𝑦𝑦). He crossed the smooth/yellow with the wrinkled/green peas creating the, so called, 𝐹1 generation consisting of plants with genes 𝑆𝑠𝑌 𝑦. Since smooth (𝑆) and yellow 𝑌 are both dominant traits, all these plants were smooth/yellow. He then crossed 556 pairs of the 𝐹1 generation to create the 𝐹2 generation. We would expect 1/4 of the 𝐹2 generation to have two smooth genes (𝑆𝑆), 1/4 to have two wrinkled genes (𝑠𝑠), and the remaining 1/2 to be heterozygous (𝑆𝑠). We also expect these fractions for yellow (𝑌 ) and green (𝑦) genes. If the color and smoothness genes are inherited independently and smooth and yellow are both dominant we’d expect the following table of frequencies for phenotypes. Yellow Green 9/16 3/16 3/4 3/16 1/16 1/4 3/4 1/4 1 Probability table for the null hypothesis Smooth Wrinkled So from the 556 crosses the expected number of smooth yellow peas is 556 × 9/16 = 312.75. Likewise for the other possibilities. Here is a table giving the observed and expected counts from Mendel’s experiments. Smooth yellow Smooth green Wrinkled yellow Wrinkled green Observed count 315 108 102 31 Expected count 312.75 104.25 104.25 34.75 The null hypothesis is that the observed counts are random samples distributed according to the frequency table given above. We use the counts to compute our statistics The likelihood ratio statistic is 𝑂𝑖 ) 𝐸𝑖 315 108 102 31 = 2 ∗ (315 ln ( ) + 108 ln ( ) + 102 ln ( ) + 31 ln ( )) 34.75 412.75 104.25 104.25 = 0.618 𝐺 = 2 ∗ ∑ 𝑂𝑖 ln ( Pearson’s chi-square statistic is 𝑋2 = ∑ 2.752 (𝑂𝑖 − 𝐸𝑖 )2 3.752 2.252 3.752 = = 0.604 + + + 𝐸𝑖 312.75 104.25 104.25 34.75 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 14 You can see that the two statistics are very close. This is usually the case. In general the likelihood ratio statistic is more robust and should be preferred. The degrees of freedom is 3, because there are 4 observed quantities and one relation between them, i.e. they sum to 556. So, under the null hypothesis 𝐺 follows a 𝜒2 (3) distribution. Using R to compute the 𝑝-value we get 𝑝 = 1- pchisq(0.618, 3) = 0.8923 Assuming the null hypothesis we would see data at least this extreme almost 90% of the time. We would not reject the null hypothesis for any reasonable significance level. The 𝑝-value using Pearson’s statistic is 0.8955 –nearly identical. The script class19.r shows these calculations and also how to use chisq.test to run a chi-square test directly. 4.6 Chi-square test for homogeneity This is a test to see if several independent sets of random data are all drawn from the same distribution. (The meaning of homogeneity in this case is that all the distributions are the same.) • Use: Test whether 𝑚 different independent sets of discrete data are drawn from the same distribution. • Outcomes: 𝜔1 , 𝜔2 , … , 𝜔𝑛 are the possible outcomes. These are the same for each set of data. • Data: We assume 𝑚 independent sets of data giving counts for each of the possible outcomes. That is, for data set 𝑖 we have an observed count 𝑂𝑖,𝑗 for each possible outcome 𝜔𝑗 . • Assumptions: None • 𝐻0 : Each data set is drawn from the same distribution. (We don’t specify what this distribution is.) • 𝐻𝐴 : The data sets are not all drawn from the same distribution. • Test statistic: See the example below. There are 𝑚𝑛 cells containing counts for each outcome for each data set. Using the null distribution we can estimate expected counts for each of the data sets. The statistics 𝑋 2 and 𝐺 are computed exactly as above. • Degrees of freedom 𝑑𝑓: (𝑚 − 1)(𝑛 − 1). (See the example below.) • The null distribution 𝜒2 (𝑑𝑓). The 𝑝-values are computed just as in the chi-square test for goodness of fit. • R code: The R function chisq.test can be used to do the computations for a chisquare test use 𝑋 2 . For 𝐺 you either have to do it by hand or find a package that has a function. (It will probably be called likelihood.test or G.test. Example 12. Someone claims to have found a long lost work by William Shakespeare. They ask you to test whether or not the play was actually written by Shakespeare . 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 15 You go to https://www.opensourceshakespeare.org and pick a random 12 pages from King Lear and count the use of common words. You do the same thing for the ‘long lost work’. You get the following table of counts. Word King Lear Long lost work a 150 90 an 30 20 this 30 10 that 90 80 Using this data, set up and evaluate a significance test of the claim that the long lost book is by William Shakespeare. Use a significance level of 0.1. Solution: The null hypothesis 𝐻0 : For the 4 words counted the long lost book has the same relative frequencies as the counts taken from King Lear. The total word count of both books combined is 500, so the the maximum likelihood estimate of the relative frequencies assuming 𝐻0 is simply the total count for each word divided by the total word count. Word a an this that Total count King Lear 150 30 30 90 300 Long lost work 90 20 10 80 200 totals 240 50 40 170 500 rel. frequencies under 𝐻0 240/500 50/500 40/500 170/500 500/500 Now the expected counts for each book under 𝐻0 are the total count for that book times the relative frequencies in the above table. The following table gives the counts: (observed, expected) for each book. Word King Lear Long lost work Totals a (150, 144) (90, 96) (249, 240) an (30, 30) (20, 20) (50, 50) this (30, 24) (10, 16) (40, 40) that (90, 102) (80, 68) (170, 170) Totals (300, 300) (200, 200) (500, 500) The chi-square statistic is (𝑂𝑖 − 𝐸𝑖 )2 𝐸𝑖 2 6 62 122 02 62 02 62 122 = + + + + + + + 144 30 24 102 96 20 16 68 ≈ 7.9 𝑋2 = ∑ There are 8 cells and all the marginal counts are fixed because they were needed to determine the expected counts. To be consistent with these statistics we could freely set the values in 3 cells in the table, e.g. the 3 blue cells, then the rest of the cells are determined in order to make the marginal totals correct. Thus 𝑑𝑓 = 3. (Or we could recall that 𝑑𝑓 = (𝑚 − 1)(𝑛 − 1) = (3)(1) = 3, where 𝑚 is the number of columns and 𝑛 is the number of rows.) Using R we find p = 1-pchisq(7.9,3) = 0.048. Since this is less than our significance level of 0.1 we reject the null hypothesis that the relative frequencies of the words are the same in both books. If we make the further assumption that all of Shakespeare’s plays have similar word frequencies (which is something we could check) we conclude that the book is probably not 18.05 Class 19, Null Hypothesis Significance Testing III, Spring 2022 16 by Shakespeare. 4.7 Other tests There are far too many other tests to even make a dent. We will see some of them in class and on psets. Again, we urge you to master the paradigm of NHST and recognize the importance of choosing a test statistic with a known null distribution.