18.05 S22 Class 17: Null Hypothesis Significance Testing I

Null Hypothesis Significance Testing I Class 17, 18.05 1 Learning Goals 1. Know the definitions of the significance testing terms: null hypothesis, alternative hypothesis, NHST, simple hypothesis, composite hypothesis, significance level, power. 2. Be able to design and run a significance test for Bernoulli or binomial data. 3. Be able to compute a 𝑝-value for a normal hypothesis and use it in a significance test. 2 Introduction Frequentist statistics is often applied in the framework of null hypothesis significance testing (NHST). We will look at the Neyman-Pearson paradigm which focuses on one hypothesis called the null hypothesis. There are other paradigms for hypothesis testing, but NeymanPearson is the most common. Stated simply, this method asks if the data is well outside the region where we would expect to see it under the null hypothesis. If so, then we reject the null hypothesis in favor of a second hypothesis called the alternative hypothesis. The reasoning is that such extreme data is very unlikely in a world where the null hypothesis is true. We have said before that statistics is an art. We will see that this statement is certainly valid when we discuss choosing null and alternative hypotheses. The computations done here all involve the likelihood function. There are two main differences between what we’ll do here and what we did in Bayesian updating. 1. The evidence of the data will be considered purely through the likelihood function: it will not be weighted by our prior beliefs. 2. We will need a notion of extreme data, e.g. 95 out of 100 heads in a coin toss or a mayfly that lives for a month. 2.1 A suggestion of how to learn this material There are seemingly a lot of terms with similar definitions. If you pay careful attention to the figures and how they are shaded and labeled we think you will find that is not so complicated. 2.2 Motivating examples Example 1. Suppose you want to decide whether a coin is fair. If you toss it 100 times and get 85 heads, would you think the coin is likely to be unfair? What about 60 heads? Or 52 heads? Most people would guess that 85 heads is strong evidence that the coin is unfair, 1 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 2 whereas 52 heads is no evidence at all. Sixty heads is less clear. Null hypothesis significance testing (NHST) is a frequentist approach to thinking quantitatively about these questions. Example 2. Suppose you want to compare a new medical treatment to a placebo or to the current standard of care. What sort of evidence would convince you that the new treatment is better than the placebo or the current standard? Again, NHST is a quantitative framework for answering these questions. 3 Significance testing We’ll start by listing the ingredients for NHST. Formally they are pretty simple. There is an art to choosing good ingredients. We will explore the art in examples. If you have never seen NHST before just scan this list now and come back to it after reading through the examples and explanations given below. 3.1 Ingredients • 𝐻0 : the null hypothesis. This is the default assumption for the model generating the data. • 𝐻𝐴 : the alternative hypothesis. If we reject the null hypothesis we accept this alternative as the best explanation for the data. • 𝑋: the test statistic. We compute this from the data. It is a random variable, because it is computed from random data. • Null distribution: the probability distribution of 𝑋 assuming 𝐻0 . • Rejection region: if 𝑋 is in the rejection region we reject 𝐻0 in favor of 𝐻𝐴 . • Non-rejection region: the complement to the rejection region. If 𝑋 is in this region we do not reject 𝐻0 . Note that we say ‘do not reject’ rather than ‘accept’ because usually the best we can say is that the data does not support rejecting 𝐻0 . The null hypothesis 𝐻0 and the alternative hypothesis 𝐻𝐴 play different roles. Typically we choose 𝐻0 to be either a simple hypothesis or the default which we’ll only reject if we have enough evidence against it. The examples below will clarify this. 4 NHST Terminology In this section we will use one extended example to introduce and explore the terminology used in null hypothesis significance testing (NHST). Example 3. To test whether a coin is fair we flip it 10 times. If we get an unexpectedly large or small number of heads we’ll suspect the coin is unfair. To make this precise in the language of NHST we set up the ingredients as follows. Let 𝜃 be the probability that the coin lands heads when flipped. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 3 1. 2. 3. 4. Null hypothesis: 𝐻0 = ‘the coin is fair’, i.e. 𝜃 = 0.5. Alternative hypothesis: 𝐻𝐴 = ‘the coin is not fair’, i.e. 𝜃 ≠ 0.5 Test statistic: 𝑋 = number of heads in 10 flips Null distribution: This is the probability function based on the null hypothesis 𝑝(𝑥 | 𝜃 = 0.5) ∼ binomial(10, 0.5). Here is the probability table for the null distribution. 𝑥 𝑝(𝑥 | 𝐻0 ) 0 .001 1 .010 2 .044 3 .117 4 .205 5 .246 6 .205 7 .117 8 .044 9 .010 10 .001 5. Rejection region: under the null hypothesis we expect to get about 5 heads in 10 tosses. Whereas, under the alternate hypothesis we expect the number of heads to be biased either above or below 5. So, we’ll reject 𝐻0 in favor of 𝐻𝐴 if the number of heads is much fewer or greater than 5. Let’s set the rejection region as {0, 1, 2, 8, 9, 10}. That is, if the number of heads in 10 tosses is in this region we will reject the hypothesis that the coin is fair in favor of the hypothesis that it is not. We can summarize all this in the graph and probability table below. The rejection region consists of those values of 𝑥 in orange, i.e. 0, 1, 2, 8, 9, 10. The probabilities corresponding to it are shaded in orange. We also show the null distribution as a stem plot with the rejection values of 𝑥 in orange. 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 Rejection region and null probabilities as a table for Example 3. 𝑝(𝑥 | 𝐻0 ) .25 .15 .05 𝑥 0 1 2 3 4 5 6 7 8 9 10 Rejection region and null probabilities as a stem plot for Example 3. Notes for Example 3: 1. The null hypothesis is the cautious default: we won’t claim the coin is unfair unless we have compelling evidence. 2. The rejection region consists of data that is extreme under the null hypothesis and more likely under the alternative hypothesis. That is, it consists of the outcomes that are in the tail of the null distribution away from the high probability center. As we’ll discuss soon, how far away depends on the significance level 𝛼 of the test. 3. If we get 3 heads in 10 tosses, then the test statistic is in the non-rejection region. The usual scientific language would be to say that the data ‘does not support rejecting the null hypothesis’. Even if we got 5 heads, we would not claim that the data proves the null hypothesis is true. Question: If we have a fair coin what is the probability that we will decide incorrectly it is unfair? 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 4 Solution: The null hypothesis is that the coin is fair. The question asks for the probability the data from a fair coin will be in the rejection region. That is, the probability that we will get 0, 1, 2, 8, 9 or 10 heads in 10 tosses. This is the sum of the probabilities in the table in shaded orange boxes. That is, 𝑃 (rejecting 𝐻0 | 𝐻0 is true) = 0.11 Below we will continue with Example 3, define more terms used in NHST and see how to quantify properties of the significance test. 4.1 Simple and composite hypotheses Definition: simple hypothesis: A simple hypothesis is one for which we can specify its distribution completely. A typical simple hypothesis is that a parameter of interest takes a specific value. Definition: composite hypotheses: If its distribution cannot be fully specified, we say that the hypothesis is composite. A typical composite hypothesis is that a parameter of interest lies in a range of values. In Example 3 the null hypothesis is that 𝜃 = 0.5, so the null distribution is binomial(10, 0.5). Since the null distribution is fully specified, 𝐻0 is simple. The alternative hypothesis is that 𝜃 ≠ 0.5. This is really many hypotheses in one: 𝜃 could be 0.51, 0.7, 0.99, etc. Since the alternative distribution binomial(10, 𝜃) is not fully specified, 𝐻𝐴 is composite. Example 4. Suppose we have data 𝑥1 , … , 𝑥𝑛 . Suppose also that our hypotheses are 𝐻0 : the data is drawn from 𝑁 (0, 1) 𝐻𝐴 : the data is drawn from 𝑁 (1, 1). These are both simple hypotheses – each hypothesis completely specifies a distribution. Example 5. (Composite hypotheses.) Now suppose that our hypotheses are 𝐻0 : the data is drawn from a Poisson distribution of unknown parameter. 𝐻𝐴 : the data is not drawn from a Poisson distribution. These are both composite hypotheses, as they don’t fully specify the distribution. Example 6. In an ESP experiment a subject is asked to identify the suits of 100 cards drawn (with replacement) from a deck of cards. Let 𝑇 be the number of successes. The (simple) null hypothesis that the subject does not have ESP is given by 𝐻0 : 𝑇 ∼ binomial(100, 0.25) The (composite) alternative hypothesis that the subject has ESP is given by 𝐻𝐴 : 𝑇 ∼ binomial(100, 𝑝) with 𝑝 > 0.25 Another (composite) alternative hypothesis that something besides pure chance is going on, i.e. the subject has ESP or anti-ESP. This is given by 𝐻𝐴 : 𝑇 ∼ binomial(100, 𝑝), with 𝑝 ≠ 0.25 Values of 𝑝 < 0.25 represent hypotheses that the subject has a kind of anti-esp. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 4.2 5 Types of error There are two types of errors we can make. We can incorrectly reject the null hypothesis when it is true or we can incorrectly fail to reject it when it is false. These are unimaginatively labeled type I and type II errors. We summarize this in the following table. Our decision Reject 𝐻0 ‘Don’t reject’ 𝐻0 True state of nature 𝐻0 𝐻𝐴 Type I error correct decision correct decision Type II error Type I: false rejection of 𝐻0 Type II: false non-rejection (‘acceptance’) of 𝐻0 4.3 Significance level and power Significance level and power are used to quantify the quality of the significance test. Ideally a significance test would not make errors. That is, it would not reject 𝐻0 when 𝐻0 was true and would reject 𝐻0 in favor of 𝐻𝐴 when 𝐻𝐴 was true. Altogether there are 4 important probabilities corresponding to the 2 × 2 table just above. 𝑃 (reject 𝐻0 |𝐻0 ) 𝑃 (do not reject 𝐻0 |𝐻0 ) 𝑃 (reject 𝐻0 |𝐻𝐴 ) 𝑃 (do not reject 𝐻0 |𝐻𝐴 ) The two probabilities we focus on are: Significance level Power = 𝑃 (reject 𝐻0 |𝐻0 ) = probability we incorrectly reject 𝐻0 = 𝑃 (type I error). = probability we correctly reject 𝐻0 = 𝑃 (reject 𝐻0 |𝐻𝐴 ) = 1 − 𝑃 (type II error). Ideally, a hypothesis test should have a small significance level (near 0) and a large power (near 1). Here are two analogies to help you remember the meanings of significance and power. Some analogies 1. Think of 𝐻0 as the hypothesis ‘nothing noteworthy is going on’, i.e. ‘the coin is fair’, ‘the treatment is no better than placebo’ etc. And think of 𝐻𝐴 as the opposite: ‘something interesting is happening’. Then power is the probability of detecting something interesting when it’s present and significance level is the probability of mistakenly claiming something interesting has occured. 2. In the U.S. criminal defendents are presumed innocent until proven guilty beyond a reasonable doubt. We can phrase this in NHST terms as 𝐻0 : the defendent is innocent (the default) 𝐻𝐴 : the defendent is guilty. Significance level is the probability of finding an innocent person guilty. Power is the probability of correctly finding a guilty party guilty. ‘Beyond a reasonable doubt’ means we should demand the significance level be very small. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 6 Composite hypotheses 𝐻𝐴 is composite in Example 3, so the power is different for different values of 𝜃. We expand the previous probability table to include some alternate values of 𝜃. We do the same with the stem plots. As always in the NHST game, we look at likelihoods: the probability of the data given a hypothesis. The rejection range consists of the extreme values of 𝑥. The non-rejection range (3-7) is a set of center values of 𝑥. 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝐻0 ∶ 𝑝(𝑥|𝜃 = 0.5) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 .001 𝐻𝐴 ∶ 𝑝(𝑥|𝜃 = 0.6) 0.000 0.002 0.011 0.042 0.111 0.201 0.251 0.215 0.121 0.040 0.006 𝐻𝐴 ∶ 𝑝(𝑥|𝜃 = 0.7) 0.000 0.000 0.001 0.009 0.037 0.103 0.200 0.267 0.233 0.121 0.028 𝑝(𝑥 | 𝐻0 ) 𝑝(𝑥 | 𝜃 = .6) 𝑝(𝑥 | 𝜃 = .7) .25 .25 .25 .15 .15 .15 .05 .05 .05 𝑥 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Rejection region and null and alternative probabilities for example 3 We use the probability table to compute the significance level and power of this test. Significance level = = = = = probability we reject 𝐻0 when it is true probability the test statistic is in the rejection region when 𝐻0 is true probability the test stat. is in the rejection region of the 𝐻0 row of the table sum of shaded orange boxes in the 𝜃 = 0.5 row 0.11 Power when 𝜃 = 0.6 = = = = = probability we reject 𝐻0 when 𝜃 = 0.6 probability the test statistic is in the rejection region when 𝜃 = 0.6 probability the test stat. is in the rejection region of the 𝜃 = 0.6 row of the table sum of blue boxes in the 𝜃 = 0.6 row 0.180 Power when 𝜃 = 0.7 = = = = = probability we reject 𝐻0 when 𝜃 = 0.7 probability the test statistic is in the rejection region when 𝜃 = 0.7 probability the test stat. is in the rejection region of the 𝜃 = 0.7 row of the table sum of blue boxes in the 𝜃 = 0.7 row 0.384 We see that the power is greater for 𝜃 = 0.7 than for 𝜃 = 0.6. This isn’t surprising since we expect it to be easier to recognize that a 0.7 coin is unfair than is is to recognize 0.6 coin is unfair. Typically, we get higher power when the alternate hypothesis is farther from the null hypothesis. In Example 3, it would be quite hard to distinguish a fair coin from one with 𝜃 = 0.51. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 4.4 7 Conceptual sketches We illustrate the notions of null hypothesis, rejection region and power with some sketches of the pdfs for the null and alternative hypotheses. 4.4.1 Null distribution: rejection and non-rejection regions The first diagram below illustrates a null distribution with rejection and non-rejection regions. Also shown are two possible test statistics: 𝑥1 and 𝑥2 . 𝜙(𝑥|𝐻0 ) 𝑥2 -3 reject 𝐻0 𝑥1 0 don’t reject 𝐻0 𝑥 3 reject 𝐻0 The test statistic 𝑥1 is in the non-rejection region. So, if our data produced the test statistic 𝑥1 then we would not reject the null hypothesis 𝐻0 . On the other hand the test statistic 𝑥2 is in the rejection region, so if our data produced 𝑥2 then we would reject the null hypothesis in favor of the alternative hypothesis. There are several things to note in this picture. 1. The rejection region consists of values far from the center of the null distribution. 2. The rejection region is two-sided. We will also see examples of one-sided rejection regions as well. 3. The probability of rejection (significance) is the shaded area under the curve, i.e. the probability of data being in the rejection region assuming 𝐻0 is true. 4. The alternative hypothesis is not mentioned. We reject or don’t reject 𝐻0 based only on the likelihood 𝜙(𝑥|𝐻0 ), i.e. the probability of the test statistic conditioned on 𝐻0 . As we will see, the alternative hypothesis 𝐻𝐴 should be considered when choosing a rejection region, but formally it does not play a role in rejecting or not rejecting 𝐻0 . 5. Sometimes we rather lazily call the non-rejection region the acceptance region. This is technically incorrect because we never truly accept the null hypothesis. We either reject or say the data does not support rejecting 𝐻0 . This is often summarized by the statement: you can never prove the null hypothesis. 4.4.2 High and low power tests The next two figures show high and low power tests. In both tests the null distributions are standard normal. Likewise, the alternative distributions are both normal with variance 1, but different means: -4.0 in the top figure and -0.4 in the bottom one. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 8 Since the alternative distribution is to the left of the null distribution we use a one-sided rejection region. It is the same for both tests. The shaded area under 𝜙(𝑥|𝐻0 ) represents the significance level – it is the same for both tests. Remember the significance level can be described two ways: • The probability of falsely rejecting the null hypothesis when it is true. • The probabilitiy the test statistic falls in the rejection region even though 𝐻0 is true. Likewise, the shaded area under 𝜙(𝑥|𝐻𝐴 ) represents the power, i.e. the probability that the test statistic is in the rejection (of 𝐻0 ) region when 𝐻𝐴 is true. Both tests have the same significance level, but different powers. When 𝜙(𝑥|𝐻𝐴 ) has considerable overlap with 𝜙(𝑥|𝐻0 ) the power will tend to be low (bottom figure) and when they are well separated it tends to be high (top figure). It is well worth your while to thoroughly understand these graphical representations of significance testing. 𝜙(𝑥|𝐻0 ) 𝜙(𝑥|𝐻𝐴 ) 𝑥1 𝑥2 -4 0 . reject 𝐻0 region 𝑥 𝑥3 non-reject 𝐻0 region High power test 𝜙(𝑥|𝐻0 ) 𝜙(𝑥|𝐻𝐴 ) 𝑥1 reject 𝐻0 region 𝑥2 -0.4 . 0 𝑥 𝑥3 non-reject 𝐻0 region Low power test In the top figure we see that the means of the null and alternative distributions are 4 standard deviations apart. Since the areas under the densities have very little overlap the test has high power. That is, if 𝐻𝐴 was true, then seeing the data 𝑥3 would be rare and surprising and similarly for any point in the non-rejection region. That is, if 𝐻𝐴 is the true distribution we are extremely likely to correctly reject the null hypothesis, i.e. we are unlikely to make a type II error. In the bottom figure we see that the means of the null and alternative distributions are just 0.4 standard deviations apart. Since the areas under the densities have a lot of overlap the test has low power. That is, if the data 𝑥 is drawn from 𝐻𝐴 it is highly likely to be in the non-rejection region. For example 𝑥3 would be not be a very surprising outcome for the 𝐻𝐴 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 9 distribution. That is, if 𝐻𝐴 is the true distribution, we are highly likely to make a type II error. Typically we can increase the power of a test by increasing the amount of data and thereby decreasing the variance of the null and alternative distributions. In experimental design it is important to determine ahead of time the number of trials or subjects needed to achieve a desired power. Example 7. Suppose a drug for a disease is being compared to a placebo. We choose our null and alternative hypotheses as 𝐻0 = the drug does not work better than the placebo 𝐻𝐴 = the drug works better than the placebo The power of the hypothesis test is the probability that the test will conclude that the drug is better, if it is indeed truly better. The significance level is the probability that the test will conclude that the drug works better, when in fact it does not. We will look at this in more detail below. 5 Designing a hypothesis test Formally all a hypothesis test requires is 𝐻0 , 𝐻𝐴 , a test statistic and a rejection region. In practice the design is often done using the following steps. 1. Pick the null hypothesis 𝐻0 . The choice of 𝐻0 and 𝐻𝐴 is not mathematics. It’s art and custom. We often choose 𝐻0 to be simple. Or we often choose 𝐻0 to be the simplest or most cautious explanation, i.e. no effect of drug, no ESP, no bias in the coin. 2. Decide if 𝐻𝐴 is one-sided or two-sided. In Example 3 we wanted to know if the coin was unfair. An unfair coin could be biased for or against heads, so 𝐻𝐴 ∶ 𝜃 ≠ 0.5 is a two-sided hypothesis. If we only care whether or not the coin is biased for heads we could use the one-sided hypothesis 𝐻𝐴 ∶ 𝜃 > 0.5. 3. Pick a test statistic. For example, the sample mean, sample total, or sample variance. Often the choice is obvious. Some standard statistics that we will encounter are 𝑧, 𝑡, and 𝜒2 . We will learn to use these statistics as we work examples over the next few classes. One thing we will say repeatedly is that the distributions that go with these statistics are always conditioned on the null hypothesis. That is, we will compute likelihoods such as 𝜙(𝑧 | 𝐻0 ). 4. Pick a significance level and determine the rejection region. We will usually use 𝛼 to denote the significance level. The Neyman-Pearson paradigm is to pick 𝛼 in advance. Typical values are 0.1, 0.05, 0.01. Recall that the significance level is the probability of a type I error, i.e. of incorrectly rejecting the null hypothesis when it is true. The value we choose will depend on the consequences of a type I error. Once the significance level is chosen we can determine the rejection region in the tail(s) of the null distribution. In Example 3, 𝐻𝐴 is two sided so the rejection region is split between the two tails of the null distribution. This distribution is given in the following table: 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 𝑥 0 1 2 3 4 5 6 7 8 10 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 If we set 𝛼 = 0.05 then the rejection region must contain at most 0.05 probability. For a two-sided rejection region this is split between the two tails, so we get {0, 1, 9, 10}. If we set 𝛼 = 0.01 the rejection region is {0, 10}. We show these in tables with the rejection region (values of 𝑥) inside the orange rectangle and the corresponding null likelihoods in the shaded orange boxes. 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 Tables with shaded rejection regions for 𝛼 = 0.05 (top) and 𝛼 = 0.01 (bottom) Suppose we change 𝐻𝐴 to ‘the coin is biased in favor of heads’. We now have a one-sided hypothesis 𝜃 > 0.5. Our rejection region will now be in the right-hand tail since we don’t want to reject 𝐻0 in favor of 𝐻𝐴 if we get a small number of heads. Now if 𝛼 = 0.05 the rejection region is the one-sided range {9, 10}. If we set 𝛼 = 0.01 then the rejection region is {10}. As above we show the one sided rejection regions in tables. 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 𝑥 0 1 2 3 4 5 6 7 8 9 10 𝑝(𝑥|𝐻0 ) 0.001 0.010 0.044 0.117 0.205 0.246 0.205 0.117 0.044 0.010 0.001 Tables with shaded one-sided rejection regions for 𝛼 = 0.05 (top) and 𝛼 = 0.01 (bottom) 5. Determine the power(s). As we saw in Example 3, once the rejection region is set we can determine the power of the test at various values of the alternate hypothesis. Example 8. (Consequences of significance) If 𝛼 = 0.1 then we’d expect a 10% type I error rate. That is, we expect to reject the null hypothesis in 10% of those experiments 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 11 where the null hypothesis is true. Whether 0.1 is a reasonable signficance level depends on the decisions that will be made using it. For example, if you were running an experiment to determine if your chocolate is more than 72% cocoa then a 10% error type I error rate is probably okay. That is, falsely believing some 72% chocalate is greater that 72%, is probably acceptable. On the other hand, if your forensic lab is identifying fingerprints for a murder trial then a 10% type I error rate, i.e. mistakenly claiming that fingerprints found at the crime scene belonged to someone who was truly innocent, is definitely not acceptable. Significance for a composite null hypothesis. If 𝐻0 is composite then P(type I error) depends on which member of 𝐻0 is true. In this case the significance level is defined as the maximum of these probabilities. 6 Critical values Critical values are like quantiles except they refer to the probability to the right of the value instead of the left. Example 9. Use R to find the 0.05 critical value for the standard normal distribution. Solution: We label this critical value 𝑧0.05 . The critical value 𝑧0.05 is just the 0.95 quantile, i.e. it has 5% probability to its right and therefore 95% probability to its left. We computed it with the R function qnorm: qnorm(0.95, 0, 1), which returns 1.64. In a typical significance test the rejection region consists of one or both tails of the null distribution. The value of the test statistic that marks the start of the rejection region is a critical value. We show this and the notation used in some examples. Example 10. Critical values and rejection regions. Suppose our test statistic 𝑥 has null distribution N(100, 152 ), i.e. 𝜙(𝑥|𝐻0 ) ∼ N(100, 152 ). Suppose also that our rejection region is right-sided and we have a significance level of 0.05. Find the critical value and sketch the null distribution and rejection region. Solution: The notation used for the critical value with right tail containing probability 0.05 is 𝑥0.05 . The critical value 𝑥0.05 is just the 0.95 quantile, i.e. it has 5% probability to its right and therefore 95% probability to its left. We computed it with the R function qnorm: qnorm(0.95, 100, 15), which returned 124.7. This is shown in the figure below. 𝜙(𝑥|𝐻0 ) ∼ N(100, 152 ) 𝑥0.05 = 124.7 𝛼 = shaded area = 0.05 100 non-reject 𝐻0 𝑥 𝑥0.05 reject 𝐻0 Example 11. Critical values and rejection regions. Repeat the previous example for a left-sided rejection region with significance level 0.05. Solution: In this case the critical value has 0.05 probability to its left and therefore 0.95 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 12 probability to its right. So we label it 𝑥0.95 . Since it is the 0.05 quantile compute it with the R function: qnorm(0.05, 100, 15), which returned 75.3. 𝜙(𝑥|𝐻0 ) ∼ N(100, 152 ) 𝑥0.95 = 75.3 𝛼 = shaded area = 0.05 𝑥0.95 reject 𝐻0 𝑥 100 non-reject 𝐻0 Example 12. Critical values. Repeat the previous example for a two-sided rejection region. Put half the significance in each tail. Solution: To have a total significance of 0.05 we put 0.025 in each tail. That is, the left tail starts at 𝑥0.975 = 𝑞0.025 and the right tail starts at 𝑥0.025 = 𝑞0.975 . We compute these values with qnorm(0.025, 100, 15) and qnorm(0.975, 100, 15). The values are shown in the figure below. 𝜙(𝑥|𝐻0 ) ∼ N(100, 152 ) 𝑥0.025 = 129.4 𝑥0.975 = 70.6 𝛼 = shaded area = 0.05 𝑥0.975 reject 𝐻0 7 100 non-reject 𝐻0 𝑥0.025 𝑥 reject 𝐻0 𝑝-values In practice people often specify the significance level and do the significance test using what are called 𝑝-values. We will first define 𝑝-value and then state the 𝑝-test. After that, we will illustrate it with figures and examples. Definition. The 𝑝-value is the probability, assuming the null hypothesis, of seeing data at least as extreme as the experimental data. What ‘at least as extreme’ means depends on the experimental design. P-test. If the 𝑝-value is less than the significance level 𝛼 then we reject 𝐻0 . Otherwise we do not reject 𝐻0 . We first illustrate 𝑝-values graphically and then we will work a simple one-sided example. We will look at two-sided examples in later classes. Suppose we have a right-sided alternate hypothesis, so our rejection region is in the right tail of the range of possible outcomes. This is illustrated in the following figure. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 13 𝜙(𝑥|𝐻0 ) = null pdf 𝑥𝑞 = critical point 𝛼 = shaded area 𝑥 𝑥𝑞 non-reject 𝐻0 reject 𝐻0 Right-sided rejection region Suppose we get data 𝑥1 which is in the rejection region. This is shown in the figure below. 𝜙(𝑥|𝐻0 ) = null pdf 𝑥𝑞 = critical point 𝛼 = shaded area 𝑝 = striped area 𝑥𝑞 non-reject 𝐻0 𝑥 𝑥1 reject 𝐻0 𝑝-value = probability of data ‘more extreme’ than 𝑥1 : 𝑝 < 𝛼 Since the rejection region is right-sided, the phrase ‘at least as extreme’ means all values to the right of 𝑥1 . So, the 𝑝-value is the area of the striped region. Here is the key to connecting the 𝑝-test, rejection and signifcance: Since 𝑥1 is in the rejection region, the striped area 𝑝 is less than the shaded area 𝛼. That is, 𝑝 < 𝛼. In other words, we reject the null hypothesis when 𝑝 < 𝛼. For completeness we show a figure where 𝑥1 is not in the rejection region, so 𝑝 > 𝛼. That is, we do not reject 𝐻0 when 𝑝 > 𝛼. 𝜙(𝑥|𝐻0 ) = null pdf 𝑥𝑞 = critical point 𝛼 = shaded area 𝑝 = striped area 𝑥1 non-reject 𝐻0 𝑥 𝑥𝑞 reject 𝐻0 𝑝-value = probability of data ‘more extreme’ than 𝑥1 : 𝑝 > 𝛼 7.1 Example: 𝑧-tests When our test statistic is standard normal we will call it 𝑧, and the corresponding test for significance will be called a 𝑧-test. Example 13. The 𝑧-test for normal hypotheses IQ is normally distributed in the population according to a N(100, 152 ) distribution. We suspect that most MIT students have above average IQ so we frame the following hypothe- 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 ses. 𝐻0 𝐻𝐴 = = = = 14 MIT student IQs are distributed identically to the general population MIT IQ’s follow a N(100, 152 ) distribution. MIT student IQs tend to be higher than those of the general population the average MIT student IQ is greater than 100. Notice that 𝐻𝐴 is one-sided. Suppose we test 9 students and find they have an average IQ of 𝑥 = 112. Can we reject 𝐻0 at a significance level 𝛼 = 0.05? Solution: To compute 𝑝 we first standardize the data: Under the null hypothesis 𝑥̄ ∼ N(100, 152 /9) and therefore 𝑧= 𝑥̄ − 100 36 √ = = 2.4 ∼ N(0, 1). 15 15/ 9 That is, the null distribution for 𝑧 is standard normal. We call 𝑧 a 𝑧-statistic, we will use it as our test statistic. For a right-sided alternative hypothesis the phrase ‘data at least as extreme’ is a one-sided tail to the right of 𝑧. The 𝑝-value is then 𝑝 = 𝑃 (𝑍 ≥ 2.4) = 1- pnorm(2.4,0,1) = 0.0081975. Since 𝑝 ≤ 𝛼 we reject the null hypothesis. The reason this works is explained below. We phrase our conclusion as We reject the null hypothesis in favor of the alternative hypothesis that MIT students have higher IQs on average. We have done this at significance level 0.05 with a 𝑝-value of 0.008. Notes: 1. The average 𝑥 = 112 is random: if we ran the experiment again we could get a different value for 𝑥. 2. We could use the statistic 𝑥 directly. Standardizing is fairly standard because, with practice, we will have a good feel for the meaning of different 𝑧-values. The justification for rejecting 𝐻0 when 𝑝 ≤ 𝛼 is given in the following figure. 𝜙(𝑧|𝐻0 ) ∼ N(0, 1) 𝑧0.05 = 1.645 𝛼 = shaded area = 0.05 𝑝 = striped area = 0.008 -1 0 non-reject 𝐻0 1 𝑧0.05 2.4 reject 𝐻0 𝑥 In this example 𝛼 = 0.05, 𝑧0.05 = 1.64 and the rejection region is the range to the right of 𝑧0.05 . Also, 𝑧 = 2.4 and the 𝑝-value is the probability to the right of 𝑧. The picture illustrates that • 𝑧 = 2.64 is in the rejection region 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 15 • is the same as 𝑧 is to the right of 𝑧0.05 • is the same as the probability to the right of 𝑧 is less than 0.05 • which means 𝑝 < 0.05. 8 More examples Hypothesis testing is widely used in inferential statistics. We don’t expect that the following examples will make perfect sense at this time. Read them quickly just to get a sense of how hypothesis testing is used. We will explore the details of these examples in class. Example 14. The chi-square statistic and goodness of fit. (Rice, example B, p.313) To test the level of bacterial contamination, milk was spread over a grid with 400 squares. The amount of bacteria in each square was measured. We summarize in the table below. The bottom row of the table is the number of different squares that had a given amount of bacteria. Amount of bacteria 0 1 2 3 4 5 6 7 8 9 10 19 Number of squares 56 104 80 62 42 27 9 9 5 3 2 1 We compute that the average amount of bacteria per square is 2.44. Since the Poisson(𝜆) distribution is used to model counts of relatively rare events and the parameter 𝜆 is the expected value of the distribution. we decide to see if these counts could come from a Poisson distribution. To do this we first graphically compare the observed frequencies with those expected from Poisson(2.44). 20 40 60 80 ● ● ● ● ● Poisson(2.44) Observed ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 Number of squares ● ● 5 10 15 Number of bacteria in square The picture is suggestive, so we do a hypothesis test with 𝐻0 ∶ the samples come from a Poisson(2.44) distribution. 𝐻𝐴 ∶ the samples come from a different distribution. 18.05 Class 17, Null Hypothesis Significance Testing I, Spring 2022 16 We use a chi-square statistic, so called because it (approximately) follows a chi-square distribution. To compute 𝑋 2 we first combine the last few cells in the table so that the minimum expected count is around 5 (a general rule-of-thumb in this game.) The expected number of squares with a certain amount of bacteria comes from considering 400 trials from a Poisson(2.44) distribution, e.g., with = 2.44 the expected number of squares with 3 bacteria is 400 × e− 3 = 84.4. 3! (𝑂 − 𝐸𝑖 )2 The chi-square statistic is ∑ 𝑖 , where 𝑂𝑖 is the observed number and 𝐸𝑖 is the 𝐸𝑖 expected number. Number per square Observed Expected Component of 𝑋 2 0 56 34.9 12.8 1 104 85.1 4.2 2 80 103.8 5.5 3 62 84.4 6.0 4 42 51.5 1.7 5 27 25.1 0.14 6 9 10.2 0.15 >6 20 5.0 44.5 Summing up we get 𝑋 2 = 74.9. Since the mean (2.44) and the total number of trials (400) are fixed, the 8 cells only have 6 degrees of freedom. So, assuming 𝐻0 , our chi-square statistic follows (approximately) a 𝜒26 distribution. Using this distribution, 𝑃 (𝑋 2 > 74.59) = 0 (to at least 6 decimal places). Thus we decisively reject the null hpothesis in favor of the alternate hypothesis that the distribution is not Poisson(2.44). To analyze further, look at the individual components of 𝑋 2 . There are large contributions in the tail of the distribution, so that is where the fit goes awry. Example 15. Student’s 𝑡 test. Suppose we want to compare a medical treatment for increasing life expectancy with a placebo. We give 𝑛 people the treatment and 𝑚 people the placebo. Let 𝑋1 , … , 𝑋𝑛 be the number of years people live after receiving the treatment. Likewise, let 𝑌1 , … , 𝑌𝑚 be the number of years people live after receiving the placebo. Let 𝑋̄ and 𝑌 ̄ be the sample means. We want to know if the difference between 𝑋̄ and 𝑌 ̄ is statistically significant. We frame this as a hypothesis test. Let 𝜇𝑋 and 𝜇𝑌 be the (unknown) means. 𝐻0 ∶ 𝜇𝑋 = 𝜇𝑌 , 𝐻𝐴 ∶ 𝜇𝑋 ≠ 𝜇𝑌 . With certain assumptions and a proper formula for the pooled standard error 𝑠𝑝 the test 𝑋̄ − 𝑌 ̄ statistic 𝑡 = follow a 𝑡 distribution with 𝑛 + 𝑚 − 2 degrees of freedom. So our 𝑠𝑝 rejection region is determined by a threshold 𝑡0 with 𝑃 (𝑡 > 𝑡0 ) = 𝛼.