Lecture Notes on Hypothesis Testing

9.07 Introduction to Statistics for Brain and Cognitive Sciences Emery N. Brown Lecture 12: Hypothesis Testing I. Objectives 1. Understand the hypothesis testing paradigm. 2. Understand how hypothesis testing procedures are constructed. 3. Understand how to do sample size calculations. 4. Understand the relation between hypothesis testing, confidence intervals, likelihood and Bayesian methods and their uses for inference purposes. II. The Hypothesis Testing Paradigm and One-Sample Tests A. One-Sample Tests To motivate the hypothesis testing paradigm we review first two problems. In both cases there is a single sample of data. Example 3.2 (continued) Analysis of MEG Sensor Bias. Is there a noise bias in this SQUID sensor? Here we have a sample x1,..., xn where we model each xi : N (µ , σ 2 ). We assume that µ is unknown and σ 2 is known. If there is bias, then µ ≠ 0, and µ > 0 would suggest a positive bias where µ < 0 would suggest a negative bias. Example 2.1 (continued) Behavioral Learning. Has the animal learned the task? We have data x1,..., xn and we recall that our model is xi : B(n, p) where p is unknown. We define learning as performance greater than expected by chance. In particular, chance suggests p = 1 2 1 2 1 2 (binary choice), learning suggests p > , whereas impaired learning suggests p < . To define the hypothesis testing paradigm we state a series of definitions. A hypothesis is the statement of a scientific question in terms of a proposed value of a parameter in a probability model. Hypothesis testing is a process of establishing proof by falsification. It has two essential components: a null hypothesis and an alternative hypothesis. The null hypothesis is a stated value of the parameter which defines the hypothesis we want to falsify. It is usually stated as a single value although it can be composite. We denote it as H0 . The alternative hypothesis is the hypothesis whose veracity we wish to establish. It is usually defined by a value or set of values of the parameter that are different from the one specified in the null hypothesis. We denote it as H A . page 2: 9.07 Lecture 12: Hypothesis Testing To establish the result, we carry out a test in an attempt to reject the null hypothesis. The test is a procedure based on the observed data that allows us to choose between the null and alternative hypothesis. Example 3.2 (continued) Analysis of MEG Sensor Bias. For this example the null hypothesis could be H0 : µ = 0 and alternative hypotheses could be HA : µ > 0 or H A : µ < 0. Example 2.1 (continued) Behavioral Learning. Here the null hypothesis is H0 : p = alternative hypotheses could be one-sided as either HA : p > 1 2 HA : p < 1 2 or 1 and the 2 or two-sided 1 HA : p ≠ . 2 To investigate the hypothesis we require a test statistic. A test statistic is a statistic whose values will allow us to distinguish between the null and the alternative hypotheses. Example 3.2 (continued) Analysis of MEG Sensor Bias. For this example we recall that σ2 x : N (µ , ). Hence, we can choose x, because of its distributional properties. In general, n choosing the statistic is an important issue. Here there are two cases: Case i): If x >> 0 or x << 0 we conclude µ ≠ 0. That is, we would be willing to conclude µ ≠ 0 if | x | is sufficiently large. Indeed, the larger x is in absolute value, the more likely we are to conclude µ ≠ 0. If our data allow us to reach this conclusion, we say we reject the null hypothesis H0 . Case ii) If x is close to 0, we say we fail to reject the null hypothesis H0 . We do not say we accept the null hypothesis because, in this case, we do not reach a conclusion. There are two types of errors we can commit in hypothesis testing. If we reject H0 when it is true, this is an error. It is called a Type 1 error. The probability of the error is denoted as α . We write page 3: 9.07 Lecture 12: Hypothesis Testing α = Pr(rejecting H0 | H0 is true). In Example 3.2 we have α = Pr( x > cα | µ = 0). We choose α as small as possible. Typical values are α = 0.01 and α = 0.05. What is cα ? To determine it, we compute Pr(Type I error) = Pr(rejecting H0 | H0 true) = α. We want α to be small α = Pr( x > cα | µ = µ0 = 0) 1 α = Pr( 1 n 2 ( x − µ0 ) > σ n 2 (cα − µ0 ) σ ) 1 = Pr( n 2 x − µ0 σ > z1−α ) where z1−α is a quantile of the standard Gaussian. Hence, we have cα = µ0 + z1−α values of z1−α for the corresponding values of α are α z1−α 0.05 0.025 1.645 1.96 0.01 2.325 σ 1 n2 . Some The area to the right of z1−α or cα is the critical region. It has probability content of α . The value cα is the cut-off value. This test based on the normal distribution is the z-test. If H0 is not true, i.e. H A is true and we fail to reject H 0 , then this is also an error. It is termed a Type II error or β error and is defined as Pr(Type II error) = Pr(fail to reject H0 | H A is true) Assume H A is true, i.e. µ = µ A > 0. Then we have β = Pr( x < cα | µ = µ A ) 1 1 = Pr( n 2 (x − µ A ) σ < n 2 (cα − µ A ) 1 = Pr( n 2 (x − µ A ) σ < zβ ) σ ) page 4: 9.07 Lecture 12: Hypothesis Testing We do not talk in terms of β but 1− β which is the power of the test. The power of the test is the probability of rejecting the null hypothesis when it is false. We compute it as power = 1− β = Pr(rejecting H 0 | H A is true) = Pr( x > cα | µ = µ A ). Remark 12.1. You should not carry out a test if the power is not at least > 0.80 for important H A . Remark 12.2. Never report a negative result (failing to reject H0 ) without reporting the power. Remark 12.3. A statistical test is an “information assay.” As such, it is only as useful as it is powerful against an important alternative hypothesis. If we reject H0 we report the p-value which is the smallest value of α for which we can reject H0 . The p-value is also the probability of all events that are at least as rare as the observed statistic. It is the probability that we are making a mistake in rejecting the null hypothesis when the null hypothesis is true. An observed value of the test statistic has statistical significance if p < α . Statistical significance does not imply scientific significance. Example 3.2 (continued) Analysis of MEG Sensor Bias. In this example assume we have 500 observations x1,..., xn and the standard deviation is σ = 1.1×10−11 f Tesla . Suppose we wish to test H0 : µ = 0 against the alternative hypothesis H A : µ > 0 with α = 0.05. If we have x = 0.11×10−11 f Tesla then 1 z= n 2 (x ) σ 1 = (500) 2 (0.11) = 2.25 1.1 From the Table of a standard normal distribution we have z1−α = z0.95 = 1.645 or α = 0.05 and we see p = 0.0122. Therefore, we reject and conclude that there is a positive bias in the magnetic field around this recording sensor. Example 2.1 (continued) Behavioral Learning. For this experiment if our hypotheses are 1 2 1 HA : p > 2 H0 : p = k . Based on our Central Limit Theorem 40 results in Lecture 7, we can analyze this question using the Gaussian approximation to the binomial provided np > 5 and n(1 − p) > 5. Because we have We have n = 40 trials and observe k = 22 and p̂ = page 5: 9.07 Lecture 12: Hypothesis Testing ⎛1⎞1 np(1 − p) = 40 ⎜ ⎟ = 10 > 5 ⎝2⎠2 we can use the Gaussian approximation to the binomial to test H0 . Notice that if np(1 − p) > 5 then it must be that np > 5 and n(1 − p) > 5. If we take α = 0.05, our test statistic is 1 z= n 2 ( pˆ − p) [ p(1 − p)] 1 2 1 = (40) 2 (0.55 − 0.5) ⎛1⎞ ⎜ ⎟ ⎝4⎠ 1 2 = 6.32(0.05) = 0.632 1 2 z1−α = 1.645, hence z < z1−α and we fail to reject H0 . We see that 1 cα = p + z1−α [ p(1 − p)] 2 1 n2 1 = + 2 ⎛1⎞ 1.645 ⎜ ⎟ ⎝ 2 ⎠ = 0.63 6.32 What is the power of this test if the true probability of a correct response is pA = 0.72? power = Pr(rejecting H 0 | H A = 0.72) = Pr( p̂ > 0.63 | p A = 0.72) 1 1 = Pr( n 2 ( pˆ − p A ) 1 [ p A (1 − p A )] 2 > n 2 (cα − p A ) 1 [ p A (1 − p A )] 2 | p A = 0.72) 1 = 1− Φ[ (40) 2 (0.63 − 0.72) 1 [(0.18)(0.72)] 2 −(6.32)(0.09) = 1− Φ[ ] (.36) = 1− Φ[−1.58] = 1− 0.057 = 0.943. Therefore, if the true propensity to respond correctly were pA = 0.72 then there was a probability of 0.943 of rejecting the null hypothesis in this case. B. One-Sample Test, Power and Sample Size Calculation Given a null and a one-sided alternative hypothesis H 0 : µ = µ0 H A : µ = µ A > µ0 we compute the power as page 6: 9.07 Lecture 12: Hypothesis Testing Power = Pr(rejecting H 0 | H A is true) 1 = Pr( n 2 ( x − µ0 ) σ > z1−α | µ = µ A ) z1−α σ = Pr( x > µ0 + | µ = µA) 1 n2 z1−α σ = Pr( x − µ A > µ0 − µ A + σ | µ = µA) 1 1 n 2 (x − µA ) = Pr( 1 n2 > n 2 ( µ0 − µ A ) σ + z1−α | µ = µ A ) 1 1− Φ( 1 n 2 ( µ0 − µ A ) + z1−α ) = Φ (− z1−α − σ n 2 ( µ0 − µ A ) σ ) 1 = Φ(zα + n 2 ( µ A − µ0 ) σ ) because −z1−α = zα . Similarly, if we have H 0 : µ = µ0 H A : µ = µ A < µ0 then it is easy to show that 1 power = Φ[ n 2 ( µ0 − µ A ) σ + zα ] In general for a one-sided alternative 1 n 2 | µ0 − µ A | power = Φ[ + zα ] σ We use these formulae to derive expressions for sample size calculations. Notice that 1 n 2 | µ0 − µ A | + zα ) power = Φ( σ 1 n 2 | µ0 − µ A | + zα ) power = 1 − β = Φ( σ If we apply Φ −1 to the left and right hand sides of the equation above we get 1 −1 −1 Φ (1 − β ) = Φ [Φ( n 2 | µ A − µ0 | σ 1 z1− β = n 2 | µ A − µ0 | σ 1 z1− β − zα = + zα n 2 | µ A − µ0 | σ + zα )] page 7: 9.07 Lecture 12: Hypothesis Testing Or since −zα = z1−α we obtain the sample size formula n= σ 2 (z1− β + z1−α )2 Δ2 where Δ =| µ0 − µ A | . Example 3.2 (continued) Analysis of MEG Sensor Bias. How many measurements should Steve Stufflebeam make daily to be at least 80% sure that if there is a positive drift of 0.1×10−11 f Tesla he can detect it with α = 0.05? To answer this question, we apply our sample size formula with z0.95 = 1.645, z0.80 = 0.84, σ = 1.1×10−1 f Tesla and we obtain n= (1.1)2 (1.645 + 0.84)2 (0.1)2 1.21× 6.18 n= 0.01 n = 748 Therefore, in our problem studied above, we should have taken around 750 measurements instead of 500. Remark 12.4. The ratio σ2 Δ2 is like the inverse of a signal-to-noise ratio. As we want a smaller Type I error (z1−α ), and/or more power n increases. Similarly, n increases with σ 2 and decreases with Δ 2 . III. One-Sample Two-Sided Tests If the alternative hypothesis is two-sided then we need to construct a two-sided test. A. Two-Sided Tests Example 3.2 (continued) Analysis of MEG Sensor Bias. Suppose our null and alternative hypotheses for this problem are respectively H0 : µ = 0 H A : µ ≠ 0. This alternative hypothesis implies that we would reject H0 for either a positive bias or negative σ2 ). We would therefore reject H 0 if x >> µ0 = 0 or if n x << µ0 = 0. We take as our test statistic x and we will reject H0 if | x |>> 0. Pick α and take bias. Under H0 we have x : N (µ , α = α1 + α2 . Since we do not favor µ > 0 more than µ < 0, we take α1 = α 2 = consider α 2 . To reject H 0 , we page 8: 9.07 Lecture 12: Hypothesis Testing Pr(| x |> cα 2 ) = Pr( x > cα 2 or x < −cα 2 ) 1 1 = Pr( n 2 ( x − µ0 ) σ > n 2 (cα 2 − µ0 ) 1 1 or σ n 2 ( x − µ0 ) σ = Pr(z > z1−α 2 or z < −z1−α 2 ) <− n 2 (cα 2 − µ0 ) σ ) = Pr(| z |> z1−α 2 ) This is a two-sided test because we reject H0 for either very large positive or negative values of the test statistic. We reject H0 for | z |> z1−α 2 or equivalently, we reject H0 if | x |> cα 2 = µ0 + Examples of α and z1− α are σ 1 n2 z1− α 2 2 α z1− α 2 0.10 0.05 0.01 1.645 1.96 2.58 Example 3.2 (continued) Analysis of MEG Sensor Bias. We consider H0 : µ = 0 HA : µ ≠ 0 Suppose we pick α = 0.05, we assume σ = 1.1 and we compute x = −0.11, then we have 1 z= n 2 (x ) σ 1 −(500) 2 0.11 = = −2.25 1.1 and z1−α 2 = z0.975 =1.96 . Because −2.25 < −1.96, we have z < −z1−α 2 and hence, we reject H0 . Example 2.1 (continued) Behavioral Learning. We can perform a similar analysis for the learning example 1 2 1 HA : p ≠ 2 H 0 : p0 = This alternative implies either impaired learning or learning and would lead us to reject if p̂ >> 1 1 or p̂ << . We have that under the Gaussian approximation 2 2 page 9: 9.07 Lecture 12: Hypothesis Testing 1 z= n 2 ( pˆ − p0 ) 1 [ p0 (1− p0 )] 2 Hence, given α we reject H0 if z > z1−α 2 or z < zα 2 or equivalently if pˆ > cα 2 or p̂ < −cα 2 where cα 2 = p0 +[ p̂ = p0 (1 − p0 ) 12 ] z1−α 2 . Given α = 0.10 , we obtain n z1−α 2 = z0.95 =1.645 and since k 22 = , we have 40 40 1 z= n 2 ( pˆ − p0 ) [ p0 (1 − p0 )] 1 2 1 = (40) 2 (0.55 − 0.5) 1 ( 14 ) 2 = 6.32(0.05) 1 2 = 0.632 Because z < z0.95 we fail to reject H0 . B. Power for the Two-Sided Alternative It is straightforward to show that for the mean of a Gaussian distribution with known variance if the null hypothesis is H0 : µ = µ0 versus the two-sided alternative H A : µ ≠ µ0 the power of the two-sided test is defined as 1 Pr(| n 2 ( x − µ0 ) σ 1 |> z1−α 2 | H A is true) = Pr( n 2 ( x − µ0 ) σ 1 > z1−α 2 or n 2 ( x − µ0 ) σ < zα 2 | H A is true) This simplifies to 1 power = Φ[− z1−α 2 + n 2 ( µ0 − µ A ) σ 1 ] + Φ[− z1−α 2 + n 2 ( µ A − µ0 ) σ ]. The corresponding sample size formula is n= σ2 Δ2 (z1−α 2 + z1− β )2 . where Δ =| µ A − µ0 | . Example 3.2 (continued) Analysis of MEG Sensor Bias. If Steve wanted to worry about both positive and negative drift, then the previous sample size calculation becomes with z1−α / 2 = z0.975 = 1.96, page 10: 9.07 Lecture 12: Hypothesis Testing n= (1.1)2 (1.96 + 0.84) 2 (0.1)2 1.21(2.8)2 1.21(7.84) = 0.01 0.01 = 949. = Remark 12.5. Notice that (z0.975 + z0.8 )2 ≈ 8. Hence, n≈ 8 SNR Remark 12.6. In Homework Assignment 9 we will explore a similar formula for the binomial distribution. C. Adjustments for the Gaussian Assumptions 1. One-Sample t- Test for Unknown σ 2 . The z − test allows us to test hypotheses about the mean of a Gaussian distribution under the assumption that the variance is known. The t- test allows us to test the same hypotheses when the sample size is not large and the variance is unknown and must be estimated from the sample. The t- test was developed by Gossett in 1908 while he was working in the Guinness Brewery. Gossett wrote under the pseudonym of Student. For this reason it is still referred to as Student’s t-test. The distribution was worked out later by R.A. Fisher. Suppose x1,..., xn is a random sample from a Gaussian probability model N(µ , σ 2 ) and we wish to test the null hypothesis H0 : µ = µ0 against the alternative H A : µ ≠ µ0 . Assume σ 2 is not known and n is not large, say 15 < n < 20. Therefore, as discussed in Lecture 8, we construct a t- test by estimating σ 2 with an unbiased estimate, and instead of a z − statistic we construct a t- statistic as 1 t= n 2 ( x − µ0 ) s where s 2 = (n −1) −1 n ∑ (xi − x )2 i=1 t : tn−1, a t- distribution on n −1 degrees of freedom. Recall that we showed in the practice problems for the second In Class Examination that s 2 is an unbiased estimate of σ 2 . Given α , to test H0 : µ = µ0 against H A : µ ≠ µ0 we reject H0 for | t |> tn−1,1−α 2 page 11: 9.07 Lecture 12: Hypothesis Testing or equivalently if either x > µ0 + tn−1,1−α 2 s n 1 2 or x < µ0 − tn−1,1−α 2 s 1 n2 Example 12.1 Reaction Time Measurements. In a learning experiment, along with the correct and incorrect responses, we record the reaction times which are the times it takes the animal to execute the task. In a previous study, once it had been determined that an animal had learned a task, it was found that the average reaction time was 10 seconds. On the 14 trials after the animal learned the task by the behavioral criteria, the average reaction time was 8.25 seconds. The sample standard deviation was 2.1. Is this animal’s performance different from that previously reported? We have H 0 : µ = 10.0 H A : µ ≠ 10.0 1 1 n 2 ( x − µ ) (14) 2 (8.25 −10) (3.74)(−1.75) t= = = = −2.26 s (2.1) (2.1) Now it follows from the Table of the t-distribution that t13,0.975 = 2.16. Because | t |> t13,0.975 we reject H0 . 2. Binomial Exact Method If np0 (1 − p0 ) < 5, we cannot use the Gaussian approximation to the binomial to tests hypotheses about the binomial proportion. In this case, we base the test on the exact binomial probabilities. k We have x : B(n, p) and we observe k successes, and we take p̂ = . The p-value depends on n whether pˆ ≤ p0 or pˆ > p0 . If pˆ ≤ p0 , then p − value = Pr( ≤ k successes in n trials | H 0 ) 2 k = ∑ (nj ) p0j (1 − p0 )n− j j=0 If pˆ > p0 , then p − value = Pr(≥ k successes in n trials | H 0 ) 2 n = ∑ (nj )p0j (1 − p0 )n− j j =k Example 12.2. A New Learning Experiment. Assume that we execute the learning experiment 1 probability of a correct response by chance. Suppose k = 12 and 3 12 3 1 1 p̂ = = . We want to test H 0 : p = against H A : p ≠ . We see that 20 5 3 3 with 20 trials and there is a page 12: 9.07 Lecture 12: Hypothesis Testing 4 ⎛ 1 ⎞ ⎛ 2 ⎞ 40 np0 (1 − p0 ) = 20 ⎜ ⎟⎜ ⎟ = =4 <5 9 ⎝ 3 ⎠⎝ 3 ⎠ 9 We have 12 1 > and hence pˆ > p0 and we compute the p-value as 20 3 p = Pr( x ≥ 12) = 2 20 ∑( ) 20 j j=12 j ⎛1⎞ ⎛ 2⎞ ⎜ ⎟ ⎜ ⎟ ⎝3⎠ ⎝ 3⎠ 20− j = 0.01286 or equivalently p = 2(0.01286) = 0.02572. Therefore, we reject H0 and conclude that the animal is not performing at chance and most likely has learned. Remark 12.7. An important topic that we have not considered is non-parametric tests. Each of the main parametric tests we considered, i.e., the z-test and t-test has a non-parametric analog. It is important to use these nonparametric tests when the sample size is small and the Gaussian assumption on which most of the tests are based is not valid (Rosner, 2006). Remark 12.8. The requirement of the Gaussian assumption for most the standard hypothesis testing paradigms is very limiting. For this reason, we have spent more time in the course learning about principles of modern statistical analysis such as likelihood methods, the bootstrap and other Monte Carlo methods, Bayesian methods and, as we shall see in Lecture 16, the generalized linear model. D. The Relation Among Confidence Intervals, Hypothesis Tests and p- Values The three statistics we have considered thus far for two-sided hypothesis tests can be used to construct 100 × (1 − α ) confidence intervals. These are z- test (Gaussian mean and variance known) x ± z1−α 2 σ 1 n2 z- test (Binomial proportion) 1 ⎡ p̂(1 − p̂) ⎤ 2 pˆ ± z1−α 2 ⎢ ⎥ ⎣ n ⎦ t- test (Gaussian mean and variance unknown) x ± tn−1,1−α 2 s 1 n2 page 13: 9.07 Lecture 12: Hypothesis Testing We can reject H0 with α = 0.05 if the value of the parameter under the null hypothesis is not in the 100%(1− α ) confidence interval. In this way, a confidence interval provides a hypothesis test. A similar construction of a confidence bound can be used to carry out a one-sided test. Most importantly, the confidence interval tells us the reasonable range for the parameter that can be inferred from the data analysis. Confidence intervals report the analysis results on the physical scale on which the problem takes place. The p- value only tells us how likely the observed statistic would be under H0 . In this way, the hypothesis test simply provides a mechanism for making a decision. Confidence intervals are always more informative than p-values. Realizing this in the early 80s, the New England Journal of Medicine set out a recommendation obliging that all statistical results be reported in terms of confidence intervals. This is now the standard for publication of research articles in that journal. Indeed, p- values alone mean nothing! Example 3.2 (continued) Analysis of MEG Sensor Bias. To illustrate this point, we note that in this problem, the 95% confidence interval is 0.11 ±1.96 1.1 1 (500) 2 0.11 ± 0.096 [0.014, 0.206] and we reject H0 because 0 is not in the confidence interval. The p- value was < 0.05 which essentially tells us nothing about the magnitude of the bias in the sensor. Example 2.1 (continued) Behavioral Learning. In this problem, the 95% confidence interval is 1 ⎛ 1 ⎞2 ⎜ ⎟ 4 p̂ ±1.96 ⎝ ⎠ 1 40 2 1 0.55 ±1.96 2 6.32 0.55 ± 0.158 [0.392, 0.708]. Hence, the p- value is > 0.05. We fail to reject the null hypothesis. The confidence interval not only makes it clear that the null hypothesis is not rejected, it also shows what the reasonable range of uncertainty is for the animal’s propensity to learn. Remark 12.9. One of the true values of the hypothesis-testing paradigm is sample size calculations. page 14: 9.07 Lecture 12: Hypothesis Testing IV. Two-Sample Tests A. Two-Sample t − Test Example 12.3. Reaction Time Analysis in a Learning Experiment. Suppose we have the reaction times on trial 50 of the 9 rats from the treatment group and 13 rats from the control group we studied in Homework Assignment 8-9. The mean reaction time for the treatment group was 15.5 sec with a standard deviation of 3.25 sec and the mean reaction time for the control group was 11.5 sec with a standard deviation of 3.1 sec. What can be said about the underlying mean differences in reaction times between the two groups? Assume we have xiT : N ( µT , σ T2 ) i = 1,..., nT xcj : N ( µc , σ c2 ) j = 1,..., nc Take H 0 : µT = µc H A : µT ≠ µc We have xT = 15.5sec sT = 3.25sec xc = 11.5sec sc = 3.1sec Under H0 we have E( xT − xc ) = E( xT ) − E( xc ) = µT − µc = 0 and hence, 2 σ xT : N ( µT , T ) nT 2 σ xc : N ( µc , c ) nc where nT = 9 and nc = 13. Now under the independence assumption of the two samples 2 2 σ σ Var ( xT − xc ) = Var ( xT ) +Var ( xc ) = T + c nT nc Hence, under H0 and the assumption that σ T2 = σ c2 = σ 2 , page 15: 9.07 Lecture 12: Hypothesis Testing xT − xc : N(0, σ 2 ( 1 1 + )). nT nc If σ 2 were known, then we would have xT − xc : N (0,1) 1 1 1 σ ( + )2 nT nc and we could base our hypothesis test on this z- statistic. Since σ 2 is unknown, let us consider the estimate of σ 2 defined by s2 = (nT −1)sT2 + (nc −1)sc2 nT + nc − 2 where sT2 = (nT −1) −1 nT ∑ (xTj − xT )2 j=1 sc2 = (nc −1) −1 nc ∑ (xic − xc )2 . i=1 Notice that if we assume that σ T2 = σ c2 = σ 2 then E (s 2 ) = (nT −1) E ( sT2 ) + (nc −1) E ( sc2 ) (nT −1)σ 2 + (nc − 1)σ 2 = = σ 2, nT + nc − 2 nT + nc − 2 and s 2 is an unbiased estimate of σ 2 . Given α we can test H0 with the following test statistic t= xT − xc 1 1 1 s( + ) 2 nT nc termed the two-sample t- statistic with equal variance with n = nT + nc − 2 degrees of freedom. We reject H0 at level α | t |> tn,1−α 2 Example 12.3 (continued). For this problem we have page 16: 9.07 Lecture 12: Hypothesis Testing s2 = (nT −1) sT2 + (nc −1) sc2 nT + nc − 2 8(3.25) 2 + (12)(3.1) 2 20 8(10.56) +12(9.61) = 20 84.48 +115.32 = 20 199.8 = 9.999 20 = and hence, t= 15.5 −11.5 4 = 2.92 = 1 1 12 (1.88) 12 [10 × ( + )] 9 13 From the Table of the t-distribution we have t20,0.975 = 2.086. Since t > t20,0.975 we reject H0 and conclude that the mean reaction times of the two groups are different. The longer average reaction time for the treatment group suggests that learning may be impaired in that group. B. Confidence Interval for the True Difference in the Means Because our test statistic follows a t- distribution, we can construct a 100%(1− α ) confidence interval for the true difference in the measure as follows 1 ⎛ 1 1 ⎞2 xT − xc ± tn,1−α 2 s ⎜ + ⎟ . ⎝ nT nC ⎠ Example 12.3 (continued) Reaction Time Analysis in a Behavioral Experiment. If we apply this formula to the data from Example 12.3, we obtain with α = 0.05 a 95% confidence interval for the true mean difference of ⎛1 1 ⎞ 1 4 ± 2.086 ×[10 × ⎜ + ⎟] 2 ⎝ 9 13 ⎠ 4 ± 2.086 × (1.37) 4 ± 2.86 which is [1.14 6.86]. The interval does not contain zero as expected based on our hypothesis test. More importantly, we see that not only is it unlikely that the true mean difference is 0, but that the difference could be as small as 1.14 sec or as large as 6.86 sec. page 17: 9.07 Lecture 12: Hypothesis Testing Remark 12.10. If we cannot assume that the variances in the two samples are equal but unknown, then we have to devise an alternative t- statistic that takes account of the unknown and estimated variances. It can be shown that an appropriate t- statistic in this case is t= xT − xc 1 ⎛ sT2 sc2 ⎞ 2 + ⎟ ⎜ ⎜ nT nc ⎟ ⎠ ⎝ where the number of degrees of freedom is ( d'= ⎛ sT2 ⎜ ⎜ ⎝ nT sT2 sc2 2 + ) nT nc 2 2 ⎛ s2 ⎞ ⎞ ⎟ (nT −1)−1 + ⎜ c ⎟ (nc −1)−1 ⎟ ⎜ ⎟ ⎠ ⎝ nc ⎠ This statistic is the Satterthwaite approximation to the degrees of freedom for a t- statistic with unknown and unequal variance (Rosner, 2006). C. Two-Sample Test for Binomial Proportions Example 2.1 (continued) Behavioral Learning Experiment. Let us assume that on Day 1, the rat had k1 = 22 correct responses and on Day 2 we had k2 = 15 correct responses. Day 1 is out of 40 trials and Day 2 is out of 20 trials. What can we say about the difference in performance between the two days? We could treat the results from Day 1 as “truth” and compare Day 2 to Day 1. A more appropriate way to proceed is to treat the data from both days as if they were observed with uncertainty. For this we assume x1j : B(n1, p1 ) j = 1, ..., n1 x2j : B(n2 , p2 ) j = 1, ..., n2 . Take H 0 : p1 = p2 H A : p1 ≠ p2 Our test statistic can be derived from the estimates of p1 and p2 p̂1 = k1 n1 p̂2 = k2 n2 page 18: 9.07 Lecture 12: Hypothesis Testing Under H0 and using the Gaussian approximation to the binomial p̂1 ≈ N ( p, p(1 − p) ) n1 p̂2 ≈ N ( p, p(1 − p) ) n1 and if we assume the samples are independent we have the approximate z- statistic z= ( p̂1 − p̂2 ) ≈ N (0,1) ⎛ 1 1 ⎞ 12 [ p̂(1 − p̂) ⎜ + ⎟] ⎝ n1 n2 ⎠ where, since p is unknown, we estimate it as p̂ defined as pˆ = n1 pˆ1 + n2 p̂2 k1 + k2 = n1 + n2 n1 + n2 Hence, given a level α we have the approximate z- statistic is z= | p̂1 − p̂2 | ⎛1 1 ⎞1 [ p̂(1− p̂) ⎜ + ⎟] 2 ⎝ n1 n2 ⎠ We reject H0 : if z > z1−α 2 and the approximate p- value is p = 2[1 − Φ( z)]. An alternative form of z that includes the continuity correction (Lecture 8) to make the Gaussian approximation to the binomial more accurate is defined as ⎛ 1 1 ⎞ | p̂1 − p̂2 | − ⎜ + ⎟ 2n1 2n2 ⎠ ⎝ . z= ⎛ 1 1 ⎞ 12 [ p̂(1− p̂) ⎜ + ⎟] ⎝ n1 n2 ⎠ Example 2.1 (continued) Behavioral Learning Experiment. For this problem, we have k +k 22 +15 37 p̂ = 1 2 = = = 0.6167 n1 + n2 60 60 1− p̂ = 0.3833 or page 19: 9.07 Lecture 12: Hypothesis Testing | 0.55 − 0.75 | 1 ⎞1 ⎛ 1 [(0.6167)(0.3833) ⎜ + ⎟] 2 ⎝ 40 20 ⎠ 0.20 0.20 0.20 = = = = 1.50. 1 ⎛ 3 ⎞ 12 (0.01773) 2 0.133 [0.23638 ⎜ ⎟] ⎝ 40 ⎠ z= Because z1−α 2 = z0.975 =1.96 we have z0.975 >| z | so we fail to reject the null hypothesis of no difference in performance. Here the p- value is p = 2[1 − Φ(1.50)] = 2[0.0668] = 0.1336. Similarly, the 95% confidence interval is ⎛1 1 ⎞1 pˆ1 − pˆ 2 ± z0.975 [ p̂(1 − p̂) ⎜ + ⎟] 2 ⎝ n1 n2 ⎠ −0.20 ±1.96(0.133) −0.20 ± 0.26 or [-0.46, 0.06]. As expected, the 95% confidence interval includes zero which explains why we fail to reject H0 . The confidence interval suggests that there may be evidence of improved performance on the second day relative to the first since most of the interval includes negative values of the difference. ⎛ 1 1 ⎞ ⎛ 1 1 ⎞ The value of the continuity correction is ⎜ + ⎟ = ⎜ + ⎟ = 0.0375. This changes z from ⎝ 2n1 2n2 ⎠ ⎝ 80 40 ⎠ 1.50 to z = 1.4625 and the corrected p-value is 0.1442. Remark 12.11. The two sample binomial data can also be analyzed as a 2 × 2 contingency table Correct Day 1 Day 2 k1 k2 k1 + k2 Incorrect n1 − k1 n2 − k2 n1 + n2 − (k1 + k2 ) Total n1 n2 n1 + n2 A 2 × 2 contingency table is a table consisting of two rows cross-classified by two columns. The contingency table analysis uses a test statistic based on a chi-squared distribution with one degree of freedom and gives the same result as the z- test. This is to be expected since we showed in Lecture 4 that the square of a standard Gaussian random variable is a chi-squared random variable with one degree of freedom. We will study this in Homework Assignment 9. Remark 12.12. Use of the Gaussian approximation to the binomial to construct this z- test is valid provided n1 p1 ≥ 5, n2 p2 ≥ 5, n1 (1 − p1 ) ≥ 5 and n2 (1 − p2 ) ≥ 5. This corresponds to the condition page 20: 9.07 Lecture 12: Hypothesis Testing that the analysis of the contingency table using a chi-squared statistic is valid if the expected number of observations per cell is at least 5. Remark 12.13. This problem could easily be analyzed using a Bayesian analysis in which p1 and p2 had uniform priors. We could then use Algorithm 10.1 to compare the posterior densities of p1 and p2 . Remark 12.14. We can perform a likelihood analysis and compare the overlap in the likelihoods. We could alternatively construct 1− α confidence intervals for p1 and p2 separately using the likelihood theory and see if they overlap. Remark 12.15. All the tests we have discussed here can be derived from the likelihood theory we presented by the likelihood ratio procedure. A detailed discussion of this approach is beyond the scope of this course. (See DeGroot and Schervish, 2002; Rice, 2007). V. Summary Hypothesis testing is a key part of classical statistics. It emphasizes procedures based primarily on the Gaussian distribution and Gaussian approximations. The hypothesis testing paradigm is very useful for prospective planning of studies using sample size formulae. Confidence intervals are always more informative than p-values. Confidence intervals report the analysis results on the physical scale on which the problem takes place. Many of the classical problems in hypothesis testing can now be carried out in a more informative way using more modern approaches such as Monte Carlo methods, bootstrapping, and Bayesian methods. Acknowledgments I am grateful to Julie Scott for technical assistance in preparing this lecture and to Jim Mutch for careful proofreading and comments. References DeGroot MH, Schervish MJ. Probability and Statistics, 3rd edition. Boston, MA: Addison Wesley, 2002. Rice JA. Mathematical Statistics and Data Analysis, 3rd edition. Boston, MA, 2007. Rosner B. Fundamentals of Biostatistics, 6th edition. Boston, MA: Duxbury Press, 2006.