Testing Hypotheses about Parameters of Normal Distribution, T-Tests and F-Tests

Section 7 Testing hypotheses about parameters of normal distribution. T-tests and F-tests. We will postpone a more systematic approach to hypotheses testing until the following lectures and in this lecture we will describe in an ad hoc way T-tests and F-tests about the parameters of normal distribution, since they are based on a very similar ideas to confidence intervals for parameters of normal distribution - the topic we have just covered. Suppose that we are given an i.i.d. sample from normal distribution N (µ, ν 2 ) with some unknown parameters µ and ν 2 . We will need to decide between two hypotheses about these unknown parameters - null hypothesis H0 and alternative hypothesis H1 . Hypotheses H0 and H1 will be one of the following: H0 : µ = µ 0 , H 1 : µ = ∞ µ0 , H0 : µ ∼ µ 0 , H 1 : µ < µ 0 , H0 : µ ≈ µ 0 , H1 : µ > µ 0 , where µ0 is a given ’hypothesized’ parameter. We will also consider similar hypotheses about parameter ν 2 . We want to construct a decision rule α : X n � {H0 , H1 } that given an i.i.d. sample (X1 , . . . , Xn ) ≥ X n either accepts H0 or rejects H0 (accepts H1 ). Null hypothesis is usually a ’main’ hypothesis in a sense that it is expected or presumed to be true and we need a lot of evidence to the contrary to reject it. To quantify this, we pick a parameter � ≥ [0, 1], called level of significance, and make sure that a decision rule α rejects H0 when it is actually true with probability ≈ �, i.e. P(α = H1 |H0 ) ≈ �. The probability on the left hand side is understood as a worse case scenario given that the null hypothesis is true, i.e. P (α = H1 |H0 ) = sup (µ,� 2 )�H0 Pµ,�2 (α = H1 ). Level of significance � is usually small, for example, � = 0.05. Example. Let us consider a Matlab example about normal body temperature from the lecture about confidence intervals. If a vector ’normtemp’ represents body temperature measurements of 130 people then typing the following command in Matlab [H,P,CI,STATS] = ttest(normtemp,98.6,0.05,’both’) produces the following output: H = 1, P = 2.4106e-007, CI = [98.1220, 98.3765] STATS = tstat: -5.4548, df: 129, sd: 0.7332. Here µ0 = 98.6, � = 0.05, ’both’ means that we consider a null hypothesis µ = µ0 in which ∞ µ0 is a two-sided hypothesis. The alternative µ > µ0 corresponds to case the alternative µ = parameter ’right’, and µ < µ0 corresponds to parameter ’left’. H = 1 means that we reject null hypothesis and accept H1 , P=2.4106e-007 is a p-value that we will discuss below, CI is a 95% confidence interval for µ0 that we constructed in the previous lecture. If we want to test the hypothesis µ ∼ 98.6 then typing [H,P,CI,STATS] = ttest(normtemp(1:130,1),98.6,0.05,’left’) outputs H = 1, P = 1.2053e-007, CI = [-Inf, 98.3558], STATS = tstat: -5.4548, df: 129, sd: 0.7332. Notice that CI and P are different in this case. The fact that (in both cases) we rejected H 0 means that there is a significant evidence against it. In fact, we will see below that a p-value quantifies in some sense how unlikely it is to observe this dataset assuming that the null hypothesis is true. p-value of order 10−7 is a strong evidence against the hypothesis that a normal body temperature is µ0 = 98.6. Let us explain how these tests are constructed. They are based on the result that we proved before that for MLE µ̂ = X̄ and ν̂ 2 = X̄ 2 − X̄ 2 satisfy ≤ n(µ̂ − µ) nν̂ 2 A= � N(0, 1) and B = 2 � �2n−1 ν ν and the random variables A and B are independent. Hypotheses about mean of one normal sample. We showed that a random variable ≤ µ̂ − µ n−1 � tn−1 ν̂ has tn−1 -distribution with n − 1 degrees of freedom. Let us consider a t-statistic T = ≤ n−1 µ̂ − µ0 . ν̂ This statistic behaves differently depending on whether the ’true’ unknown mean µ = µ 0 , µ < µ0 or µ > µ0 . First of all, if µ = µ0 then T � tn−1 . If µ < µ0 then we can rewrite T = ≤ µ̂ − µ ≤ µ − µ0 n−1 + n−1 � −→ ν̂ ν̂ since the first term has proper distribution tn−1 and the second term goes to infinity. Similarly, when µ > µ0 then T � +→. Therefore, we can make a decision about our hypotheses based on this information about the behavior of T. I. (H0 : µ = µ0 .) In this case, the indication that H0 is not true would be if |T | becomes too large, i.e. T � ±→. Therefore, we consider a decision rule � H0 , if − c ≈ T ≈ c α= H1 , if |T | > c. The choice of the threshold c depends on the level of significance �. We would like to have P(α = H1 |H0 ) = P(|T | > c|H0 ) ≈ �. But given that µ = µ0 , we know that T � tn−1 and, therefore, we can choose c from a condition P(|T | > c|H0 ) = tn−1 (|T | > c) = 2tn−1 ((c, →)) = � using the table of tn−1 -distribution. Notice that this decision rule is equivalent to finding the (1 − �)-confidence interval for unknown parameter µ and making a decision based on whether µ0 falls into this interval. II. (H0 : µ ∼ µ0 .) In this case, the indication that H0 is not true, i.e. µ < µ0 , would be if T � −→. Therefore, we consider a decision rule � H0 , if T ∼ c α= H1 , if T < c. The choice of the threshold c depends on the condition P(α = H1 |H0 ) = P(T < c|H0 ) ≈ �. Since we know that T≤ = T − ≤ n−1 µ − µ0 � tn−1 ν̂ we can write � ≤ µ − µ0 � P(T < c|H0 ) = sup P T ≤ ≈ c − n − 1 = P(T ≤ ≈ c) = tn−1 ((−→, c]) = � ν̂ µ�µ0 and we can find c using the table of tn−1 -distribution. g replacements III. (H0 : µ ≈ µ0 .) Similar to the previous case, the decision rule will be � H0 , if T ≈ c α= H1 , if T > c. and we find c from the condition tn−1 ([c, +→)) = �. p-value. Figure 7.1 illustrates the definition of p-value for all three cases above. p-value can be understood as a probability, given that the null hypothesis H0 is true, to observe a value of T -statistic equally or less likely than the one that was observed. Thus, the small p-value means that the observed T -statistic is very unlikely under the null hypothesis which provides a strong evidence against H0 . The confidence level � defines what we consider as ’unlikely enough’ to reject the null hypothesis. 0.4 0.3 µ = µ0 p-value= 2tn−1 (|T |, →) 0.2 0.1 0 −6 0.4 0.3 −4 −2 0 |T | 2 4 6 2 4 6 µ ∼ µ0 p-value= tn−1 (−→, T ) 0.2 0.1 0 −6 0.4 0.3 −4 −2 T 0 µ ≈ µ0 p-value= tn−1 (T, +→) 0.2 0.1 0 −6 −4 −2 0 T 2 4 6 Figure 7.1: p-values for different cases. Hypotheses about variance of one normal sample. Next we will test similar twosided or one sided hypotheses about the variance, for example, H0 : ν = ν0 vs. H1 : ν = ∞ ν0 , 2 2 2 etc. We will use the fact that nν̂ /ν � �n−1 -distribution and as a result the test will be based of the following statistic: Q= Since we can write Q= nν̂ 2 . ν02 nν̂ 2 ν 2 ν2 2 � � , ν 2 ν02 ν02 n−1 then, clearly, Q will behave differently depending on whether ν = ν0 , ν > ν0 or ν < ν0 . I. (H0 : ν = ν0 .) In this case the decision rule will be � H0 , if c1 ≈ Q ≈ c2 α= H1 , if Q < c1 , c2 < Q. Thresholds c1 , c2 should satisfy the condition P(α = H1 |H0 ) = P(Q < c1 |ν = ν0 ) + P(Q > c2 |ν = ν0 ) = �2n−1 (0, c1 ) + �2n−1 (c2 , →) = �. For example, we can take �2n−1 (0, c1 ) = � � and �2n−1 (c2 , →) = . 2 2 II. (H0 : ν ≈ ν0 .) In this case the decision rule will be � H0 , if Q ≈ c α= H1 , if Q > c. Threshold c should satisfy the condition P(α = H1 |H0 ) = sup P(Q > c) = sup P ���0 ���0 � nν̂ 2 ν2 > � nν̂ 2 � ν02 � c = P > c = �2n−1 (c, →) = �. ν2 ν2 III. (H0 : ν ∼ ν0 .) In this case the decision rule will be � H0 , if Q ∼ c α= H1 , if Q < c. Threshold c is determined by P(α = H1 |H0 ) = sup P(Q < c) = P ���0 � nν̂ 2 ν2 � < c = �2n−1 (0, c) = �. Comparing means of two normal samples. In the normal body temperature dataset first 65 observations correspond to men and last 65 observations correspond to women. We would like to test the hypothesis that normal body temperature of men and women is the same. There are several way to do this. Paired t-test. First, we can perform the so called paired t-test. Since the number of observations is the same in both groups, we can pair them together and assume that their differences Zi = Xi − Yi will also be normal. This sounds like a reasonable assumption since Xi and Yi should be independent if the measurements were taken independently. Since µz = µx − µy , hypothesis µx = µy is equivalent to µz = 0 which means that we can do the usual t-test for one sample Z1 , . . . , Zn . Running [H,P,CI,STATS]=ttest(men,women,0.05,’both’) outputs H = 1, P = 3.9773e-019, CI = [-0.3348,-0.2437] STATS = tstat: -12.6858, df: 64, sd: 0.1838. We reject null hypothesis that the means are equal and, in fact, p-value of order 10 −19 is a strong evidence against it. However, it seems rather suspicious that there is such a strong evidence against H0 , especially after we perform a two sample t-test below which also rejects H0 but with a much higher p-value of 0.0239. When we examine the data file more closely we notice that the body temperatures were arranged in an increasing order both for men and women. This means that the assumption that our samples are i.i.d. is not longer true. To restore this, we randomly permute both vectors and denote their difference by ’z’. (To permute ’men’ type ’men(randperm(65))’, the same for women.) Then performing t-test for the difference ’z’ [H,P,CI,STATS]=ttest(z,0,0.05,’both’) we get H = 1, P =0.0442, CI = [0.0078, 0.5707] STATS = tstat: 2.0528, df: 64, sd: 1.1359 which is a more reasonable (and correct) outcome. Two sample t-test assuming equal variances. If we run the following command in Matlab: [H,P,CI,STATS]=ttest2(men,women,0.05,’both’,’equal’) we get the following output: H = 1, P = 0.0239, CI = [-0.5396, -0.0388], STATS = tstat: -2.2854, df: 128, sd: 0.7215. We again reject the hypothesis that µx = µy at the level of significance � = 0.05 but this time p-value is equal to 0.0239. Here ’both’ means that we test two-sided hypothesis µ x = µy , and ’equal’ means that we assume that the ’true’ unknown variances of the distributions of two samples νx2 and νy2 are equal, i.e. νx2 = νy2 = ν 2 Let n and m be the number of observations in the first sample (Xs) and second sample (Y s) correspondingly. We proved that ≤ n(µ̂x − µx ) nν̂ 2 Ax = � N(0, 1) and Bx = 2x � �2n−1 νx νx and Ay = ≤ mˆ νy2 m(µ̂y − µy ) � N(0, 1) and By = 2 � �2m−1 νy νy and Ax , Bx , Ay , By are independent. Using the properties of normal distribution we get � (µ̂ − µ ) (µ̂ − µ ) ��� 1 1� x x y y A= − + � N(0, 1) νx n m νy and by definition of �2 -distributions, B= Therefore, nν̂x2 mν̂y2 + 2 � �2n+m−2 . νx2 νy �� 1 B � tn+m−2 . n+m−2 Notice that because νx2 and νy2 are unknown, in general, we can not compute this expression. However, if we assume that variances are equal then all νx and νy will cancel out and we will get � nm(n + m − 2) �1/2 (µ̂ − µ̂ ) − (µ − µ ) x y x y � tn+m−2. 2 2 1/2 n+m (nν̂x + mν̂y ) A Since this expression depends only on the difference of means µx −µy , we can test hypotheses about this difference based on the statistic � nm(n + m − 2) �1/2 µ̂x − µ̂y T = 2 n+m (nν̂x + mν̂y2 )1/2 For example, if we want to test H0 : µx = µy or, equivalently, µx − µy = 0 we can consider a decision rule � H0 , if − c ≈ T ≈ c α= H1 , if |T | > c and find c from the condition 2tn+m−2 (c, →) = �. Notice that in the Matlab output above we have df = 128, i.e. n + m − 2 = 65 + 65 − 2 = 128 degrees of freedom. One-sided tests are also similar to the case of one sample test. t-test with unequal variances. Assuming that variances are equal could be unjusti­ fied. There is a version of t-test which does not make this assumption. However, we can no compute exactly the distribution of t-statistic above (since variances do not cancel out), and we can only construct ’approximate’ tests. For example, running in Matlab: [H,P,CI,STATS]=ttest2(men,women,0.05,’both’,’unequal’) gives H = 1, P = 0.0239, CI =[-0.5396,-0.0388], STATS = tstat: -2.2854, df: 127.5103, sd: [2x1 double]. Notice non integer value for degrees of freedom 127.5103. To construct the test for this general case we can start with (using properties of normal distribution) or � ν2 ν2 � y (µ̂x − µx ) − (µ̂y − µy ) � N 0, x + n m νy2 �1/2 ((µ̂x − µ̂y ) − (µx − µy ))/ + � N(0, 1) n m We do not know the variances νx2 and νy2 but we know by law of large numbers that their estimates ν̂x2 and ν̂y2 converge and, therefore, � ν2 x (µ̂x − µ̂y ) − (µx − µy )) ν̂y2 �1/2 + � N(0, 1) n m �� ν̂ 2 x will have approximately normal distribution when n and m are large. We can now construct all the tests as above, only now they will be approximate. However, usually a different (supposedly better) approximation is used, called Satterthwaite approximation, also used by Matlab. First of all, instead of ν̂x2 we will use unbiased estimated of variance: 2 νx≤ = nν̂x2 n−1 which will give us a slightly different expression ((µ̂x − µ̂y ) − (µx − µy )) �� ν̂ 2 ν̂y2 �1/2 x + � N(0, 1). n−1 m−1 Unbiased estimate νx≤ 2 is different from MLE ν̂x2 only by a fraction n/(n − 1) and we can see that this makes very small difference between two expressions above. More important differ­ ence is that instead of using normal approximation � N(0, 1) we will use a t� -distribution approximation �� ν̂ 2 ν̂y2 �1/2 x + � t� (7.0.1) ((µ̂x − µ̂y ) − (µx − µy )) n−1 m−1 where the number of degrees of freedom � is determined from the following consideration. We know from the definition of tn -distribution and properties of �2n -distribution that (using informal notations) N(0, 1) N(0, 1) tn = � =� � �. 1 2 1 n 1 � � 2, 2 n n n This could be used as a definition of tn -distribution even when degrees of freedom parameter n is not integer. To find a good approximation in (7.0.1), we need to find a good approximation � ν̂ 2 ν̂y2 � � νx2 νy2 � 1 � � 1 � x + / + � � , . n−1 m−1 n n � 2 2 It is easy to check that the expectations of both sides are equal, so we will choose � from the condition that the variances of both sides are also equal, which will give � 2 � �y2 2 �x + m n �= � 2 �2 � 2 �2 . �y �x 1 1 + n−1 n y−1 m Finally, since the variances are unknown we will replace them by their unbiased estimates and take �2 � 2 � ˆy2 � ˆx + n−1 m−1 �= � 2 �2 � 2 �2 . � ˆ � ˆx 1 1 + y−1 my n−1 n−1 Therefore, we obtain approximation (7.0.1) which is supposedly better than a simple normal approximation. The degrees of freedom df : 127.5103 in the Matlab output is precisely given by this formula. Comparing the variances of two normal distributions: F -test. Suppose that we want to test whether the variances of two normal distributions are equal. For example, in the first two sample t-test we assumed that νx2 = νy2 . We can test this in Matlab: [H,P,CI,STATS]=vartest2(men,women,0.05,’both’) and we get the following output: H = 0, P = 0.6211, CI = [0.5388, 1.4481] STATS = fstat: 0.8833, df1: 64, df2: 64. We accept the two-sided null hypothesis H0 : νx = νy . The high p-value 0.6211 means that there is no evidence against null hypothesis. This test is constructed as follows. Since we know that mν̂y2 nν̂x2 2 Bx = 2 � �n−1 and By = 2 � �2m−1 νx νy the ratio Bx /(n − 1) n(m − 1)ν̂x2 νy2 = � Fn−1,m−1 By /(m − 1) m(n − 1)ν̂y2 νx2 has Fm−1,n−1 -distribution with (n − 1, m − 1) degrees of freedom. Let us consider a statistic n(m − 1)ν̂x2 νx2 F = � 2 Fn−1,m−1 . m(n − 1)ν̂y2 νy When νx2 = νy2 , we have F � Fn−1,m−1 , when νx2 > νy2 , F will tend to be above the ’typical range’ of Fn−1,m−1 distribution, and when νx2 < νy2 , F will tend to be below the ’typical range’ of Fn−1,m−1 distribution. As a result, we get the following tests. I. (H0 : νx = νy .) The decision rule will be � H0 , if c1 ≈ F ≈ c2 α= H1 , if F < c1 , c2 < F. Thresholds c1 , c2 should satisfy the condition P(α = H1 |H0 ) = P(F < c1 |νx = νy ) + P(F > c2 |νx = νy ) = Fn−1,m−1 (0, c1 ) + Fn−1,m−1 (c2 , →) = �. For example, we can take Fn−1,m−1 (0, c1 ) = � � and Fn−1,m−1 (c2 , →) = . 2 2 II. (H0 : νx ≈ νy .) The decision rule will be � H0 , if F ≈ c α= H1 , if F > c. Thresholds c should satisfy the condition P(α = H1 |H0 ) = P(F > c|νx = νy ) = Fn−1,m−1 (c, →) = �. The test for H0 : νx ∼ νy is similar.