Tests of Independence and Homogeneity

Section 12 Tests of independence and homogeneity. In this lecture we will consider a situation when our observations are classified by two different features and we would like to test if these features are independent. For example, we can ask if the number of children in a family and family income are independent. Our sample space X will consist of a × b pairs X = {(i, j) : i = 1, . . . , a, j = 1, . . . , b} where the first coordinate represents the first feature that belongs to one of a categories and the second coordinate represents the second feature that belongs to one of b categories. An i.i.d. sample X1 , . . . , Xn can be represented by a contingency table below where Nij is the number all observations in a cell (i, j). Table 12.1: Contingency table. Feature 2 Feature 1 1 2 ··· b N11 N12 · · · N1b 1 N21 N22 · · · N2b 2 .. .. .. .. .. . . . . . a Na1 Na2 · · · Nab We would like to test the independence of two features which means that P(X = (i, j)) = P(X 1 = i)P(X 2 = j). If we introduce the notations P(X = (i, j)) = αij , P(X 1 = i) = pi and P(X 2 = j) = qj , 77 then we want to test that for all i and j we have αij = pi qj . Therefore, our hypotheses can be formulated as follows: H0 : αij = pi qj for all (i, j) for some (p1 , . . . , pa ) and (q1 , . . . , qb ) H1 : otherwise. We can see that this null hypothesis H0 is a special case of the composite hypotheses from previous lecture and it can be tested using the chi-squared goodness-of-fit test. The total number of groups is r = a × b. Since pi s and qj s should add up to one p1 + . . . + pa = 1 and q1 + . . . + qb = 1 one parameter in each sequence, for example pa and qb , can be computed in terms of other probabilities and we can take (p1 , . . . , pa−1 ) and (q1 , . . . , qb−1 ) as free parameters of the model. This means that the dimension of the parameter set is s = (a − 1) + (b − 1). Therefore, if we find the maximum likelihood estimates for the parameters of this model then the chi-squared statistic: T = � (Nij − np�i qj� )2 2 � �2r−s−1 = �2ab−(a−1)−(b−1)−1 = �(a−1)(b−1) � � np q i j i,j converges in distribution to �2(a−1)(b−1) distribution with (a − 1)(b − 1) degrees of freedom. To formulate the test it remains to find the maximum likelihood estimates of the parameters. We need to maximize the likelihood function � � P Nij � P N � N � N Nij (pi qj ) = pi j qj i ij = pi i+ qj +j i,j i j i j where we introduced the notations Ni+ = � Nij and N+j = j � Nij i for the total number of observations in the ith row and jth column. Since pi s and qj s are not related other, maximizing the likelihood function is equivalent to maxi� Ntoi+ each � �a above N+j Ni+ mizing i pi and j qj separately. Let us maximize i=1 pi or, taking the logarithm, maximize a a−1 � � Ni+ log pi = Ni+ log pi + Na+ log(1 − p1 − . . . − pa ), i=1 i=1 since the probabilities add up to one. Setting derivative in pi equal to zero, we get Na+ Ni+ Na+ Ni+ − = − =0 pi pi 1 − p1 − . . . − pa−1 pa 78 or Ni+ pa = Na+ pi . Adding up these equations for all i � a gives npa = Na+ =≤ pa = Na+ Ni+ =≤ pi = . n n Therefore, we get that the MLE for pi : p�i = Ni+ . n Similarly, the MLE for qj is: N+j . n Therefore, chi-square statistic T in this case can be written as qj� = T = � (Nij − Ni+ N+j /n)2 Ni+ N+j /n i,j and the decision rule is given by �= � H1 : T � c H2 : T > c where the threshold is determined from the condition �2(a−1)(b−1) (c, +→) = �. Example. In 1992 poll 189 Montana residents were asked whether their personal finan­ cial status was worse, the same or better than one year ago. The opinions were divided into three groups by income range: under 20K, between 20K and 35K, and over 35K. We would like to test if opinions were independent of income. Table 12.2: Montana outlook poll. b=3 a=3 Worse Same Better � 20K 47 20 15 12 (20K, 35K) 83 24 27 32 59 � 35K 14 22 23 189 58 64 67 The chi-squared statistic is T = (23 − 67 × 59/189)2 (20 − 47 × 58/189)2 +...+ = 5.21. 47 × 58/189 67 × 59/189 79 If we take level of significance � = 0.05 then the threshold c is: �2(a−1)(b−1) (c, +→) = �24 (c, →) = � = 0.05 ≤ c = 9.488. Since T = 5.21 < c = 9.488 we accept the null hypothesis that opinions are independent of income. Test of homogeneity. Suppose that the population is divided into R groups and each group (or the entire population) is divided into C categories. We would like to test whether the distribution of categories in each group is the same. Table 12.3: Test of homogeneity Category 1 · · · Category C Group 1 N11 ··· N1C .. .. .. .. . . . . Group NR1 ··· NRC � R N+1 ··· N+C � N1+ .. . NR+ n If we denote P(Categoryj |Groupi ) = pij so that for each group i � R we have C � pij = 1 j=1 then we want to test the following hypotheses: H0 : pij = pj for all groups i � R H1 : otherwise If observations X1 , . . . , Xn are sampled independently from the entire population then ho­ mogeneity over groups is the same as independence of groups and categories. Indeed, if have homogeneity P(Categoryj |Groupi ) = P(Categoryj ) then we have P(Groupi , Categoryj ) = P(Categoryj |Groupi )P(Groupi ) = P(Categoryj )P(Groupi ) which means the groups and categories are independent. Another way around, if we have independence then P(Groupi , Categoryj ) P(Groupi ) P(Categoryj )P(Groupi ) = P(Categoryj ) = P(Groupi ) P(Categoryj |Groupi ) = 80 which is homogeneity. This means that to test homogeneity we can use the test of indepen­ dence above. Interestingly, the same test can be used in the case when the sampling is done not from the entire population but from each group separately which means that we decide a priori about the sample size in each group - N1+ , . . . , NR+ . When we sample from the entire pop­ ulation these numbers are random and by the LLN Ni+ /n will approximate the probability P(Groupi ), i.e. Ni+ reflects the proportion of group i in the population. When we pick these numbers a priori one can simply think that we artificially renormalize the proportion of each group in the population and test for homogeneity among groups as independence in this new artificial population. Another way to argue that the test will be the same is as follows. Assume that P(Categoryj |Groupi ) = pj where the probabilities pj are all given. Then by Pearson’s theorem we have the convergence in distribution C � (Nij − Ni+ pj )2 � �2C −1 Ni+ pj j=1 for each group i � R which implies that R � C � (Nij − Ni+ pj )2 � �2R(C−1) N p i+ j i=1 j=1 since the samples in different groups are independent. If now we assume that probabilities p1 , . . . , pC are unknown and plug in the maximum likelihood estimates p�j = N+j /n then R � C � (Nij − Ni+ N+j /n)2 2 � �2R(C−1)−(C−1) = �(R−1)(C−1) N N /n i+ +j i=1 j=1 because we have C − 1 free parameters p1 , . . . , pC−1 and estimating each unknown parameter results in losing one degree of freedom. Example (Textbook, page 560). In this example, 100 people were asked whether the service provided by the fire department in the city was satisfactory. Shortly after the survey, a large fire occured in the city. Suppose that the same 100 people were asked whether they thought that the service provided by the fire department was satisfactory. The result are in the following table: Satisfactory Unsatisfactory Before fire 80 20 After fire 72 28 Suppose that we would like to test whether the opinions changed after the fire by using a chi-squared test. However, the i.i.d. sample consisted of pairs of opinions of 100 people 1 2 (X11 , X12 ), . . . , (X100 , X100 ) 81 where the first coordinate/feature is a person’s opinion before the fire and it belongs to one of two categories {“Satisfactory“, ”Unsatisfactory”}, and the second coordinate/feature is a person’s opinion after the fire and it also belongs to one of two categories {“Satisfactory“, ”Unsatisfactory”}. So the correct contingency table corresponding to the above data and satisfying the assump­ tion of the chi-squared test would be the following: Sat. after Uns. after Sat. before 70 2 Uns. before 10 18 In order to use the first contingency table, we would have to poll 100 people after the fire independently of the 100 people polled before the fire. 82