Experimental Research Methods - Lecture 4

Marie Steinkampf, 2036703 {Experimental Research Methods} Lecture 4 - Assumptions of ANOVA • ANOVA vs. t-test vs. regression ○ Independent samples t-test: Do male and female students differ in their self-esteem? ○ Can also use ANOVA to answer research question ○ Why are the results the same?  Clear relationship between the results of the two-sided t-test under the assumption of equal population variances and the results of the ANOVA if there are only two groups  t2 = F → 1.6222 = 2.632  Corrected Model in SPSS output of ANOVA: testing whether any of the independent variables has an effect ∙ Only one independent variable: results in Corrected Model and sex/therapy exactly the same ∙ Include multiples independent variables: results differ from each other  Intercept in ANOVA irrelevant, not interpreted (same as in regression analysis → only look at regression coefficients) Can also use regression analysis to answer research question ○   Including a dummy variable for gender (0 = females, 1 = males) Coefficients table: t-value for dummy variable is the same as in the independent samples t-test Experimental Research Methods 1 Marie Steinkampf, 2036703  • • SPSS also reports table with ANOVA results (compares whether effect of any of the variables): F-value is the same as in the ANOVA  Results are the same as in ANOVA and independent samples t-test ○ Conclusion  Hypothesis tested in ANOVA, independent samples t-test and regression analysis is the same if K = 2 (total number of groups)  Decision which procedure to use depends on type of variables ∙ Independent samples t-test: compare only two groups ∙ Regression analysis or ANOVA: compare more than two groups  Regression analysis: continuous independent variable  ANOVA: categorical independent variable Assumptions of a statistical test ○ The result of a statistical test is valid if all assumptions of the test are met  If all assumptions met: assume reported p-value very close to true p-value ○ If an assumption of a test is violated, then the test is: (i) robust of (ii) not robust to a violation of that assumption  Either one can still interpret or not interpret the results anymore (wrong conclusions) ○ A test is robust to the violation if the true p-value is approximately equal to the reported p-value (SPSS output) ○ A test is not robust if the true p-value of the test is not equal to the reported p-value → cannot be interpreted  Reported p-value much smaller than true p-value: falsely reject H0 ○ If (i) the test is robust to the violation, then the results of the results of the test (reported p-value) are still considered valid and can be interpreted  If an assumption violated & we know given our design that analysis robust to violation of the assumption: assume reported p-values in SPSS very close to true pvalues → still interpret results ○ If (ii) the test is not robust to the violation, the results of the test (reported p-value) are not considered valid and cannot be interpreted → use different test Assumptions of ANOVA 1. Independent observations (methodological assumption)  Violated in case of repeated measures of the same person (after multiple manipulations/treatments)  ANOVA not robust to violation of this assumption → use Repeated Measures ANOVA as a different test 2. Residuals are normally distributed for each group (ik in statistical model)  ANOVA robust to violation of this assumption if sample size per group  30  If n < 30 per group → use different test not discussed in this course  Related to central limit theorem: the larger the sample size, the more likely the sampling distribution of the mean approximates a normal distribution → less likely violation of assumption of normally distributed residuals problematic 3. The variance of the residuals is the same across group (homogeneity of variance)  Tested using Levene’s test in SPSS in both independent samples t-test and ANOVA ∙ Violation: Levene’s statistically significant (population variances of different groups not the same)  ANOVA robust to violation of this assumption if the highest sample size per group nk is not more than 1.5 times as high as the lowest nk Experimental Research Methods 2 Marie Steinkampf, 2036703 ∙ • • - - If sample sizes approximately equal, then ANOVA robust to violation of homogeneity of variance assumption ∙ If variances within groups are unequal and the highest nk is more than 1.5 times as high as the lowest nk (samples sizes per group differ too much between different groups), problematic for ANOVA → results cannot be interpreted without another test not discussed in this course Alternative tests with fewer assumptions also have disadvantages ○ If the assumptions are not violated, the regular ANOVA has more power to reject H0 compared to other tests with fewer assumptions ○ Important to always check whether assumptions violated and whether analysis robust to violation before using any (other) statistical technique Decision tree of assumptions Recap ANOVA • Technique used to compare multiple population means between groups in one Omnibus hypothesis test (analysis of variance) ○ H0: all K groups have the same mean (mew1 = mew2 = mew3 = mew4) ○ H1: not H0 • In ANOVA, we can split the total Sum of Squares (SST) into the Sum of Squares between (SSB) and the Sum of Squares within (SSW) • We calculate the F-statistic using the Mean Squares between (MSB) and the Mean Squares within (MSW) → compare to critical F-value or p-value in SPSS with alpha level → reject H0? • Differences among group means ○ If statistically significant effect (H0 is rejected), we have convincing evidence that not all of the K group means are the same, but we do not know yet which group means differ from each other (e.g., between all four or only two groups) ○ To better understand the nature of the group-mean differences, we need a way to decide which individual means do indeed differ from each other → specific group comparisons Testing contrasts • Contrast: a specific comparison between group means Experimental Research Methods 3 Marie Steinkampf, 2036703 • • Individual comparisons of interest can be specified a priori (hypothesis part of study design before collecting or seeing data) or post hoc (e.g., to explore significant ANOVA after conducting) ○ A priori analysis: plan in advance to contrast the mean of each experimental group against the mean of the control group ○ Post hoc analysis: contrast the group with the highest mean with the experimental group that had the lowest mean based on sample results Simple vs. complex comparisons ○ Both for a priori and post hoc contrasts Alpha level gets inflated with doing multiple comparisons Simple (pairwise) comparison: comparing population means of two different groups → many different options  Comparable to independent samples t-test comparing population means of two different groups, but the way hypothesis tested in contrast for simple (pairwise) comparisons slightly different ○ Complex comparison: comparing multiple groups with each other (e.g., mew1 with mew2 & mew3 combined → very similar)  Test null hypothesis that the average in the combined behavioral therapy is the same as the average in the Rogerian therapy group: test whether on average the behavioral therapy groups have an effect compared to the Rogerian therapy group ○ Not all groups need to be included  Not necessary to compare all different population means with each other Many different types and methods of testing ○ Not all contrasts and methods implemented in SPSS  A priori: six different options  Post hoc: even more options ○ We will focus on a selection of these contrasts ○ • • Specifying contrasts ○ Contrasts are denoted by Greek letter  (psi) and in general have the form:  cs (contrast coefficients) multiplied with population means refer to particular weight that each group mean will get in hypothesis test ∙ Chosen by researcher such that they reflect the hypothesis of interest ∙ Restriction: sum of the cs must be 0 →   = contrast value reflecting the mean difference of the groups that are contrasted  To test the significance of a contrast, one tests ∙ H0:  = 0 against H1:   0 Experimental Research Methods 4 Marie Steinkampf, 2036703 ○ Example ○ Procedure to determine contrast coefficients when nk are equal  Divide all K groups into three categories that correspond with H0 1. For all groups that do not participate: ck = 0 2. For all groups that participate on one side of equal sign in H0: ck = 1 divided by the number of groups on this one side 3. For groups that participate on the other side of equal sign in H0: ck = -1 divided by the number of groups on this other side Remark  Only restriction on the contrast coefficients is that they sum up to 0 → allowed to multiply the cs with a constant without changing the statistical test  Example: testing a contrast with c = (1, - ½, - ½) is equivalent to testing a contrast with c = (2, -1, -1) or c = (-2, 1, 1) → all coefficients multiplied by 2; all coefficients multiplied by -1  But: c = (-1, -1, 2) → testing different hypothesis: comparing combined mean of first and second group with mean of third group Procedure to determine contrast coefficients when nk are unequal ○ ○ sum of all sample sizes that belong to the same side (category)  Divide all K groups in three categories that correspond with H0 1. For all groups that do not participate: ck = 0 2. For all groups that participate on one side: where the summation is over all groups on this side Experimental Research Methods 5 Marie Steinkampf, 2036703 - 3. For all groups that participate on the other side: where the summation is over all groups on the other side Interpreting contrasts in SPSS output • Planned contrasts ○ Testing of contrast(s) (i) before you have seen the data, and (ii) instead of ANOVA and/or next to ANOVA  Instead of ANOVA: not interested in whether at least one population mean different from the others, but only interested whether H0 can be rejected for particular combination of population means  Next to ANOVA: interested in hypothesis of ANOVA, but also interested in another hypothesis ○ Conducting planned contrasts after having seen the data is not allowed (a priori): higher Type I error rate → more likely incorrectly reject H0 ○ Number of planned contrasts is often small (smaller than the number of K groups): very specific hypotheses • If testing multiple contrasts, need to take higher Type I error rate into account: each test brings risk of making Type I error (claiming presence of non-existent effect) ○ Experiment-wise Type I error rate  Only testing one hypothesis with alpha level 0.05: probability of making Type I error = 5%  Testing multiple hypotheses (effect of experiment by means of several contrasts): inflated Type I error rate ∙ Effect is found if at least one of the contrasts is significant (at least one of the experimental methods shows an effect) ∙ If multiple contrasts tested, each using alpha level 0.05, higher than 0.05 chance of finding at least one significant result ∙ Experiment-wise Type I error level (probability of finding at least one Type I error) will be larger than alpha level ○ To control the overall experiment-wise Type I error rate, one needs to use a smaller alpha level for each tested contrast → Bonferroni Correction  alphacontrast = alpha/C  C = number of contrasts being tested  Set the bar for significance higher: observe smaller p-value in data in order to conclude statistical significance • Testing hypothesis including planned contrast Experimental Research Methods 6 Marie Steinkampf, 2036703 ○ ○ Planned contrast can only be obtained by modifying the syntax of the ANOVA when the General Linear Model/Univariate procedure is used  Specifying planned contrast: comparing first group with second and third groups Output    Contrast estimate =  p < alpha of 0.05 → significant result Conclude that population mean of Rogerian therapy differs from combined population means of both behavioral therapies Experimental Research Methods 7